Skip to content
AWS for Query.Farm

aws.emr.list_instance_groups

Example SQL Queries

SELECT * FROM
aws.emr.list_instance_groups
WHERE
"cluster_id" = 'VALUE';

Description

Provides all available details about the instance groups in a cluster.

Table Definition

Column NameColumn Data Type
cluster_id Required Input Column

The identifier of the cluster for which to list the instance groups.

VARCHAR
_aws_profile Input Column

The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role.

STRUCT(
    "type" VARCHAR,
    "name" VARCHAR,
    "account_id" VARCHAR,
    "via_profile_name" VARCHAR,
    "assumed_role_arn" VARCHAR,
    "organization" STRUCT(
        "account_name" VARCHAR,
        "id" VARCHAR,
        "tags" STRUCT(
            "key" VARCHAR,
            "value" VARCHAR
        )[],
        "master_account" STRUCT(
            "id" VARCHAR,
            "email" VARCHAR
        ),
        "parents" STRUCT(
            "type" VARCHAR,
            "id" VARCHAR,
            "name" VARCHAR,
            "tags" STRUCT(
                "key" VARCHAR,
                "value" VARCHAR
            )[]
        )[]
    )
)
Show child fields
_aws_profile.account_id

The AWS account id

_aws_profile.assumed_role_arn

The ARN of the assumed role

_aws_profile.name

The unique name of the profile.

_aws_profile.organization

Information about this profile's membership in the AWS organization.

Show child fields
_aws_profile.organization.account_name

The name of account speciifed by the organization

_aws_profile.organization.id

The organization id

_aws_profile.organization.master_account
Show child fields
_aws_profile.organization.master_account.email

The organization master account email address

_aws_profile.organization.master_account.id

The organization master account id

_aws_profile.organization.parents[]
Show child fields
_aws_profile.organization.parents[].id

The id of the parent

_aws_profile.organization.parents[].name

The name of the parent

_aws_profile.organization.parents[].tags[]
Show child fields
_aws_profile.organization.parents[].tags[].key
_aws_profile.organization.parents[].tags[].value
_aws_profile.organization.parents[].type

The type of parent can be an organization unit or a root

_aws_profile.organization.tags[]
Show child fields
_aws_profile.organization.tags[].key
_aws_profile.organization.tags[].value
_aws_profile.type

The type of profile, either 'credentials' or 'assumed_role'

_aws_profile.via_profile_name

This IAM role for this profile is assumed by first utilizing another profile with this name to obtain credentials.

auto_scaling_policy

An automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster. The automatic scaling policy defines how an instance group dynamically adds and terminates Amazon EC2 instances in response to the value of a CloudWatch metric. See PutAutoScalingPolicy.

STRUCT(
    "status" STRUCT(
        "state" VARCHAR,
        "state_change_reason" STRUCT(
            "code" VARCHAR,
            "message" VARCHAR
        )
    ),
    "constraints" STRUCT(
        "min_capacity" BIGINT,
        "max_capacity" BIGINT
    ),
    "rules" STRUCT(
        "name" VARCHAR,
        "description" VARCHAR,
        "action" STRUCT(
            "market" VARCHAR,
            "simple_scaling_policy_configuration" STRUCT(
                "adjustment_type" VARCHAR,
                "scaling_adjustment" BIGINT,
                "cool_down" BIGINT
            )
        ),
        "trigger" STRUCT(
            "cloud_watch_alarm_definition" STRUCT(
                "comparison_operator" VARCHAR,
                "evaluation_periods" BIGINT,
                "metric_name" VARCHAR,
                "namespace" VARCHAR,
                "period" BIGINT,
                "statistic" VARCHAR,
                "threshold" DOUBLE,
                "unit" VARCHAR,
                "dimensions" STRUCT(
                    "key" VARCHAR,
                    "value" VARCHAR
                )[]
            )
        )
    )[]
)
Show child fields
auto_scaling_policy.constraints

The upper and lower Amazon EC2 instance limits for an automatic scaling policy. Automatic scaling activity will not cause an instance group to grow above or below these limits.

Show child fields
auto_scaling_policy.constraints.max_capacity

The upper boundary of Amazon EC2 instances in an instance group beyond which scaling activities are not allowed to grow. Scale-out activities will not add instances beyond this boundary.

auto_scaling_policy.constraints.min_capacity

The lower boundary of Amazon EC2 instances in an instance group below which scaling activities are not allowed to shrink. Scale-in activities will not terminate instances below this boundary.

auto_scaling_policy.rules[]
Show child fields
auto_scaling_policy.rules[].action

The conditions that trigger an automatic scaling activity.

Show child fields
auto_scaling_policy.rules[].action.market

Not available for instance groups. Instance groups use the market type specified for the group.

auto_scaling_policy.rules[].action.simple_scaling_policy_configuration

The type of adjustment the automatic scaling activity makes when triggered, and the periodicity of the adjustment.

Show child fields
auto_scaling_policy.rules[].action.simple_scaling_policy_configuration.adjustment_type

The way in which Amazon EC2 instances are added (if ScalingAdjustment is a positive number) or terminated (if ScalingAdjustment is a negative number) each time the scaling activity is triggered. CHANGE_IN_CAPACITY is the default. CHANGE_IN_CAPACITY indicates that the Amazon EC2 instance count increments or decrements by ScalingAdjustment, which should be expressed as an integer. PERCENT_CHANGE_IN_CAPACITY indicates the instance count increments or decrements by the percentage specified by ScalingAdjustment, which should be expressed as an integer. For example, 20 indicates an increase in 20% increments of cluster capacity. EXACT_CAPACITY indicates the scaling activity results in an instance group with the number of Amazon EC2 instances specified by ScalingAdjustment, which should be expressed as a positive integer.

auto_scaling_policy.rules[].action.simple_scaling_policy_configuration.cool_down

The amount of time, in seconds, after a scaling activity completes before any further trigger-related scaling activities can start. The default value is 0.

auto_scaling_policy.rules[].action.simple_scaling_policy_configuration.scaling_adjustment

The amount by which to scale in or scale out, based on the specified AdjustmentType. A positive value adds to the instance group's Amazon EC2 instance count while a negative number removes instances. If AdjustmentType is set to EXACT_CAPACITY, the number should only be a positive integer. If AdjustmentType is set to PERCENT_CHANGE_IN_CAPACITY, the value should express the percentage as an integer. For example, -20 indicates a decrease in 20% increments of cluster capacity.

auto_scaling_policy.rules[].description

A friendly, more verbose description of the automatic scaling rule.

auto_scaling_policy.rules[].name

The name used to identify an automatic scaling rule. Rule names must be unique within a scaling policy.

auto_scaling_policy.rules[].trigger

The CloudWatch alarm definition that determines when automatic scaling activity is triggered.

Show child fields
auto_scaling_policy.rules[].trigger.cloud_watch_alarm_definition

The definition of a CloudWatch metric alarm. When the defined alarm conditions are met along with other trigger parameters, scaling activity begins.

Show child fields
auto_scaling_policy.rules[].trigger.cloud_watch_alarm_definition.comparison_operator

Determines how the metric specified by MetricName is compared to the value specified by Threshold.

auto_scaling_policy.rules[].trigger.cloud_watch_alarm_definition.dimensions[]
Show child fields
auto_scaling_policy.rules[].trigger.cloud_watch_alarm_definition.dimensions[].key

The dimension name.

auto_scaling_policy.rules[].trigger.cloud_watch_alarm_definition.dimensions[].value

The dimension value.

auto_scaling_policy.rules[].trigger.cloud_watch_alarm_definition.evaluation_periods

The number of periods, in five-minute increments, during which the alarm condition must exist before the alarm triggers automatic scaling activity. The default value is 1.

auto_scaling_policy.rules[].trigger.cloud_watch_alarm_definition.metric_name

The name of the CloudWatch metric that is watched to determine an alarm condition.

auto_scaling_policy.rules[].trigger.cloud_watch_alarm_definition.namespace

The namespace for the CloudWatch metric. The default is AWS/ElasticMapReduce.

auto_scaling_policy.rules[].trigger.cloud_watch_alarm_definition.period

The period, in seconds, over which the statistic is applied. CloudWatch metrics for Amazon EMR are emitted every five minutes (300 seconds), so if you specify a CloudWatch metric, specify 300.

auto_scaling_policy.rules[].trigger.cloud_watch_alarm_definition.statistic

The statistic to apply to the metric associated with the alarm. The default is AVERAGE.

auto_scaling_policy.rules[].trigger.cloud_watch_alarm_definition.threshold

The value against which the specified statistic is compared.

auto_scaling_policy.rules[].trigger.cloud_watch_alarm_definition.unit

The unit of measure associated with the CloudWatch metric being watched. The value specified for Unit must correspond to the units specified in the CloudWatch metric.

auto_scaling_policy.status

The status of an automatic scaling policy.

Show child fields
auto_scaling_policy.status.state

Indicates the status of the automatic scaling policy.

auto_scaling_policy.status.state_change_reason

The reason for a change in status.

Show child fields
auto_scaling_policy.status.state_change_reason.code

The code indicating the reason for the change in status.USER_REQUEST indicates that the scaling policy status was changed by a user. PROVISION_FAILURE indicates that the status change was because the policy failed to provision. CLEANUP_FAILURE indicates an error.

auto_scaling_policy.status.state_change_reason.message

A friendly, more verbose message that accompanies an automatic scaling policy state change.

bid_price

If specified, indicates that the instance group uses Spot Instances. This is the maximum price you are willing to pay for Spot Instances. Specify OnDemandPrice to set the amount equal to the On-Demand price, or specify an amount in USD.

VARCHAR
configurations

Amazon EMR releases 4.x or later.

The list of configurations supplied for an Amazon EMR cluster instance group. You can specify a separate configuration for each instance group (master, core, and task).

STRUCT(
    "classification" VARCHAR,
    "properties" MAP(VARCHAR, VARCHAR)
)[]
Show child fields
configurations[]
Show child fields
configurations[].classification

The classification within a configuration.

configurations[].properties

A set of properties specified within a configuration classification.

configurations_version

The version number of the requested configuration specification for this instance group.

BIGINT
custom_ami_id

The custom AMI ID to use for the provisioned instance group.

VARCHAR
ebs_block_devices

The EBS block devices that are mapped to this instance group.

STRUCT(
    "volume_specification" STRUCT(
        "volume_type" VARCHAR,
        "iops" BIGINT,
        "size_in_gb" BIGINT,
        "throughput" BIGINT
    ),
    "device" VARCHAR
)[]
Show child fields
ebs_block_devices[]
Show child fields
ebs_block_devices[].device

The device name that is exposed to the instance, such as /dev/sdh.

ebs_block_devices[].volume_specification

EBS volume specifications such as volume type, IOPS, size (GiB) and throughput (MiB/s) that are requested for the EBS volume attached to an Amazon EC2 instance in the cluster.

Show child fields
ebs_block_devices[].volume_specification.iops

The number of I/O operations per second (IOPS) that the volume supports.

ebs_block_devices[].volume_specification.size_in_gb

The volume size, in gibibytes (GiB). This can be a number from 1 - 1024. If the volume type is EBS-optimized, the minimum value is 10.

ebs_block_devices[].volume_specification.throughput

The throughput, in mebibyte per second (MiB/s). This optional parameter can be a number from 125 - 1000 and is valid only for gp3 volumes.

ebs_block_devices[].volume_specification.volume_type

The volume type. Volume types supported are gp3, gp2, io1, st1, sc1, and standard.

ebs_optimized

If the instance group is EBS-optimized. An Amazon EBS-optimized instance uses an optimized configuration stack and provides additional, dedicated capacity for Amazon EBS I/O.

BOOLEAN
id

The identifier of the instance group.

VARCHAR
instance_group_type

The type of the instance group. Valid values are MASTER, CORE or TASK.

VARCHAR
instance_type

The Amazon EC2 instance type for all instances in the instance group.

VARCHAR
last_successfully_applied_configurations

A list of configurations that were successfully applied for an instance group last time.

STRUCT(
    "classification" VARCHAR,
    "properties" MAP(VARCHAR, VARCHAR)
)[]
Show child fields
last_successfully_applied_configurations[]
Show child fields
last_successfully_applied_configurations[].classification

The classification within a configuration.

last_successfully_applied_configurations[].properties

A set of properties specified within a configuration classification.

last_successfully_applied_configurations_version

The version number of a configuration specification that was successfully applied for an instance group last time. 

BIGINT
market

The marketplace to provision instances for this group. Valid values are ON_DEMAND or SPOT.

VARCHAR
name

The name of the instance group.

VARCHAR
requested_instance_count

The target number of instances for the instance group.

BIGINT
running_instance_count

The number of instances currently running in this instance group.

BIGINT
shrink_policy

Policy for customizing shrink operations.

STRUCT(
    "decommission_timeout" BIGINT,
    "instance_resize_policy" STRUCT(
        "instances_to_terminate" VARCHAR[],
        "instances_to_protect" VARCHAR[],
        "instance_termination_timeout" BIGINT
    )
)
Show child fields
shrink_policy.decommission_timeout

The desired timeout for decommissioning an instance. Overrides the default YARN decommissioning timeout.

shrink_policy.instance_resize_policy

Custom policy for requesting termination protection or termination of specific instances when shrinking an instance group.

Show child fields
shrink_policy.instance_resize_policy.instance_termination_timeout

Decommissioning timeout override for the specific list of instances to be terminated.

shrink_policy.instance_resize_policy.instances_to_protect[]
shrink_policy.instance_resize_policy.instances_to_terminate[]
status

The current status of the instance group.

STRUCT(
    "state" VARCHAR,
    "state_change_reason" STRUCT(
        "code" VARCHAR,
        "message" VARCHAR
    ),
    "timeline" STRUCT(
        "creation_date_time" TIMESTAMP_S,
        "ready_date_time" TIMESTAMP_S,
        "end_date_time" TIMESTAMP_S
    )
)
Show child fields
status.state

The current state of the instance group.

status.state_change_reason

The status change reason details for the instance group.

Show child fields
status.state_change_reason.code

The programmable code for the state change reason.

status.state_change_reason.message

The status change reason description.

status.timeline

The timeline of the instance group status over time.

Show child fields
status.timeline.creation_date_time

The creation date and time of the instance group.

status.timeline.end_date_time

The date and time when the instance group terminated.

status.timeline.ready_date_time

The date and time when the instance group became ready to perform tasks.