_aws_profile Input Column
The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role. | STRUCT(
"type" VARCHAR,
"name" VARCHAR,
"account_id" VARCHAR,
"via_profile_name" VARCHAR,
"assumed_role_arn" VARCHAR,
"organization" STRUCT(
"account_name" VARCHAR,
"id" VARCHAR,
"tags" STRUCT(
"key" VARCHAR,
"value" VARCHAR
)[],
"master_account" STRUCT(
"id" VARCHAR,
"email" VARCHAR
),
"parents" STRUCT(
"type" VARCHAR,
"id" VARCHAR,
"name" VARCHAR,
"tags" STRUCT(
"key" VARCHAR,
"value" VARCHAR
)[]
)[]
)
) |
job_flows
A list of job flows matching the parameters supplied. | STRUCT(
"job_flow_id" VARCHAR,
"name" VARCHAR,
"log_uri" VARCHAR,
"log_encryption_kms_key_id" VARCHAR,
"ami_version" VARCHAR,
"execution_status_detail" STRUCT(
"state" VARCHAR,
"creation_date_time" TIMESTAMP_S,
"start_date_time" TIMESTAMP_S,
"ready_date_time" TIMESTAMP_S,
"end_date_time" TIMESTAMP_S,
"last_state_change_reason" VARCHAR
),
"instances" STRUCT(
"master_instance_type" VARCHAR,
"master_public_dns_name" VARCHAR,
"master_instance_id" VARCHAR,
"slave_instance_type" VARCHAR,
"instance_count" BIGINT,
"instance_groups" STRUCT(
"instance_group_id" VARCHAR,
"name" VARCHAR,
"market" VARCHAR,
"instance_role" VARCHAR,
"bid_price" VARCHAR,
"instance_type" VARCHAR,
"instance_request_count" BIGINT,
"instance_running_count" BIGINT,
"state" VARCHAR,
"last_state_change_reason" VARCHAR,
"creation_date_time" TIMESTAMP_S,
"start_date_time" TIMESTAMP_S,
"ready_date_time" TIMESTAMP_S,
"end_date_time" TIMESTAMP_S,
"custom_ami_id" VARCHAR
)[],
"normalized_instance_hours" BIGINT,
"ec2_key_name" VARCHAR,
"ec2_subnet_id" VARCHAR,
"placement" STRUCT(
"availability_zone" VARCHAR,
"availability_zones" VARCHAR[]
),
"keep_job_flow_alive_when_no_steps" BOOLEAN,
"termination_protected" BOOLEAN,
"unhealthy_node_replacement" BOOLEAN,
"hadoop_version" VARCHAR
),
"steps" STRUCT(
"step_config" STRUCT(
"name" VARCHAR,
"action_on_failure" VARCHAR,
"hadoop_jar_step" STRUCT(
"properties" STRUCT(
"key" VARCHAR,
"value" VARCHAR
)[],
"jar" VARCHAR,
"main_class" VARCHAR,
"args" VARCHAR[]
)
),
"execution_status_detail" STRUCT(
"state" VARCHAR,
"creation_date_time" TIMESTAMP_S,
"start_date_time" TIMESTAMP_S,
"end_date_time" TIMESTAMP_S,
"last_state_change_reason" VARCHAR
)
)[],
"bootstrap_actions" STRUCT(
"bootstrap_action_config" STRUCT(
"name" VARCHAR,
"script_bootstrap_action" STRUCT(
"path" VARCHAR,
"args" VARCHAR[]
)
)
)[],
"supported_products" VARCHAR[],
"visible_to_all_users" BOOLEAN,
"job_flow_role" VARCHAR,
"service_role" VARCHAR,
"auto_scaling_role" VARCHAR,
"scale_down_behavior" VARCHAR
)[] |
Show child fields- job_flows[]
Show child fields- job_flows[].ami_version
Applies only to Amazon EMR AMI versions 3.x and 2.x. For Amazon EMR releases 4.0 and later, ReleaseLabel is used. To specify a custom AMI, use CustomAmiID. - job_flows[].auto_scaling_role
An IAM role for automatic scaling policies. The default role is EMR_AutoScaling_DefaultRole. The IAM role provides a way for the automatic scaling feature to get the required permissions it needs to launch and terminate Amazon EC2 instances in an instance group. - job_flows[].bootstrap_actions[]
Show child fields- job_flows[].bootstrap_actions[].bootstrap_action_config
A description of the bootstrap action. Show child fields- job_flows[].bootstrap_actions[].bootstrap_action_config.name
The name of the bootstrap action. - job_flows[].bootstrap_actions[].bootstrap_action_config.script_bootstrap_action
The script run by the bootstrap action. Show child fields- job_flows[].bootstrap_actions[].bootstrap_action_config.script_bootstrap_action.args[]
- job_flows[].bootstrap_actions[].bootstrap_action_config.script_bootstrap_action.path
Location in Amazon S3 of the script to run during a bootstrap action.
- job_flows[].execution_status_detail
Describes the execution status of the job flow. Show child fields- job_flows[].execution_status_detail.creation_date_time
The creation date and time of the job flow. - job_flows[].execution_status_detail.end_date_time
The completion date and time of the job flow. - job_flows[].execution_status_detail.last_state_change_reason
Description of the job flow last changed state. - job_flows[].execution_status_detail.ready_date_time
The date and time when the job flow was ready to start running bootstrap actions. - job_flows[].execution_status_detail.start_date_time
The start date and time of the job flow. - job_flows[].execution_status_detail.state
The state of the job flow.
- job_flows[].instances
Describes the Amazon EC2 instances of the job flow. Show child fields- job_flows[].instances.ec2_key_name
The name of an Amazon EC2 key pair that can be used to connect to the master node using SSH. - job_flows[].instances.ec2_subnet_id
For clusters launched within Amazon Virtual Private Cloud, this is the identifier of the subnet where the cluster was launched. - job_flows[].instances.hadoop_version
The Hadoop version for the cluster. - job_flows[].instances.instance_count
The number of Amazon EC2 instances in the cluster. If the value is 1, the same instance serves as both the master and core and task node. If the value is greater than 1, one instance is the master node and all others are core and task nodes. - job_flows[].instances.instance_groups[]
Show child fields- job_flows[].instances.instance_groups[].bid_price
If specified, indicates that the instance group uses Spot Instances. This is the maximum price you are willing to pay for Spot Instances. Specify OnDemandPrice to set the amount equal to the On-Demand price, or specify an amount in USD. - job_flows[].instances.instance_groups[].creation_date_time
The date/time the instance group was created. - job_flows[].instances.instance_groups[].custom_ami_id
The custom AMI ID to use for the provisioned instance group. - job_flows[].instances.instance_groups[].end_date_time
The date/time the instance group was terminated. - job_flows[].instances.instance_groups[].instance_group_id
Unique identifier for the instance group. - job_flows[].instances.instance_groups[].instance_request_count
Target number of instances to run in the instance group. - job_flows[].instances.instance_groups[].instance_role
Instance group role in the cluster - job_flows[].instances.instance_groups[].instance_running_count
Actual count of running instances. - job_flows[].instances.instance_groups[].instance_type
Amazon EC2 instance type. - job_flows[].instances.instance_groups[].last_state_change_reason
Details regarding the state of the instance group. - job_flows[].instances.instance_groups[].market
Market type of the Amazon EC2 instances used to create a cluster node. - job_flows[].instances.instance_groups[].name
Friendly name for the instance group. - job_flows[].instances.instance_groups[].ready_date_time
The date/time the instance group was available to the cluster. - job_flows[].instances.instance_groups[].start_date_time
The date/time the instance group was started. - job_flows[].instances.instance_groups[].state
State of instance group. The following values are no longer supported: STARTING, TERMINATED, and FAILED.
- job_flows[].instances.keep_job_flow_alive_when_no_steps
Specifies whether the cluster should remain available after completing all steps. - job_flows[].instances.master_instance_id
The Amazon EC2 instance identifier of the master node. - job_flows[].instances.master_instance_type
The Amazon EC2 master node instance type. - job_flows[].instances.master_public_dns_name
The DNS name of the master node. If the cluster is on a private subnet, this is the private DNS name. On a public subnet, this is the public DNS name. - job_flows[].instances.normalized_instance_hours
An approximation of the cost of the cluster, represented in m1.small/hours. This value is increased one time for every hour that an m1.small instance runs. Larger instances are weighted more heavily, so an Amazon EC2 instance that is roughly four times more expensive would result in the normalized instance hours being increased incrementally four times. This result is only an approximation and does not reflect the actual billing rate. - job_flows[].instances.placement
The Amazon EC2 Availability Zone for the cluster. Show child fields- job_flows[].instances.placement.availability_zone
The Amazon EC2 Availability Zone for the cluster. AvailabilityZone is used for uniform instance groups, while AvailabilityZones (plural) is used for instance fleets. - job_flows[].instances.placement.availability_zones[]
- job_flows[].instances.slave_instance_type
The Amazon EC2 core and task node instance type. - job_flows[].instances.termination_protected
Specifies whether the Amazon EC2 instances in the cluster are protected from termination by API calls, user intervention, or in the event of a job-flow error. - job_flows[].instances.unhealthy_node_replacement
Indicates whether Amazon EMR should gracefully replace core nodes that have degraded within the cluster.
- job_flows[].job_flow_id
The job flow identifier. - job_flows[].job_flow_role
The IAM role that was specified when the job flow was launched. The Amazon EC2 instances of the job flow assume this role. - job_flows[].log_encryption_kms_key_id
The KMS key used for encrypting log files. This attribute is only available with Amazon EMR 5.30.0 and later, excluding 6.0.0. - job_flows[].log_uri
The location in Amazon S3 where log files for the job are stored. - job_flows[].name
The name of the job flow. - job_flows[].scale_down_behavior
The way that individual Amazon EC2 instances terminate when an automatic scale-in activity occurs or an instance group is resized. TERMINATE_AT_INSTANCE_HOUR indicates that Amazon EMR terminates nodes at the instance-hour boundary, regardless of when the request to terminate the instance was submitted. This option is only available with Amazon EMR 5.1.0 and later and is the default for clusters created using that version. TERMINATE_AT_TASK_COMPLETION indicates that Amazon EMR adds nodes to a deny list and drains tasks from nodes before terminating the Amazon EC2 instances, regardless of the instance-hour boundary. With either behavior, Amazon EMR removes the least active nodes first and blocks instance termination if it could lead to HDFS corruption. TERMINATE_AT_TASK_COMPLETION available only in Amazon EMR releases 4.1.0 and later, and is the default for releases of Amazon EMR earlier than 5.1.0. - job_flows[].service_role
The IAM role that is assumed by the Amazon EMR service to access Amazon Web Services resources on your behalf. - job_flows[].steps[]
Show child fields- job_flows[].steps[].execution_status_detail
The description of the step status. Show child fields- job_flows[].steps[].execution_status_detail.creation_date_time
The creation date and time of the step. - job_flows[].steps[].execution_status_detail.end_date_time
The completion date and time of the step. - job_flows[].steps[].execution_status_detail.last_state_change_reason
A description of the step's current state. - job_flows[].steps[].execution_status_detail.start_date_time
The start date and time of the step. - job_flows[].steps[].execution_status_detail.state
The state of the step.
- job_flows[].steps[].step_config
The step configuration. Show child fields- job_flows[].steps[].step_config.action_on_failure
The action to take when the step fails. Use one of the following values: -
TERMINATE_CLUSTER - Shuts down the cluster. -
CANCEL_AND_WAIT - Cancels any pending steps and returns the cluster to the WAITING state. -
CONTINUE - Continues to the next step in the queue. -
TERMINATE_JOB_FLOW - Shuts down the cluster. TERMINATE_JOB_FLOW is provided for backward compatibility. We recommend using TERMINATE_CLUSTER instead. If a cluster's StepConcurrencyLevel is greater than 1, do not use AddJobFlowSteps to submit a step with this parameter set to CANCEL_AND_WAIT or TERMINATE_CLUSTER. The step is not submitted and the action fails with a message that the ActionOnFailure setting is not valid. If you change a cluster's StepConcurrencyLevel to be greater than 1 while a step is running, the ActionOnFailure parameter may not behave as you expect. In this case, for a step that fails with this parameter set to CANCEL_AND_WAIT, pending steps and the running step are not canceled; for a step that fails with this parameter set to TERMINATE_CLUSTER, the cluster does not terminate. - job_flows[].steps[].step_config.hadoop_jar_step
The JAR file used for the step. Show child fields- job_flows[].steps[].step_config.hadoop_jar_step.args[]
- job_flows[].steps[].step_config.hadoop_jar_step.jar
A path to a JAR file run during the step. - job_flows[].steps[].step_config.hadoop_jar_step.main_class
The name of the main class in the specified Java file. If not specified, the JAR file should specify a Main-Class in its manifest file. - job_flows[].steps[].step_config.hadoop_jar_step.properties[]
Show child fields- job_flows[].steps[].step_config.hadoop_jar_step.properties[].key
The unique identifier of a key-value pair. - job_flows[].steps[].step_config.hadoop_jar_step.properties[].value
The value part of the identified key.
- job_flows[].steps[].step_config.name
The name of the step.
- job_flows[].supported_products[]
- job_flows[].visible_to_all_users
Indicates whether the cluster is visible to IAM principals in the Amazon Web Services account associated with the cluster. When true, IAM principals in the Amazon Web Services account can perform Amazon EMR cluster actions that their IAM policies allow. When false, only the IAM principal that created the cluster and the Amazon Web Services account root user can perform Amazon EMR actions, regardless of IAM permissions policies attached to other IAM principals. The default value is true if a value is not provided when creating a cluster using the Amazon EMR API RunJobFlow command, the CLI create-cluster command, or the Amazon Web Services Management Console.
|