_aws_profile Input Column
The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role. | STRUCT(
"type" VARCHAR,
"name" VARCHAR,
"account_id" VARCHAR,
"via_profile_name" VARCHAR,
"assumed_role_arn" VARCHAR,
"organization" STRUCT(
"account_name" VARCHAR,
"id" VARCHAR,
"tags" STRUCT(
"key" VARCHAR,
"value" VARCHAR
)[],
"master_account" STRUCT(
"id" VARCHAR,
"email" VARCHAR
),
"parents" STRUCT(
"type" VARCHAR,
"id" VARCHAR,
"name" VARCHAR,
"tags" STRUCT(
"key" VARCHAR,
"value" VARCHAR
)[]
)[]
)
) |
job_run
The output displays information about the job run. | STRUCT(
"application_id" VARCHAR,
"job_run_id" VARCHAR,
"name" VARCHAR,
"arn" VARCHAR,
"created_by" VARCHAR,
"created_at" TIMESTAMP_S,
"updated_at" TIMESTAMP_S,
"execution_role" VARCHAR,
"state" VARCHAR,
"state_details" VARCHAR,
"release_label" VARCHAR,
"configuration_overrides" STRUCT(
"application_configuration" STRUCT(
"classification" VARCHAR,
"properties" MAP(VARCHAR, VARCHAR)
)[],
"monitoring_configuration" STRUCT(
"s3_monitoring_configuration" STRUCT(
"log_uri" VARCHAR,
"encryption_key_arn" VARCHAR
),
"managed_persistence_monitoring_configuration" STRUCT(
"enabled" BOOLEAN,
"encryption_key_arn" VARCHAR
),
"cloud_watch_logging_configuration" STRUCT(
"enabled" BOOLEAN,
"log_group_name" VARCHAR,
"log_stream_name_prefix" VARCHAR,
"encryption_key_arn" VARCHAR,
"log_types" MAP(VARCHAR, VARCHAR[])
),
"prometheus_monitoring_configuration" STRUCT(
"remote_write_url" VARCHAR
)
)
),
"job_driver" STRUCT(
"spark_submit" STRUCT(
"entry_point" VARCHAR,
"entry_point_arguments" VARCHAR[],
"spark_submit_parameters" VARCHAR
),
"hive" STRUCT(
"query" VARCHAR,
"init_query_file" VARCHAR,
"parameters" VARCHAR
)
),
"tags" MAP(VARCHAR, VARCHAR),
"total_resource_utilization" STRUCT(
"v_cpu_hour" DOUBLE,
"memory_gb_hour" DOUBLE,
"storage_gb_hour" DOUBLE
),
"network_configuration" STRUCT(
"subnet_ids" VARCHAR[],
"security_group_ids" VARCHAR[]
),
"total_execution_duration_seconds" BIGINT,
"execution_timeout_minutes" BIGINT,
"billed_resource_utilization" STRUCT(
"v_cpu_hour" DOUBLE,
"memory_gb_hour" DOUBLE,
"storage_gb_hour" DOUBLE
),
"mode" VARCHAR,
"retry_policy" STRUCT(
"max_attempts" BIGINT,
"max_failed_attempts_per_hour" BIGINT
),
"attempt" BIGINT,
"attempt_created_at" TIMESTAMP_S,
"attempt_updated_at" TIMESTAMP_S
) |
Show child fields- job_run.application_id
The ID of the application the job is running on. - job_run.arn
The execution role ARN of the job run. - job_run.attempt
The attempt of the job run. - job_run.attempt_created_at
The date and time of when the job run attempt was created. - job_run.attempt_updated_at
The date and time of when the job run attempt was last updated. - job_run.billed_resource_utilization
The aggregate vCPU, memory, and storage that Amazon Web Services has billed for the job run. The billed resources include a 1-minute minimum usage for workers, plus additional storage over 20 GB per worker. Note that billed resources do not include usage for idle pre-initialized workers. Show child fields- job_run.billed_resource_utilization.memory_gb_hour
The aggregated memory used per hour from the time the job starts executing until the job is terminated. - job_run.billed_resource_utilization.storage_gb_hour
The aggregated storage used per hour from the time the job starts executing until the job is terminated. - job_run.billed_resource_utilization.v_cpu_hour
The aggregated vCPU used per hour from the time the job starts executing until the job is terminated.
- job_run.configuration_overrides
The configuration settings that are used to override default configuration. Show child fields- job_run.configuration_overrides.application_configuration[]
Show child fields- job_run.configuration_overrides.application_configuration[].classification
The classification within a configuration. - job_run.configuration_overrides.application_configuration[].properties
A set of properties specified within a configuration classification.
- job_run.configuration_overrides.monitoring_configuration
The override configurations for monitoring. Show child fields- job_run.configuration_overrides.monitoring_configuration.cloud_watch_logging_configuration
The Amazon CloudWatch configuration for monitoring logs. You can configure your jobs to send log information to CloudWatch. Show child fields- job_run.configuration_overrides.monitoring_configuration.cloud_watch_logging_configuration.enabled
Enables CloudWatch logging. - job_run.configuration_overrides.monitoring_configuration.cloud_watch_logging_configuration.encryption_key_arn
The Key Management Service (KMS) key ARN to encrypt the logs that you store in CloudWatch Logs. - job_run.configuration_overrides.monitoring_configuration.cloud_watch_logging_configuration.log_group_name
The name of the log group in Amazon CloudWatch Logs where you want to publish your logs. - job_run.configuration_overrides.monitoring_configuration.cloud_watch_logging_configuration.log_stream_name_prefix
Prefix for the CloudWatch log stream name. - job_run.configuration_overrides.monitoring_configuration.cloud_watch_logging_configuration.log_types
The types of logs that you want to publish to CloudWatch. If you don't specify any log types, driver STDOUT and STDERR logs will be published to CloudWatch Logs by default. For more information including the supported worker types for Hive and Spark, see Logging for EMR Serverless with CloudWatch. -
Key Valid Values: SPARK_DRIVER, SPARK_EXECUTOR, HIVE_DRIVER, TEZ_TASK -
Array Members Valid Values: STDOUT, STDERR, HIVE_LOG, TEZ_AM, SYSTEM_LOGS
- job_run.configuration_overrides.monitoring_configuration.managed_persistence_monitoring_configuration
The managed log persistence configuration for a job run. Show child fields- job_run.configuration_overrides.monitoring_configuration.managed_persistence_monitoring_configuration.enabled
Enables managed logging and defaults to true. If set to false, managed logging will be turned off. - job_run.configuration_overrides.monitoring_configuration.managed_persistence_monitoring_configuration.encryption_key_arn
The KMS key ARN to encrypt the logs stored in managed log persistence.
- job_run.configuration_overrides.monitoring_configuration.prometheus_monitoring_configuration
The monitoring configuration object you can configure to send metrics to Amazon Managed Service for Prometheus for a job run. Show child fields- job_run.configuration_overrides.monitoring_configuration.prometheus_monitoring_configuration.remote_write_url
The remote write URL in the Amazon Managed Service for Prometheus workspace to send metrics to.
- job_run.configuration_overrides.monitoring_configuration.s3_monitoring_configuration
The Amazon S3 configuration for monitoring log publishing. Show child fields- job_run.configuration_overrides.monitoring_configuration.s3_monitoring_configuration.encryption_key_arn
The KMS key ARN to encrypt the logs published to the given Amazon S3 destination. - job_run.configuration_overrides.monitoring_configuration.s3_monitoring_configuration.log_uri
The Amazon S3 destination URI for log publishing.
- job_run.created_at
The date and time when the job run was created. - job_run.created_by
The user who created the job run. - job_run.execution_role
The execution role ARN of the job run. - job_run.execution_timeout_minutes
Returns the job run timeout value from the StartJobRun call. If no timeout was specified, then it returns the default timeout of 720 minutes. - job_run.job_driver
The job driver for the job run. Show child fields- job_run.job_driver.hive
The job driver parameters specified for Hive. Show child fields- job_run.job_driver.hive.init_query_file
The query file for the Hive job run. - job_run.job_driver.hive.parameters
The parameters for the Hive job run. - job_run.job_driver.hive.query
The query for the Hive job run.
- job_run.job_driver.spark_submit
The job driver parameters specified for Spark. Show child fields- job_run.job_driver.spark_submit.entry_point
The entry point for the Spark submit job run. - job_run.job_driver.spark_submit.entry_point_arguments[]
- job_run.job_driver.spark_submit.spark_submit_parameters
The parameters for the Spark submit job run.
- job_run.job_run_id
The ID of the job run. - job_run.mode
The mode of the job run. - job_run.name
The optional job run name. This doesn't have to be unique. - job_run.network_configuration
The network configuration for customer VPC connectivity. Show child fields- job_run.network_configuration.security_group_ids[]
- job_run.network_configuration.subnet_ids[]
- job_run.release_label
The Amazon EMR release associated with the application your job is running on. - job_run.retry_policy
The retry policy of the job run. Show child fields- job_run.retry_policy.max_attempts
Maximum number of attempts for the job run. This parameter is only applicable for BATCH mode. - job_run.retry_policy.max_failed_attempts_per_hour
Maximum number of failed attempts per hour. This [arameter is only applicable for STREAMING mode.
- job_run.state
The state of the job run. - job_run.state_details
The state details of the job run. - job_run.tags
The tags assigned to the job run. - job_run.total_execution_duration_seconds
The job run total execution duration in seconds. This field is only available for job runs in a COMPLETED, FAILED, or CANCELLED state. - job_run.total_resource_utilization
The aggregate vCPU, memory, and storage resources used from the time the job starts to execute, until the time the job terminates, rounded up to the nearest second. Show child fields- job_run.total_resource_utilization.memory_gb_hour
The aggregated memory used per hour from the time job start executing till the time job is terminated. - job_run.total_resource_utilization.storage_gb_hour
The aggregated storage used per hour from the time job start executing till the time job is terminated. - job_run.total_resource_utilization.v_cpu_hour
The aggregated vCPU used per hour from the time job start executing till the time job is terminated.
- job_run.updated_at
The date and time when the job run was updated.
|