Skip to content
AWS for Query.Farm

aws.databrew.describe_job

Example SQL Queries

SELECT * FROM
aws.databrew.describe_job
WHERE
"name" = 'VALUE';

Description

Returns the definition of a specific DataBrew job.

Table Definition

Column NameColumn Data Type
name Required Input Column

The name of the job.

VARCHAR
_aws_profile Input Column

The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role.

STRUCT(
    "type" VARCHAR,
    "name" VARCHAR,
    "account_id" VARCHAR,
    "via_profile_name" VARCHAR,
    "assumed_role_arn" VARCHAR,
    "organization" STRUCT(
        "account_name" VARCHAR,
        "id" VARCHAR,
        "tags" STRUCT(
            "key" VARCHAR,
            "value" VARCHAR
        )[],
        "master_account" STRUCT(
            "id" VARCHAR,
            "email" VARCHAR
        ),
        "parents" STRUCT(
            "type" VARCHAR,
            "id" VARCHAR,
            "name" VARCHAR,
            "tags" STRUCT(
                "key" VARCHAR,
                "value" VARCHAR
            )[]
        )[]
    )
)
Show child fields
_aws_profile.account_id

The AWS account id

_aws_profile.assumed_role_arn

The ARN of the assumed role

_aws_profile.name

The unique name of the profile.

_aws_profile.organization

Information about this profile's membership in the AWS organization.

Show child fields
_aws_profile.organization.account_name

The name of account speciifed by the organization

_aws_profile.organization.id

The organization id

_aws_profile.organization.master_account
Show child fields
_aws_profile.organization.master_account.email

The organization master account email address

_aws_profile.organization.master_account.id

The organization master account id

_aws_profile.organization.parents[]
Show child fields
_aws_profile.organization.parents[].id

The id of the parent

_aws_profile.organization.parents[].name

The name of the parent

_aws_profile.organization.parents[].tags[]
Show child fields
_aws_profile.organization.parents[].tags[].key
_aws_profile.organization.parents[].tags[].value
_aws_profile.organization.parents[].type

The type of parent can be an organization unit or a root

_aws_profile.organization.tags[]
Show child fields
_aws_profile.organization.tags[].key
_aws_profile.organization.tags[].value
_aws_profile.type

The type of profile, either 'credentials' or 'assumed_role'

_aws_profile.via_profile_name

This IAM role for this profile is assumed by first utilizing another profile with this name to obtain credentials.

_aws_region Input Column

The AWS region to use.

VARCHAR
create_date

The date and time that the job was created.

TIMESTAMP_S
created_by

The identifier (user name) of the user associated with the creation of the job.

VARCHAR
data_catalog_outputs

One or more artifacts that represent the Glue Data Catalog output from running the job.

STRUCT(
    "catalog_id" VARCHAR,
    "database_name" VARCHAR,
    "table_name" VARCHAR,
    "s3_options" STRUCT(
        "location" STRUCT(
            "bucket" VARCHAR,
            "key" VARCHAR,
            "bucket_owner" VARCHAR
        )
    ),
    "database_options" STRUCT(
        "temp_directory" STRUCT(
            "bucket" VARCHAR,
            "key" VARCHAR,
            "bucket_owner" VARCHAR
        ),
        "table_name" VARCHAR
    ),
    "overwrite" BOOLEAN
)[]
Show child fields
data_catalog_outputs[]
Show child fields
data_catalog_outputs[].catalog_id

The unique identifier of the Amazon Web Services account that holds the Data Catalog that stores the data.

data_catalog_outputs[].database_name

The name of a database in the Data Catalog.

data_catalog_outputs[].database_options

Represents options that specify how and where DataBrew writes the database output generated by recipe jobs.

Show child fields
data_catalog_outputs[].database_options.table_name

A prefix for the name of a table DataBrew will create in the database.

data_catalog_outputs[].database_options.temp_directory

Represents an Amazon S3 location (bucket name and object key) where DataBrew can store intermediate results.

Show child fields
data_catalog_outputs[].database_options.temp_directory.bucket

The Amazon S3 bucket name.

data_catalog_outputs[].database_options.temp_directory.bucket_owner

The Amazon Web Services account ID of the bucket owner.

data_catalog_outputs[].database_options.temp_directory.key

The unique name of the object in the bucket.

data_catalog_outputs[].overwrite

A value that, if true, means that any data in the location specified for output is overwritten with new output. Not supported with DatabaseOptions.

data_catalog_outputs[].s3_options

Represents options that specify how and where DataBrew writes the Amazon S3 output generated by recipe jobs.

Show child fields
data_catalog_outputs[].s3_options.location

Represents an Amazon S3 location (bucket name and object key) where DataBrew can write output from a job.

Show child fields
data_catalog_outputs[].s3_options.location.bucket

The Amazon S3 bucket name.

data_catalog_outputs[].s3_options.location.bucket_owner

The Amazon Web Services account ID of the bucket owner.

data_catalog_outputs[].s3_options.location.key

The unique name of the object in the bucket.

data_catalog_outputs[].table_name

The name of a table in the Data Catalog.

database_outputs

Represents a list of JDBC database output objects which defines the output destination for a DataBrew recipe job to write into.

STRUCT(
    "glue_connection_name" VARCHAR,
    "database_options" STRUCT(
        "temp_directory" STRUCT(
            "bucket" VARCHAR,
            "key" VARCHAR,
            "bucket_owner" VARCHAR
        ),
        "table_name" VARCHAR
    ),
    "database_output_mode" VARCHAR
)[]
Show child fields
database_outputs[]
Show child fields
database_outputs[].database_options

Represents options that specify how and where DataBrew writes the database output generated by recipe jobs.

Show child fields
database_outputs[].database_options.table_name

A prefix for the name of a table DataBrew will create in the database.

database_outputs[].database_options.temp_directory

Represents an Amazon S3 location (bucket name and object key) where DataBrew can store intermediate results.

Show child fields
database_outputs[].database_options.temp_directory.bucket

The Amazon S3 bucket name.

database_outputs[].database_options.temp_directory.bucket_owner

The Amazon Web Services account ID of the bucket owner.

database_outputs[].database_options.temp_directory.key

The unique name of the object in the bucket.

database_outputs[].database_output_mode

The output mode to write into the database. Currently supported option: NEW_TABLE.

database_outputs[].glue_connection_name

The Glue connection that stores the connection information for the target database.

dataset_name

The dataset that the job acts upon.

VARCHAR
encryption_key_arn

The Amazon Resource Name (ARN) of an encryption key that is used to protect the job.

VARCHAR
encryption_mode

The encryption mode for the job, which can be one of the following:

  • SSE-KMS - Server-side encryption with keys managed by KMS.

  • SSE-S3 - Server-side encryption with keys managed by Amazon S3.

VARCHAR
job_sample

Sample configuration for profile jobs only. Determines the number of rows on which the profile job will be executed.

STRUCT(
    "mode" VARCHAR,
    "size" BIGINT
)
Show child fields
job_sample.mode

A value that determines whether the profile job is run on the entire dataset or a specified number of rows. This value must be one of the following:

  • FULL_DATASET - The profile job is run on the entire dataset.

  • CUSTOM_ROWS - The profile job is run on the number of rows specified in the Size parameter.

job_sample.size

The Size parameter is only required when the mode is CUSTOM_ROWS. The profile job is run on the specified number of rows. The maximum value for size is Long.MAX_VALUE.

Long.MAX_VALUE = 9223372036854775807

last_modified_by

The identifier (user name) of the user who last modified the job.

VARCHAR
last_modified_date

The date and time that the job was last modified.

TIMESTAMP_S
log_subscription

Indicates whether Amazon CloudWatch logging is enabled for this job.

VARCHAR
max_capacity

The maximum number of compute nodes that DataBrew can consume when the job processes data.

BIGINT
max_retries

The maximum number of times to retry the job after a job run fails.

BIGINT
outputs

One or more artifacts that represent the output from running the job.

STRUCT(
    "compression_format" VARCHAR,
    "format" VARCHAR,
    "partition_columns" VARCHAR[],
    "location" STRUCT(
        "bucket" VARCHAR,
        "key" VARCHAR,
        "bucket_owner" VARCHAR
    ),
    "overwrite" BOOLEAN,
    "format_options" STRUCT(
        "csv" STRUCT(
            "delimiter" VARCHAR
        )
    ),
    "max_output_files" BIGINT
)[]
Show child fields
outputs[]
Show child fields
outputs[].compression_format

The compression algorithm used to compress the output text of the job.

outputs[].format

The data format of the output of the job.

outputs[].format_options

Represents options that define how DataBrew formats job output files.

Show child fields
outputs[].format_options.csv

Represents a set of options that define the structure of comma-separated value (CSV) job output.

Show child fields
outputs[].format_options.csv.delimiter

A single character that specifies the delimiter used to create CSV job output.

outputs[].location

The location in Amazon S3 where the job writes its output.

Show child fields
outputs[].location.bucket

The Amazon S3 bucket name.

outputs[].location.bucket_owner

The Amazon Web Services account ID of the bucket owner.

outputs[].location.key

The unique name of the object in the bucket.

outputs[].max_output_files

Maximum number of files to be generated by the job and written to the output folder. For output partitioned by column(s), the MaxOutputFiles value is the maximum number of files per partition.

outputs[].overwrite

A value that, if true, means that any data in the location specified for output is overwritten with new output.

outputs[].partition_columns[]
profile_configuration

Configuration for profile jobs. Used to select columns, do evaluations, and override default parameters of evaluations. When configuration is null, the profile job will run with default settings.

STRUCT(
    "dataset_statistics_configuration" STRUCT(
        "included_statistics" VARCHAR[],
        "overrides" STRUCT(
            "statistic" VARCHAR,
            "parameters" MAP(VARCHAR, VARCHAR)
        )[]
    ),
    "profile_columns" STRUCT(
        "regex" VARCHAR,
        "name" VARCHAR
    )[],
    "column_statistics_configurations" STRUCT(
        "selectors" STRUCT(
            "regex" VARCHAR,
            "name" VARCHAR
        )[],
        "statistics" STRUCT(
            "included_statistics" VARCHAR[],
            "overrides" STRUCT(
                "statistic" VARCHAR,
                "parameters" MAP(VARCHAR, VARCHAR)
            )[]
        )
    )[],
    "entity_detector_configuration" STRUCT(
        "entity_types" VARCHAR[],
        "allowed_statistics" STRUCT(
            "statistics" VARCHAR[]
        )[]
    )
)
Show child fields
profile_configuration.column_statistics_configurations[]
Show child fields
profile_configuration.column_statistics_configurations[].selectors[]
Show child fields
profile_configuration.column_statistics_configurations[].selectors[].name

The name of a column from a dataset.

profile_configuration.column_statistics_configurations[].selectors[].regex

A regular expression for selecting a column from a dataset.

profile_configuration.column_statistics_configurations[].statistics

Configuration for evaluations. Statistics can be used to select evaluations and override parameters of evaluations.

Show child fields
profile_configuration.column_statistics_configurations[].statistics.included_statistics[]
profile_configuration.column_statistics_configurations[].statistics.overrides[]
Show child fields
profile_configuration.column_statistics_configurations[].statistics.overrides[].parameters

A map that includes overrides of an evaluation’s parameters.

profile_configuration.column_statistics_configurations[].statistics.overrides[].statistic

The name of an evaluation

profile_configuration.dataset_statistics_configuration

Configuration for inter-column evaluations. Configuration can be used to select evaluations and override parameters of evaluations. When configuration is undefined, the profile job will run all supported inter-column evaluations.

Show child fields
profile_configuration.dataset_statistics_configuration.included_statistics[]
profile_configuration.dataset_statistics_configuration.overrides[]
Show child fields
profile_configuration.dataset_statistics_configuration.overrides[].parameters

A map that includes overrides of an evaluation’s parameters.

profile_configuration.dataset_statistics_configuration.overrides[].statistic

The name of an evaluation

profile_configuration.entity_detector_configuration

Configuration of entity detection for a profile job. When undefined, entity detection is disabled.

Show child fields
profile_configuration.entity_detector_configuration.allowed_statistics[]
Show child fields
profile_configuration.entity_detector_configuration.allowed_statistics[].statistics[]
profile_configuration.entity_detector_configuration.entity_types[]
profile_configuration.profile_columns[]
Show child fields
profile_configuration.profile_columns[].name

The name of a column from a dataset.

profile_configuration.profile_columns[].regex

A regular expression for selecting a column from a dataset.

project_name

The DataBrew project associated with this job.

VARCHAR
recipe_reference

Represents the name and version of a DataBrew recipe.

STRUCT(
    "name" VARCHAR,
    "recipe_version" VARCHAR
)
Show child fields
recipe_reference.name

The name of the recipe.

recipe_reference.recipe_version

The identifier for the version for the recipe.

resource_arn

The Amazon Resource Name (ARN) of the job.

VARCHAR
role_arn

The ARN of the Identity and Access Management (IAM) role to be assumed when DataBrew runs the job.

VARCHAR
tags

Metadata tags associated with this job.

MAP(VARCHAR, VARCHAR)
timeout

The job's timeout in minutes. A job that attempts to run longer than this timeout period ends with a status of TIMEOUT.

BIGINT
type

The job type, which must be one of the following:

  • PROFILE - The job analyzes the dataset to determine its size, data types, data distribution, and more.

  • RECIPE - The job applies one or more transformations to a dataset.

VARCHAR
validation_configurations

List of validation configurations that are applied to the profile job.

STRUCT(
    "ruleset_arn" VARCHAR,
    "validation_mode" VARCHAR
)[]
Show child fields
validation_configurations[]
Show child fields
validation_configurations[].ruleset_arn

The Amazon Resource Name (ARN) for the ruleset to be validated in the profile job. The TargetArn of the selected ruleset should be the same as the Amazon Resource Name (ARN) of the dataset that is associated with the profile job.

validation_configurations[].validation_mode

Mode of data quality validation. Default mode is “CHECK_ALL” which verifies all rules defined in the selected ruleset.