Skip to content
AWS for Query.Farm

aws.databrew.list_jobs

Example SQL Queries

SELECT * FROM
aws.databrew.list_jobs;

Description

Lists all of the DataBrew jobs that are defined.

Table Definition

Column NameColumn Data Type
dataset_name Input Column

A dataset that the job is to process.

VARCHAR
project_name Input Column

The name of the project that the job is associated with.

VARCHAR
_aws_profile Input Column

The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role.

STRUCT(
    "type" VARCHAR,
    "name" VARCHAR,
    "account_id" VARCHAR,
    "via_profile_name" VARCHAR,
    "assumed_role_arn" VARCHAR,
    "organization" STRUCT(
        "account_name" VARCHAR,
        "id" VARCHAR,
        "tags" STRUCT(
            "key" VARCHAR,
            "value" VARCHAR
        )[],
        "master_account" STRUCT(
            "id" VARCHAR,
            "email" VARCHAR
        ),
        "parents" STRUCT(
            "type" VARCHAR,
            "id" VARCHAR,
            "name" VARCHAR,
            "tags" STRUCT(
                "key" VARCHAR,
                "value" VARCHAR
            )[]
        )[]
    )
)
Show child fields
_aws_profile.account_id

The AWS account id

_aws_profile.assumed_role_arn

The ARN of the assumed role

_aws_profile.name

The unique name of the profile.

_aws_profile.organization

Information about this profile's membership in the AWS organization.

Show child fields
_aws_profile.organization.account_name

The name of account speciifed by the organization

_aws_profile.organization.id

The organization id

_aws_profile.organization.master_account
Show child fields
_aws_profile.organization.master_account.email

The organization master account email address

_aws_profile.organization.master_account.id

The organization master account id

_aws_profile.organization.parents[]
Show child fields
_aws_profile.organization.parents[].id

The id of the parent

_aws_profile.organization.parents[].name

The name of the parent

_aws_profile.organization.parents[].tags[]
Show child fields
_aws_profile.organization.parents[].tags[].key
_aws_profile.organization.parents[].tags[].value
_aws_profile.organization.parents[].type

The type of parent can be an organization unit or a root

_aws_profile.organization.tags[]
Show child fields
_aws_profile.organization.tags[].key
_aws_profile.organization.tags[].value
_aws_profile.type

The type of profile, either 'credentials' or 'assumed_role'

_aws_profile.via_profile_name

This IAM role for this profile is assumed by first utilizing another profile with this name to obtain credentials.

_aws_region Input Column

The AWS region to use.

VARCHAR
account_id

The ID of the Amazon Web Services account that owns the job.

VARCHAR
create_date

The date and time that the job was created.

TIMESTAMP_S
created_by

The Amazon Resource Name (ARN) of the user who created the job.

VARCHAR
data_catalog_outputs

One or more artifacts that represent the Glue Data Catalog output from running the job.

STRUCT(
    "catalog_id" VARCHAR,
    "database_name" VARCHAR,
    "table_name" VARCHAR,
    "s3_options" STRUCT(
        "location" STRUCT(
            "bucket" VARCHAR,
            "key" VARCHAR,
            "bucket_owner" VARCHAR
        )
    ),
    "database_options" STRUCT(
        "temp_directory" STRUCT(
            "bucket" VARCHAR,
            "key" VARCHAR,
            "bucket_owner" VARCHAR
        ),
        "table_name" VARCHAR
    ),
    "overwrite" BOOLEAN
)[]
Show child fields
data_catalog_outputs[]
Show child fields
data_catalog_outputs[].catalog_id

The unique identifier of the Amazon Web Services account that holds the Data Catalog that stores the data.

data_catalog_outputs[].database_name

The name of a database in the Data Catalog.

data_catalog_outputs[].database_options

Represents options that specify how and where DataBrew writes the database output generated by recipe jobs.

Show child fields
data_catalog_outputs[].database_options.table_name

A prefix for the name of a table DataBrew will create in the database.

data_catalog_outputs[].database_options.temp_directory

Represents an Amazon S3 location (bucket name and object key) where DataBrew can store intermediate results.

Show child fields
data_catalog_outputs[].database_options.temp_directory.bucket

The Amazon S3 bucket name.

data_catalog_outputs[].database_options.temp_directory.bucket_owner

The Amazon Web Services account ID of the bucket owner.

data_catalog_outputs[].database_options.temp_directory.key

The unique name of the object in the bucket.

data_catalog_outputs[].overwrite

A value that, if true, means that any data in the location specified for output is overwritten with new output. Not supported with DatabaseOptions.

data_catalog_outputs[].s3_options

Represents options that specify how and where DataBrew writes the Amazon S3 output generated by recipe jobs.

Show child fields
data_catalog_outputs[].s3_options.location

Represents an Amazon S3 location (bucket name and object key) where DataBrew can write output from a job.

Show child fields
data_catalog_outputs[].s3_options.location.bucket

The Amazon S3 bucket name.

data_catalog_outputs[].s3_options.location.bucket_owner

The Amazon Web Services account ID of the bucket owner.

data_catalog_outputs[].s3_options.location.key

The unique name of the object in the bucket.

data_catalog_outputs[].table_name

The name of a table in the Data Catalog.

database_outputs

Represents a list of JDBC database output objects which defines the output destination for a DataBrew recipe job to write into.

STRUCT(
    "glue_connection_name" VARCHAR,
    "database_options" STRUCT(
        "temp_directory" STRUCT(
            "bucket" VARCHAR,
            "key" VARCHAR,
            "bucket_owner" VARCHAR
        ),
        "table_name" VARCHAR
    ),
    "database_output_mode" VARCHAR
)[]
Show child fields
database_outputs[]
Show child fields
database_outputs[].database_options

Represents options that specify how and where DataBrew writes the database output generated by recipe jobs.

Show child fields
database_outputs[].database_options.table_name

A prefix for the name of a table DataBrew will create in the database.

database_outputs[].database_options.temp_directory

Represents an Amazon S3 location (bucket name and object key) where DataBrew can store intermediate results.

Show child fields
database_outputs[].database_options.temp_directory.bucket

The Amazon S3 bucket name.

database_outputs[].database_options.temp_directory.bucket_owner

The Amazon Web Services account ID of the bucket owner.

database_outputs[].database_options.temp_directory.key

The unique name of the object in the bucket.

database_outputs[].database_output_mode

The output mode to write into the database. Currently supported option: NEW_TABLE.

database_outputs[].glue_connection_name

The Glue connection that stores the connection information for the target database.

encryption_key_arn

The Amazon Resource Name (ARN) of an encryption key that is used to protect the job output. For more information, see Encrypting data written by DataBrew jobs 

VARCHAR
encryption_mode

The encryption mode for the job, which can be one of the following:

  • SSE-KMS - Server-side encryption with keys managed by KMS.

  • SSE-S3 - Server-side encryption with keys managed by Amazon S3.

VARCHAR
job_sample

A sample configuration for profile jobs only, which determines the number of rows on which the profile job is run. If a JobSample value isn't provided, the default value is used. The default value is CUSTOM_ROWS for the mode parameter and 20,000 for the size parameter.

STRUCT(
    "mode" VARCHAR,
    "size" BIGINT
)
Show child fields
job_sample.mode

A value that determines whether the profile job is run on the entire dataset or a specified number of rows. This value must be one of the following:

  • FULL_DATASET - The profile job is run on the entire dataset.

  • CUSTOM_ROWS - The profile job is run on the number of rows specified in the Size parameter.

job_sample.size

The Size parameter is only required when the mode is CUSTOM_ROWS. The profile job is run on the specified number of rows. The maximum value for size is Long.MAX_VALUE.

Long.MAX_VALUE = 9223372036854775807

last_modified_by

The Amazon Resource Name (ARN) of the user who last modified the job.

VARCHAR
last_modified_date

The modification date and time of the job.

TIMESTAMP_S
log_subscription

The current status of Amazon CloudWatch logging for the job.

VARCHAR
max_capacity

The maximum number of nodes that can be consumed when the job processes data.

BIGINT
max_retries

The maximum number of times to retry the job after a job run fails.

BIGINT
name

The unique name of the job.

VARCHAR
outputs

One or more artifacts that represent output from running the job.

STRUCT(
    "compression_format" VARCHAR,
    "format" VARCHAR,
    "partition_columns" VARCHAR[],
    "location" STRUCT(
        "bucket" VARCHAR,
        "key" VARCHAR,
        "bucket_owner" VARCHAR
    ),
    "overwrite" BOOLEAN,
    "format_options" STRUCT(
        "csv" STRUCT(
            "delimiter" VARCHAR
        )
    ),
    "max_output_files" BIGINT
)[]
Show child fields
outputs[]
Show child fields
outputs[].compression_format

The compression algorithm used to compress the output text of the job.

outputs[].format

The data format of the output of the job.

outputs[].format_options

Represents options that define how DataBrew formats job output files.

Show child fields
outputs[].format_options.csv

Represents a set of options that define the structure of comma-separated value (CSV) job output.

Show child fields
outputs[].format_options.csv.delimiter

A single character that specifies the delimiter used to create CSV job output.

outputs[].location

The location in Amazon S3 where the job writes its output.

Show child fields
outputs[].location.bucket

The Amazon S3 bucket name.

outputs[].location.bucket_owner

The Amazon Web Services account ID of the bucket owner.

outputs[].location.key

The unique name of the object in the bucket.

outputs[].max_output_files

Maximum number of files to be generated by the job and written to the output folder. For output partitioned by column(s), the MaxOutputFiles value is the maximum number of files per partition.

outputs[].overwrite

A value that, if true, means that any data in the location specified for output is overwritten with new output.

outputs[].partition_columns[]
recipe_reference

A set of steps that the job runs.

STRUCT(
    "name" VARCHAR,
    "recipe_version" VARCHAR
)
Show child fields
recipe_reference.name

The name of the recipe.

recipe_reference.recipe_version

The identifier for the version for the recipe.

resource_arn

The unique Amazon Resource Name (ARN) for the job.

VARCHAR
role_arn

The Amazon Resource Name (ARN) of the role to be assumed for this job.

VARCHAR
tags

Metadata tags that have been applied to the job.

MAP(VARCHAR, VARCHAR)
timeout

The job's timeout in minutes. A job that attempts to run longer than this timeout period ends with a status of TIMEOUT.

BIGINT
type

The job type of the job, which must be one of the following:

  • PROFILE - A job to analyze a dataset, to determine its size, data types, data distribution, and more.

  • RECIPE - A job to apply one or more transformations to a dataset.

VARCHAR
validation_configurations

List of validation configurations that are applied to the profile job.

STRUCT(
    "ruleset_arn" VARCHAR,
    "validation_mode" VARCHAR
)[]
Show child fields
validation_configurations[]
Show child fields
validation_configurations[].ruleset_arn

The Amazon Resource Name (ARN) for the ruleset to be validated in the profile job. The TargetArn of the selected ruleset should be the same as the Amazon Resource Name (ARN) of the dataset that is associated with the profile job.

validation_configurations[].validation_mode

Mode of data quality validation. Default mode is “CHECK_ALL” which verifies all rules defined in the selected ruleset.