| Column Name | Column Data Type |
|---|
_aws_profile Input Column
The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role. | STRUCT(
"type" VARCHAR,
"name" VARCHAR,
"account_id" VARCHAR,
"via_profile_name" VARCHAR,
"assumed_role_arn" VARCHAR,
"organization" STRUCT(
"account_name" VARCHAR,
"id" VARCHAR,
"tags" STRUCT(
"key" VARCHAR,
"value" VARCHAR
)[],
"master_account" STRUCT(
"id" VARCHAR,
"email" VARCHAR
),
"parents" STRUCT(
"type" VARCHAR,
"id" VARCHAR,
"name" VARCHAR,
"tags" STRUCT(
"key" VARCHAR,
"value" VARCHAR
)[]
)[]
)
) |
Show child fields- _aws_profile.account_id
The AWS account id - _aws_profile.assumed_role_arn
The ARN of the assumed role - _aws_profile.name
The unique name of the profile. - _aws_profile.organization
Information about this profile's membership in the AWS organization. Show child fields- _aws_profile.organization.account_name
The name of account speciifed by the organization - _aws_profile.organization.id
The organization id - _aws_profile.organization.master_account
Show child fields- _aws_profile.organization.master_account.email
The organization master account email address - _aws_profile.organization.master_account.id
The organization master account id
- _aws_profile.organization.parents[]
Show child fields- _aws_profile.organization.parents[].id
The id of the parent - _aws_profile.organization.parents[].name
The name of the parent - _aws_profile.organization.parents[].tags[]
Show child fields- _aws_profile.organization.parents[].tags[].key
- _aws_profile.organization.parents[].tags[].value
- _aws_profile.organization.parents[].type
The type of parent can be an organization unit or a root
- _aws_profile.organization.tags[]
Show child fields- _aws_profile.organization.tags[].key
- _aws_profile.organization.tags[].value
- _aws_profile.type
The type of profile, either 'credentials' or 'assumed_role' - _aws_profile.via_profile_name
This IAM role for this profile is assumed by first utilizing another profile with this name to obtain credentials.
|
_aws_region Input Column
The AWS region to use. | VARCHAR |
account_id
The ID of the Amazon Web Services account that owns the dataset. | VARCHAR |
create_date
The date and time that the dataset was created. | TIMESTAMP_S |
created_by
The Amazon Resource Name (ARN) of the user who created the dataset. | VARCHAR |
format
The file format of a dataset that is created from an Amazon S3 file or folder. | VARCHAR |
format_options
A set of options that define how DataBrew interprets the data in the dataset. | STRUCT(
"json" STRUCT(
"multi_line" BOOLEAN
),
"excel" STRUCT(
"sheet_names" VARCHAR[],
"sheet_indexes" BIGINT[],
"header_row" BOOLEAN
),
"csv" STRUCT(
"delimiter" VARCHAR,
"header_row" BOOLEAN
)
) |
Show child fields- format_options.csv
Options that define how CSV input is to be interpreted by DataBrew. Show child fields- format_options.csv.delimiter
A single character that specifies the delimiter being used in the CSV file. - format_options.csv.header_row
A variable that specifies whether the first row in the file is parsed as the header. If this value is false, column names are auto-generated.
- format_options.excel
Options that define how Excel input is to be interpreted by DataBrew. Show child fields- format_options.excel.header_row
A variable that specifies whether the first row in the file is parsed as the header. If this value is false, column names are auto-generated. - format_options.excel.sheet_indexes[]
- format_options.excel.sheet_names[]
- format_options.json
Options that define how JSON input is to be interpreted by DataBrew. Show child fields- format_options.json.multi_line
A value that specifies whether JSON input contains embedded new line characters.
|
input
Information on how DataBrew can find the dataset, in either the Glue Data Catalog or Amazon S3. | STRUCT(
"s3_input_definition" STRUCT(
"bucket" VARCHAR,
"key" VARCHAR,
"bucket_owner" VARCHAR
),
"data_catalog_input_definition" STRUCT(
"catalog_id" VARCHAR,
"database_name" VARCHAR,
"table_name" VARCHAR,
"temp_directory" STRUCT(
"bucket" VARCHAR,
"key" VARCHAR,
"bucket_owner" VARCHAR
)
),
"database_input_definition" STRUCT(
"glue_connection_name" VARCHAR,
"database_table_name" VARCHAR,
"temp_directory" STRUCT(
"bucket" VARCHAR,
"key" VARCHAR,
"bucket_owner" VARCHAR
),
"query_string" VARCHAR
),
"metadata" STRUCT(
"source_arn" VARCHAR
)
) |
Show child fields- input.data_catalog_input_definition
The Glue Data Catalog parameters for the data. Show child fields- input.data_catalog_input_definition.catalog_id
The unique identifier of the Amazon Web Services account that holds the Data Catalog that stores the data. - input.data_catalog_input_definition.database_name
The name of a database in the Data Catalog. - input.data_catalog_input_definition.table_name
The name of a database table in the Data Catalog. This table corresponds to a DataBrew dataset. - input.data_catalog_input_definition.temp_directory
Represents an Amazon location where DataBrew can store intermediate results. Show child fields- input.data_catalog_input_definition.temp_directory.bucket
The Amazon S3 bucket name. - input.data_catalog_input_definition.temp_directory.bucket_owner
The Amazon Web Services account ID of the bucket owner. - input.data_catalog_input_definition.temp_directory.key
The unique name of the object in the bucket.
- input.database_input_definition
Connection information for dataset input files stored in a database. Show child fields- input.database_input_definition.database_table_name
The table within the target database. - input.database_input_definition.glue_connection_name
The Glue Connection that stores the connection information for the target database. - input.database_input_definition.query_string
Custom SQL to run against the provided Glue connection. This SQL will be used as the input for DataBrew projects and jobs. - input.database_input_definition.temp_directory
Represents an Amazon S3 location (bucket name, bucket owner, and object key) where DataBrew can read input data, or write output from a job. Show child fields- input.database_input_definition.temp_directory.bucket
The Amazon S3 bucket name. - input.database_input_definition.temp_directory.bucket_owner
The Amazon Web Services account ID of the bucket owner. - input.database_input_definition.temp_directory.key
The unique name of the object in the bucket.
- input.metadata
Contains additional resource information needed for specific datasets. Show child fields- input.metadata.source_arn
The Amazon Resource Name (ARN) associated with the dataset. Currently, DataBrew only supports ARNs from Amazon AppFlow.
- input.s3_input_definition
The Amazon S3 location where the data is stored. Show child fields- input.s3_input_definition.bucket
The Amazon S3 bucket name. - input.s3_input_definition.bucket_owner
The Amazon Web Services account ID of the bucket owner. - input.s3_input_definition.key
The unique name of the object in the bucket.
|
last_modified_by
The Amazon Resource Name (ARN) of the user who last modified the dataset. | VARCHAR |
last_modified_date
The last modification date and time of the dataset. | TIMESTAMP_S |
name
The unique name of the dataset. | VARCHAR |
path_options
A set of options that defines how DataBrew interprets an Amazon S3 path of the dataset. | STRUCT(
"last_modified_date_condition" STRUCT(
"expression" VARCHAR,
"values_map" MAP(VARCHAR, VARCHAR)
),
"files_limit" STRUCT(
"max_files" BIGINT,
"ordered_by" VARCHAR,
"order" VARCHAR
),
"parameters" MAP(VARCHAR, STRUCT(
"name" VARCHAR,
"type" VARCHAR,
"datetime_options" STRUCT(
"format" VARCHAR,
"timezone_offset" VARCHAR,
"locale_code" VARCHAR
),
"create_column" BOOLEAN,
"filter" STRUCT(
"expression" VARCHAR,
"values_map" MAP(VARCHAR, VARCHAR)
)
))
) |
Show child fields- path_options.files_limit
If provided, this structure imposes a limit on a number of files that should be selected. Show child fields- path_options.files_limit.max_files
The number of Amazon S3 files to select. - path_options.files_limit.order
A criteria to use for Amazon S3 files sorting before their selection. By default uses DESCENDING order, i.e. most recent files are selected first. Another possible value is ASCENDING. - path_options.files_limit.ordered_by
A criteria to use for Amazon S3 files sorting before their selection. By default uses LAST_MODIFIED_DATE as a sorting criteria. Currently it's the only allowed value.
- path_options.last_modified_date_condition
If provided, this structure defines a date range for matching Amazon S3 objects based on their LastModifiedDate attribute in Amazon S3. Show child fields- path_options.last_modified_date_condition.expression
The expression which includes condition names followed by substitution variables, possibly grouped and combined with other conditions. For example, "(starts_with :prefix1 or starts_with :prefix2) and (ends_with :suffix1 or ends_with :suffix2)". Substitution variables should start with ':' symbol. - path_options.last_modified_date_condition.values_map
The map of substitution variable names to their values used in this filter expression.
- path_options.parameters
A structure that maps names of parameters used in the Amazon S3 path of a dataset to their definitions.
|
resource_arn
The unique Amazon Resource Name (ARN) for the dataset. | VARCHAR |
source
The location of the data for the dataset, either Amazon S3 or the Glue Data Catalog. | VARCHAR |
tags
Metadata tags that have been applied to the dataset. | MAP(VARCHAR, VARCHAR) |