Skip to content
AWS for Query.Farm

aws.glue.get_crawler

Example SQL Queries

SELECT * FROM
aws.glue.get_crawler
WHERE
"name" = 'VALUE';

Description

Retrieves metadata for a specified crawler.

Table Definition

Column NameColumn Data Type
name Required Input Column

The name of the crawler to retrieve metadata for.

VARCHAR
_aws_profile Input Column

The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role.

STRUCT(
    "type" VARCHAR,
    "name" VARCHAR,
    "account_id" VARCHAR,
    "via_profile_name" VARCHAR,
    "assumed_role_arn" VARCHAR,
    "organization" STRUCT(
        "account_name" VARCHAR,
        "id" VARCHAR,
        "tags" STRUCT(
            "key" VARCHAR,
            "value" VARCHAR
        )[],
        "master_account" STRUCT(
            "id" VARCHAR,
            "email" VARCHAR
        ),
        "parents" STRUCT(
            "type" VARCHAR,
            "id" VARCHAR,
            "name" VARCHAR,
            "tags" STRUCT(
                "key" VARCHAR,
                "value" VARCHAR
            )[]
        )[]
    )
)
Show child fields
_aws_profile.account_id

The AWS account id

_aws_profile.assumed_role_arn

The ARN of the assumed role

_aws_profile.name

The unique name of the profile.

_aws_profile.organization

Information about this profile's membership in the AWS organization.

Show child fields
_aws_profile.organization.account_name

The name of account speciifed by the organization

_aws_profile.organization.id

The organization id

_aws_profile.organization.master_account
Show child fields
_aws_profile.organization.master_account.email

The organization master account email address

_aws_profile.organization.master_account.id

The organization master account id

_aws_profile.organization.parents[]
Show child fields
_aws_profile.organization.parents[].id

The id of the parent

_aws_profile.organization.parents[].name

The name of the parent

_aws_profile.organization.parents[].tags[]
Show child fields
_aws_profile.organization.parents[].tags[].key
_aws_profile.organization.parents[].tags[].value
_aws_profile.organization.parents[].type

The type of parent can be an organization unit or a root

_aws_profile.organization.tags[]
Show child fields
_aws_profile.organization.tags[].key
_aws_profile.organization.tags[].value
_aws_profile.type

The type of profile, either 'credentials' or 'assumed_role'

_aws_profile.via_profile_name

This IAM role for this profile is assumed by first utilizing another profile with this name to obtain credentials.

_aws_region Input Column

The AWS region to use.

VARCHAR
crawler

The metadata for the specified crawler.

STRUCT(
    "name" VARCHAR,
    "role" VARCHAR,
    "targets" STRUCT(
        "s3_targets" STRUCT(
            "path" VARCHAR,
            "exclusions" VARCHAR[],
            "connection_name" VARCHAR,
            "sample_size" BIGINT,
            "event_queue_arn" VARCHAR,
            "dlq_event_queue_arn" VARCHAR
        )[],
        "jdbc_targets" STRUCT(
            "connection_name" VARCHAR,
            "path" VARCHAR,
            "exclusions" VARCHAR[],
            "enable_additional_metadata" VARCHAR[]
        )[],
        "mongo_db_targets" STRUCT(
            "connection_name" VARCHAR,
            "path" VARCHAR,
            "scan_all" BOOLEAN
        )[],
        "dynamo_db_targets" STRUCT(
            "path" VARCHAR,
            "scan_all" BOOLEAN,
            "scan_rate" DOUBLE
        )[],
        "catalog_targets" STRUCT(
            "database_name" VARCHAR,
            "tables" VARCHAR[],
            "connection_name" VARCHAR,
            "event_queue_arn" VARCHAR,
            "dlq_event_queue_arn" VARCHAR
        )[],
        "delta_targets" STRUCT(
            "delta_tables" VARCHAR[],
            "connection_name" VARCHAR,
            "write_manifest" BOOLEAN,
            "create_native_delta_table" BOOLEAN
        )[],
        "iceberg_targets" STRUCT(
            "paths" VARCHAR[],
            "connection_name" VARCHAR,
            "exclusions" VARCHAR[],
            "maximum_traversal_depth" BIGINT
        )[],
        "hudi_targets" STRUCT(
            "paths" VARCHAR[],
            "connection_name" VARCHAR,
            "exclusions" VARCHAR[],
            "maximum_traversal_depth" BIGINT
        )[]
    ),
    "database_name" VARCHAR,
    "description" VARCHAR,
    "classifiers" VARCHAR[],
    "recrawl_policy" STRUCT(
        "recrawl_behavior" VARCHAR
    ),
    "schema_change_policy" STRUCT(
        "update_behavior" VARCHAR,
        "delete_behavior" VARCHAR
    ),
    "lineage_configuration" STRUCT(
        "crawler_lineage_settings" VARCHAR
    ),
    "state" VARCHAR,
    "table_prefix" VARCHAR,
    "schedule" STRUCT(
        "schedule_expression" VARCHAR,
        "state" VARCHAR
    ),
    "crawl_elapsed_time" BIGINT,
    "creation_time" TIMESTAMP_S,
    "last_updated" TIMESTAMP_S,
    "last_crawl" STRUCT(
        "status" VARCHAR,
        "error_message" VARCHAR,
        "log_group" VARCHAR,
        "log_stream" VARCHAR,
        "message_prefix" VARCHAR,
        "start_time" TIMESTAMP_S
    ),
    "version" BIGINT,
    "configuration" VARCHAR,
    "crawler_security_configuration" VARCHAR,
    "lake_formation_configuration" STRUCT(
        "use_lake_formation_credentials" BOOLEAN,
        "account_id" VARCHAR
    )
)
Show child fields
crawler.classifiers[]
crawler.configuration

Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Setting crawler configuration options.

crawler.crawl_elapsed_time

If the crawler is running, contains the total time elapsed since the last crawl began.

crawler.crawler_security_configuration

The name of the SecurityConfiguration structure to be used by this crawler.

crawler.creation_time

The time that the crawler was created.

crawler.database_name

The name of the database in which the crawler's output is stored.

crawler.description

A description of the crawler.

crawler.lake_formation_configuration

Specifies whether the crawler should use Lake Formation credentials for the crawler instead of the IAM role credentials.

Show child fields
crawler.lake_formation_configuration.account_id

Required for cross account crawls. For same account crawls as the target data, this can be left as null.

crawler.lake_formation_configuration.use_lake_formation_credentials

Specifies whether to use Lake Formation credentials for the crawler instead of the IAM role credentials.

crawler.last_crawl

The status of the last crawl, and potentially error information if an error occurred.

Show child fields
crawler.last_crawl.error_message

If an error occurred, the error information about the last crawl.

crawler.last_crawl.log_group

The log group for the last crawl.

crawler.last_crawl.log_stream

The log stream for the last crawl.

crawler.last_crawl.message_prefix

The prefix for a message about this crawl.

crawler.last_crawl.start_time

The time at which the crawl started.

crawler.last_crawl.status

Status of the last crawl.

crawler.last_updated

The time that the crawler was last updated.

crawler.lineage_configuration

A configuration that specifies whether data lineage is enabled for the crawler.

Show child fields
crawler.lineage_configuration.crawler_lineage_settings

Specifies whether data lineage is enabled for the crawler. Valid values are:

  • ENABLE: enables data lineage for the crawler

  • DISABLE: disables data lineage for the crawler

crawler.name

The name of the crawler.

crawler.recrawl_policy

A policy that specifies whether to crawl the entire dataset again, or to crawl only folders that were added since the last crawler run.

Show child fields
crawler.recrawl_policy.recrawl_behavior

Specifies whether to crawl the entire dataset again or to crawl only folders that were added since the last crawler run.

A value of CRAWL_EVERYTHING specifies crawling the entire dataset again.

A value of CRAWL_NEW_FOLDERS_ONLY specifies crawling only folders that were added since the last crawler run.

A value of CRAWL_EVENT_MODE specifies crawling only the changes identified by Amazon S3 events.

crawler.role

The Amazon Resource Name (ARN) of an IAM role that's used to access customer resources, such as Amazon Simple Storage Service (Amazon S3) data.

crawler.schedule

For scheduled crawlers, the schedule when the crawler runs.

Show child fields
crawler.schedule.schedule_expression

A cron expression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify: cron(15 12 * * ? *).

crawler.schedule.state

The state of the schedule.

crawler.schema_change_policy

The policy that specifies update and delete behaviors for the crawler.

Show child fields
crawler.schema_change_policy.delete_behavior

The deletion behavior when the crawler finds a deleted object.

crawler.schema_change_policy.update_behavior

The update behavior when the crawler finds a changed schema.

crawler.state

Indicates whether the crawler is running, or whether a run is pending.

crawler.table_prefix

The prefix added to the names of tables that are created.

crawler.targets

A collection of targets to crawl.

Show child fields
crawler.targets.catalog_targets[]
Show child fields
crawler.targets.catalog_targets[].connection_name

The name of the connection for an Amazon S3-backed Data Catalog table to be a target of the crawl when using a Catalog connection type paired with a NETWORK Connection type.

crawler.targets.catalog_targets[].database_name

The name of the database to be synchronized.

crawler.targets.catalog_targets[].dlq_event_queue_arn

A valid Amazon dead-letter SQS ARN. For example, arn:aws:sqs:region:account:deadLetterQueue.

crawler.targets.catalog_targets[].event_queue_arn

A valid Amazon SQS ARN. For example, arn:aws:sqs:region:account:sqs.

crawler.targets.catalog_targets[].tables[]
crawler.targets.delta_targets[]
Show child fields
crawler.targets.delta_targets[].connection_name

The name of the connection to use to connect to the Delta table target.

crawler.targets.delta_targets[].create_native_delta_table

Specifies whether the crawler will create native tables, to allow integration with query engines that support querying of the Delta transaction log directly.

crawler.targets.delta_targets[].delta_tables[]
crawler.targets.delta_targets[].write_manifest

Specifies whether to write the manifest files to the Delta table path.

crawler.targets.dynamo_db_targets[]
Show child fields
crawler.targets.dynamo_db_targets[].path

The name of the DynamoDB table to crawl.

crawler.targets.dynamo_db_targets[].scan_all

Indicates whether to scan all the records, or to sample rows from the table. Scanning all the records can take a long time when the table is not a high throughput table.

A value of true means to scan all records, while a value of false means to sample the records. If no value is specified, the value defaults to true.

crawler.targets.dynamo_db_targets[].scan_rate

The percentage of the configured read capacity units to use by the Glue crawler. Read capacity units is a term defined by DynamoDB, and is a numeric value that acts as rate limiter for the number of reads that can be performed on that table per second.

The valid values are null or a value between 0.1 to 1.5. A null value is used when user does not provide a value, and defaults to 0.5 of the configured Read Capacity Unit (for provisioned tables), or 0.25 of the max configured Read Capacity Unit (for tables using on-demand mode).

crawler.targets.hudi_targets[]
Show child fields
crawler.targets.hudi_targets[].connection_name

The name of the connection to use to connect to the Hudi target. If your Hudi files are stored in buckets that require VPC authorization, you can set their connection properties here.

crawler.targets.hudi_targets[].exclusions[]
crawler.targets.hudi_targets[].maximum_traversal_depth

The maximum depth of Amazon S3 paths that the crawler can traverse to discover the Hudi metadata folder in your Amazon S3 path. Used to limit the crawler run time.

crawler.targets.hudi_targets[].paths[]
crawler.targets.iceberg_targets[]
Show child fields
crawler.targets.iceberg_targets[].connection_name

The name of the connection to use to connect to the Iceberg target.

crawler.targets.iceberg_targets[].exclusions[]
crawler.targets.iceberg_targets[].maximum_traversal_depth

The maximum depth of Amazon S3 paths that the crawler can traverse to discover the Iceberg metadata folder in your Amazon S3 path. Used to limit the crawler run time.

crawler.targets.iceberg_targets[].paths[]
crawler.targets.jdbc_targets[]
Show child fields
crawler.targets.jdbc_targets[].connection_name

The name of the connection to use to connect to the JDBC target.

crawler.targets.jdbc_targets[].enable_additional_metadata[]
crawler.targets.jdbc_targets[].exclusions[]
crawler.targets.jdbc_targets[].path

The path of the JDBC target.

crawler.targets.mongo_db_targets[]
Show child fields
crawler.targets.mongo_db_targets[].connection_name

The name of the connection to use to connect to the Amazon DocumentDB or MongoDB target.

crawler.targets.mongo_db_targets[].path

The path of the Amazon DocumentDB or MongoDB target (database/collection).

crawler.targets.mongo_db_targets[].scan_all

Indicates whether to scan all the records, or to sample rows from the table. Scanning all the records can take a long time when the table is not a high throughput table.

A value of true means to scan all records, while a value of false means to sample the records. If no value is specified, the value defaults to true.

crawler.targets.s3_targets[]
Show child fields
crawler.targets.s3_targets[].connection_name

The name of a connection which allows a job or crawler to access data in Amazon S3 within an Amazon Virtual Private Cloud environment (Amazon VPC).

crawler.targets.s3_targets[].dlq_event_queue_arn

A valid Amazon dead-letter SQS ARN. For example, arn:aws:sqs:region:account:deadLetterQueue.

crawler.targets.s3_targets[].event_queue_arn

A valid Amazon SQS ARN. For example, arn:aws:sqs:region:account:sqs.

crawler.targets.s3_targets[].exclusions[]
crawler.targets.s3_targets[].path

The path to the Amazon S3 target.

crawler.targets.s3_targets[].sample_size

Sets the number of files in each leaf folder to be crawled when crawling sample files in a dataset. If not set, all the files are crawled. A valid value is an integer between 1 and 249.

crawler.version

The version of the crawler.