Skip to content
AWS for Query.Farm

aws.lookoutequipment.describe_data_ingestion_job

Example SQL Queries

SELECT * FROM
aws.lookoutequipment.describe_data_ingestion_job
WHERE
"job_id" = 'VALUE';

Description

Provides information on a specific data ingestion job such as creation time, dataset ARN, and status.

Table Definition

Column NameColumn Data Type
job_id Required Input Column

Indicates the job ID of the data ingestion job. 

VARCHAR
_aws_profile Input Column

The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role.

STRUCT(
    "type" VARCHAR,
    "name" VARCHAR,
    "account_id" VARCHAR,
    "via_profile_name" VARCHAR,
    "assumed_role_arn" VARCHAR,
    "organization" STRUCT(
        "account_name" VARCHAR,
        "id" VARCHAR,
        "tags" STRUCT(
            "key" VARCHAR,
            "value" VARCHAR
        )[],
        "master_account" STRUCT(
            "id" VARCHAR,
            "email" VARCHAR
        ),
        "parents" STRUCT(
            "type" VARCHAR,
            "id" VARCHAR,
            "name" VARCHAR,
            "tags" STRUCT(
                "key" VARCHAR,
                "value" VARCHAR
            )[]
        )[]
    )
)
Show child fields
_aws_profile.account_id

The AWS account id

_aws_profile.assumed_role_arn

The ARN of the assumed role

_aws_profile.name

The unique name of the profile.

_aws_profile.organization

Information about this profile's membership in the AWS organization.

Show child fields
_aws_profile.organization.account_name

The name of account speciifed by the organization

_aws_profile.organization.id

The organization id

_aws_profile.organization.master_account
Show child fields
_aws_profile.organization.master_account.email

The organization master account email address

_aws_profile.organization.master_account.id

The organization master account id

_aws_profile.organization.parents[]
Show child fields
_aws_profile.organization.parents[].id

The id of the parent

_aws_profile.organization.parents[].name

The name of the parent

_aws_profile.organization.parents[].tags[]
Show child fields
_aws_profile.organization.parents[].tags[].key
_aws_profile.organization.parents[].tags[].value
_aws_profile.organization.parents[].type

The type of parent can be an organization unit or a root

_aws_profile.organization.tags[]
Show child fields
_aws_profile.organization.tags[].key
_aws_profile.organization.tags[].value
_aws_profile.type

The type of profile, either 'credentials' or 'assumed_role'

_aws_profile.via_profile_name

This IAM role for this profile is assumed by first utilizing another profile with this name to obtain credentials.

_aws_region Input Column

The AWS region to use.

VARCHAR
created_at

The time at which the data ingestion job was created. 

TIMESTAMP_S
data_end_time

Indicates the latest timestamp corresponding to data that was successfully ingested during this specific ingestion job. 

TIMESTAMP_S
data_quality_summary

Gives statistics about a completed ingestion job. These statistics primarily relate to quantifying incorrect data such as MissingCompleteSensorData, MissingSensorData, UnsupportedDateFormats, InsufficientSensorData, and DuplicateTimeStamps. 

STRUCT(
    "insufficient_sensor_data" STRUCT(
        "missing_complete_sensor_data" STRUCT(
            "affected_sensor_count" BIGINT
        ),
        "sensors_with_short_date_range" STRUCT(
            "affected_sensor_count" BIGINT
        )
    ),
    "missing_sensor_data" STRUCT(
        "affected_sensor_count" BIGINT,
        "total_number_of_missing_values" BIGINT
    ),
    "invalid_sensor_data" STRUCT(
        "affected_sensor_count" BIGINT,
        "total_number_of_invalid_values" BIGINT
    ),
    "unsupported_timestamps" STRUCT(
        "total_number_of_unsupported_timestamps" BIGINT
    ),
    "duplicate_timestamps" STRUCT(
        "total_number_of_duplicate_timestamps" BIGINT
    )
)
Show child fields
data_quality_summary.duplicate_timestamps

Parameter that gives information about duplicate timestamps in the input data.

Show child fields
data_quality_summary.duplicate_timestamps.total_number_of_duplicate_timestamps

Indicates the total number of duplicate timestamps.

data_quality_summary.insufficient_sensor_data

Parameter that gives information about insufficient data for sensors in the dataset. This includes information about those sensors that have complete data missing and those with a short date range.

Show child fields
data_quality_summary.insufficient_sensor_data.missing_complete_sensor_data

Parameter that describes the total number of sensors that have data completely missing for it.

Show child fields
data_quality_summary.insufficient_sensor_data.missing_complete_sensor_data.affected_sensor_count

Indicates the number of sensors that have data missing completely.

data_quality_summary.insufficient_sensor_data.sensors_with_short_date_range

Parameter that describes the total number of sensors that have a short date range of less than 14 days of data overall.

Show child fields
data_quality_summary.insufficient_sensor_data.sensors_with_short_date_range.affected_sensor_count

Indicates the number of sensors that have less than 14 days of data.

data_quality_summary.invalid_sensor_data

Parameter that gives information about data that is invalid over all the sensors in the input data.

Show child fields
data_quality_summary.invalid_sensor_data.affected_sensor_count

Indicates the number of sensors that have at least some invalid values.

data_quality_summary.invalid_sensor_data.total_number_of_invalid_values

Indicates the total number of invalid values across all the sensors.

data_quality_summary.missing_sensor_data

Parameter that gives information about data that is missing over all the sensors in the input data.

Show child fields
data_quality_summary.missing_sensor_data.affected_sensor_count

Indicates the number of sensors that have atleast some data missing.

data_quality_summary.missing_sensor_data.total_number_of_missing_values

Indicates the total number of missing values across all the sensors.

data_quality_summary.unsupported_timestamps

Parameter that gives information about unsupported timestamps in the input data.

Show child fields
data_quality_summary.unsupported_timestamps.total_number_of_unsupported_timestamps

Indicates the total number of unsupported timestamps across the ingested data.

data_start_time

Indicates the earliest timestamp corresponding to data that was successfully ingested during this specific ingestion job. 

TIMESTAMP_S
dataset_arn

The Amazon Resource Name (ARN) of the dataset being used in the data ingestion job. 

VARCHAR
failed_reason

Specifies the reason for failure when a data ingestion job has failed. 

VARCHAR
ingested_data_size

Indicates the size of the ingested dataset. 

BIGINT
ingested_files_summary

Gives statistics about how many files have been ingested, and which files have not been ingested, for a particular ingestion job.

STRUCT(
    "total_number_of_files" BIGINT,
    "ingested_number_of_files" BIGINT,
    "discarded_files" STRUCT(
        "bucket" VARCHAR,
        "key" VARCHAR
    )[]
)
Show child fields
ingested_files_summary.discarded_files[]
Show child fields
ingested_files_summary.discarded_files[].bucket

The name of the specific S3 bucket.

ingested_files_summary.discarded_files[].key

The Amazon Web Services Key Management Service (KMS key) key being used to encrypt the S3 object. Without this key, data in the bucket is not accessible.

ingested_files_summary.ingested_number_of_files

Indicates the number of files that were successfully ingested.

ingested_files_summary.total_number_of_files

Indicates the total number of files that were submitted for ingestion.

ingestion_input_configuration

Specifies the S3 location configuration for the data input for the data ingestion job. 

STRUCT(
    "s3_input_configuration" STRUCT(
        "bucket" VARCHAR,
        "prefix" VARCHAR,
        "key_pattern" VARCHAR
    )
)
Show child fields
ingestion_input_configuration.s3_input_configuration

The location information for the S3 bucket used for input data for the data ingestion.

Show child fields
ingestion_input_configuration.s3_input_configuration.bucket

The name of the S3 bucket used for the input data for the data ingestion.

ingestion_input_configuration.s3_input_configuration.key_pattern

The pattern for matching the Amazon S3 files that will be used for ingestion. If the schema was created previously without any KeyPattern, then the default KeyPattern {prefix}/{component_name}/* is used to download files from Amazon S3 according to the schema. This field is required when ingestion is being done for the first time.

Valid Values: {prefix}/{component_name}_* | {prefix}/{component_name}/* | {prefix}/{component_name}[DELIMITER]* (Allowed delimiters : space, dot, underscore, hyphen)

ingestion_input_configuration.s3_input_configuration.prefix

The prefix for the S3 location being used for the input data for the data ingestion.

role_arn

The Amazon Resource Name (ARN) of an IAM role with permission to access the data source being ingested. 

VARCHAR
source_dataset_arn

The Amazon Resource Name (ARN) of the source dataset from which the data used for the data ingestion job was imported from.

VARCHAR
status

Indicates the status of the DataIngestionJob operation. 

VARCHAR
status_detail

Provides details about status of the ingestion job that is currently in progress. 

VARCHAR