Skip to content
AWS for Query.Farm

aws.kafka.list_cluster_operations

Example SQL Queries

SELECT * FROM
aws.kafka.list_cluster_operations
WHERE
"cluster_arn" = 'VALUE';

Description

Returns a list of all the operations that have been performed on the specified MSK cluster.

Table Definition

Column NameColumn Data Type
cluster_arn Required Input Column

ARN of the cluster.

VARCHAR
_aws_profile Input Column

The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role.

STRUCT(
    "type" VARCHAR,
    "name" VARCHAR,
    "account_id" VARCHAR,
    "via_profile_name" VARCHAR,
    "assumed_role_arn" VARCHAR,
    "organization" STRUCT(
        "account_name" VARCHAR,
        "id" VARCHAR,
        "tags" STRUCT(
            "key" VARCHAR,
            "value" VARCHAR
        )[],
        "master_account" STRUCT(
            "id" VARCHAR,
            "email" VARCHAR
        ),
        "parents" STRUCT(
            "type" VARCHAR,
            "id" VARCHAR,
            "name" VARCHAR,
            "tags" STRUCT(
                "key" VARCHAR,
                "value" VARCHAR
            )[]
        )[]
    )
)
Show child fields
_aws_profile.account_id

The AWS account id

_aws_profile.assumed_role_arn

The ARN of the assumed role

_aws_profile.name

The unique name of the profile.

_aws_profile.organization

Information about this profile's membership in the AWS organization.

Show child fields
_aws_profile.organization.account_name

The name of account speciifed by the organization

_aws_profile.organization.id

The organization id

_aws_profile.organization.master_account
Show child fields
_aws_profile.organization.master_account.email

The organization master account email address

_aws_profile.organization.master_account.id

The organization master account id

_aws_profile.organization.parents[]
Show child fields
_aws_profile.organization.parents[].id

The id of the parent

_aws_profile.organization.parents[].name

The name of the parent

_aws_profile.organization.parents[].tags[]
Show child fields
_aws_profile.organization.parents[].tags[].key
_aws_profile.organization.parents[].tags[].value
_aws_profile.organization.parents[].type

The type of parent can be an organization unit or a root

_aws_profile.organization.tags[]
Show child fields
_aws_profile.organization.tags[].key
_aws_profile.organization.tags[].value
_aws_profile.type

The type of profile, either 'credentials' or 'assumed_role'

_aws_profile.via_profile_name

This IAM role for this profile is assumed by first utilizing another profile with this name to obtain credentials.

_aws_region Input Column

The AWS region to use.

VARCHAR
client_request_id

The ID of the API request that triggered this operation.

VARCHAR
creation_time

The time that the operation was created.

TIMESTAMP_S
end_time

The time at which the operation finished.

TIMESTAMP_S
error_info

Describes the error if the operation fails.

STRUCT(
    "error_code" VARCHAR,
    "error_string" VARCHAR
)
Show child fields
error_info.error_code

A number describing the error programmatically.

error_info.error_string

An optional field to provide more details about the error.

operation_arn

ARN of the cluster operation.

VARCHAR
operation_state

State of the cluster operation.

VARCHAR
operation_steps

Steps completed during the operation.

STRUCT(
    "step_info" STRUCT(
        "step_status" VARCHAR
    ),
    "step_name" VARCHAR
)[]
Show child fields
operation_steps[]
Show child fields
operation_steps[].step_info

Information about the step and its status.

Show child fields
operation_steps[].step_info.step_status

The steps current status.

operation_steps[].step_name

The name of the step.

operation_type

Type of the cluster operation.

VARCHAR
source_cluster_info

Information about cluster attributes before a cluster is updated.

STRUCT(
    "broker_ebs_volume_info" STRUCT(
        "kafka_broker_node_id" VARCHAR,
        "provisioned_throughput" STRUCT(
            "enabled" BOOLEAN,
            "volume_throughput" BIGINT
        ),
        "volume_size_gb" BIGINT
    )[],
    "configuration_info" STRUCT(
        "arn" VARCHAR,
        "revision" BIGINT
    ),
    "number_of_broker_nodes" BIGINT,
    "enhanced_monitoring" VARCHAR,
    "open_monitoring" STRUCT(
        "prometheus" STRUCT(
            "jmx_exporter" STRUCT(
                "enabled_in_broker" BOOLEAN
            ),
            "node_exporter" STRUCT(
                "enabled_in_broker" BOOLEAN
            )
        )
    ),
    "kafka_version" VARCHAR,
    "logging_info" STRUCT(
        "broker_logs" STRUCT(
            "cloud_watch_logs" STRUCT(
                "enabled" BOOLEAN,
                "log_group" VARCHAR
            ),
            "firehose" STRUCT(
                "delivery_stream" VARCHAR,
                "enabled" BOOLEAN
            ),
            "s3" STRUCT(
                "bucket" VARCHAR,
                "enabled" BOOLEAN,
                "prefix" VARCHAR
            )
        )
    ),
    "instance_type" VARCHAR,
    "client_authentication" STRUCT(
        "sasl" STRUCT(
            "scram" STRUCT(
                "enabled" BOOLEAN
            ),
            "iam" STRUCT(
                "enabled" BOOLEAN
            )
        ),
        "tls" STRUCT(
            "certificate_authority_arn_list" VARCHAR[],
            "enabled" BOOLEAN
        ),
        "unauthenticated" STRUCT(
            "enabled" BOOLEAN
        )
    ),
    "encryption_info" STRUCT(
        "encryption_at_rest" STRUCT(
            "data_volume_kms_key_id" VARCHAR
        ),
        "encryption_in_transit" STRUCT(
            "client_broker" VARCHAR,
            "in_cluster" BOOLEAN
        )
    ),
    "connectivity_info" STRUCT(
        "public_access" STRUCT(
            "type" VARCHAR
        ),
        "vpc_connectivity" STRUCT(
            "client_authentication" STRUCT(
                "sasl" STRUCT(
                    "scram" STRUCT(
                        "enabled" BOOLEAN
                    ),
                    "iam" STRUCT(
                        "enabled" BOOLEAN
                    )
                ),
                "tls" STRUCT(
                    "enabled" BOOLEAN
                )
            )
        )
    ),
    "storage_mode" VARCHAR,
    "broker_count_update_info" STRUCT(
        "created_broker_ids" DOUBLE[],
        "deleted_broker_ids" DOUBLE[]
    )
)
Show child fields
source_cluster_info.broker_count_update_info

Describes brokers being changed during a broker count update.

Show child fields
source_cluster_info.broker_count_update_info.created_broker_ids[]
source_cluster_info.broker_count_update_info.deleted_broker_ids[]
source_cluster_info.broker_ebs_volume_info[]
Show child fields
source_cluster_info.broker_ebs_volume_info[].kafka_broker_node_id

The ID of the broker to update.

source_cluster_info.broker_ebs_volume_info[].provisioned_throughput

EBS volume provisioned throughput information.

Show child fields
source_cluster_info.broker_ebs_volume_info[].provisioned_throughput.enabled

Provisioned throughput is enabled or not.

source_cluster_info.broker_ebs_volume_info[].provisioned_throughput.volume_throughput

Throughput value of the EBS volumes for the data drive on each kafka broker node in MiB per second.

source_cluster_info.broker_ebs_volume_info[].volume_size_gb

Size of the EBS volume to update.

source_cluster_info.client_authentication

Includes all client authentication information.

Show child fields
source_cluster_info.client_authentication.sasl

Details for ClientAuthentication using SASL.

Show child fields
source_cluster_info.client_authentication.sasl.iam

Indicates whether IAM access control is enabled.

Show child fields
source_cluster_info.client_authentication.sasl.iam.enabled

Indicates whether IAM access control is enabled.

source_cluster_info.client_authentication.sasl.scram

Details for SASL/SCRAM client authentication.

Show child fields
source_cluster_info.client_authentication.sasl.scram.enabled

SASL/SCRAM authentication is enabled or not.

source_cluster_info.client_authentication.tls

Details for ClientAuthentication using TLS.

Show child fields
source_cluster_info.client_authentication.tls.certificate_authority_arn_list[]
source_cluster_info.client_authentication.tls.enabled

Specifies whether you want to turn on or turn off TLS authentication.

source_cluster_info.client_authentication.unauthenticated

Contains information about unauthenticated traffic to the cluster.

Show child fields
source_cluster_info.client_authentication.unauthenticated.enabled

Specifies whether you want to turn on or turn off unauthenticated traffic to your cluster.

source_cluster_info.configuration_info

Information about the changes in the configuration of the brokers.

Show child fields
source_cluster_info.configuration_info.arn

ARN of the configuration to use.

source_cluster_info.configuration_info.revision

The revision of the configuration to use.

source_cluster_info.connectivity_info

Information about the broker access configuration.

Show child fields
source_cluster_info.connectivity_info.public_access

Public access control for brokers.

Show child fields
source_cluster_info.connectivity_info.public_access.type

The value DISABLED indicates that public access is turned off. SERVICE_PROVIDED_EIPS indicates that public access is turned on.

source_cluster_info.connectivity_info.vpc_connectivity

VPC connectivity access control for brokers.

Show child fields
source_cluster_info.connectivity_info.vpc_connectivity.client_authentication

Includes all client authentication information for VPC connectivity.

Show child fields
source_cluster_info.connectivity_info.vpc_connectivity.client_authentication.sasl

SASL authentication type details for VPC connectivity.

Show child fields
source_cluster_info.connectivity_info.vpc_connectivity.client_authentication.sasl.iam

Details for SASL/IAM client authentication for VPC connectivity.

Show child fields
source_cluster_info.connectivity_info.vpc_connectivity.client_authentication.sasl.iam.enabled

SASL/IAM authentication is on or off for VPC connectivity.

source_cluster_info.connectivity_info.vpc_connectivity.client_authentication.sasl.scram

Details for SASL/SCRAM client authentication for VPC connectivity.

Show child fields
source_cluster_info.connectivity_info.vpc_connectivity.client_authentication.sasl.scram.enabled

SASL/SCRAM authentication is on or off for VPC connectivity.

source_cluster_info.connectivity_info.vpc_connectivity.client_authentication.tls

TLS authentication type details for VPC connectivity.

Show child fields
source_cluster_info.connectivity_info.vpc_connectivity.client_authentication.tls.enabled

TLS authentication is on or off for VPC connectivity.

source_cluster_info.encryption_info

Includes all encryption-related information.

Show child fields
source_cluster_info.encryption_info.encryption_at_rest

The data-volume encryption details.

Show child fields
source_cluster_info.encryption_info.encryption_at_rest.data_volume_kms_key_id

The ARN of the AWS KMS key for encrypting data at rest. If you don't specify a KMS key, MSK creates one for you and uses it.

source_cluster_info.encryption_info.encryption_in_transit

The details for encryption in transit.

Show child fields
source_cluster_info.encryption_info.encryption_in_transit.client_broker

Indicates the encryption setting for data in transit between clients and brokers. The following are the possible values.

TLS means that client-broker communication is enabled with TLS only.

TLS_PLAINTEXT means that client-broker communication is enabled for both TLS-encrypted, as well as plaintext data.

PLAINTEXT means that client-broker communication is enabled in plaintext only.

The default value is TLS_PLAINTEXT.

source_cluster_info.encryption_info.encryption_in_transit.in_cluster

When set to true, it indicates that data communication among the broker nodes of the cluster is encrypted. When set to false, the communication happens in plaintext.

The default value is true.

source_cluster_info.enhanced_monitoring

Specifies which Apache Kafka metrics Amazon MSK gathers and sends to Amazon CloudWatch for this cluster.

source_cluster_info.instance_type

Information about the Amazon MSK broker type.

source_cluster_info.kafka_version

The Apache Kafka version.

source_cluster_info.logging_info

You can configure your MSK cluster to send broker logs to different destination types. This is a container for the configuration details related to broker logs.

Show child fields
source_cluster_info.logging_info.broker_logs
Show child fields
source_cluster_info.logging_info.broker_logs.cloud_watch_logs
Show child fields
source_cluster_info.logging_info.broker_logs.cloud_watch_logs.enabled
source_cluster_info.logging_info.broker_logs.cloud_watch_logs.log_group
source_cluster_info.logging_info.broker_logs.firehose
Show child fields
source_cluster_info.logging_info.broker_logs.firehose.delivery_stream
source_cluster_info.logging_info.broker_logs.firehose.enabled
source_cluster_info.logging_info.broker_logs.s3
Show child fields
source_cluster_info.logging_info.broker_logs.s3.bucket
source_cluster_info.logging_info.broker_logs.s3.enabled
source_cluster_info.logging_info.broker_logs.s3.prefix
source_cluster_info.number_of_broker_nodes

The number of broker nodes in the cluster.

source_cluster_info.open_monitoring

The settings for open monitoring.

Show child fields
source_cluster_info.open_monitoring.prometheus

Prometheus settings.

Show child fields
source_cluster_info.open_monitoring.prometheus.jmx_exporter

Indicates whether you want to turn on or turn off the JMX Exporter.

Show child fields
source_cluster_info.open_monitoring.prometheus.jmx_exporter.enabled_in_broker

Indicates whether you want to turn on or turn off the JMX Exporter.

source_cluster_info.open_monitoring.prometheus.node_exporter

Indicates whether you want to turn on or turn off the Node Exporter.

Show child fields
source_cluster_info.open_monitoring.prometheus.node_exporter.enabled_in_broker

Indicates whether you want to turn on or turn off the Node Exporter.

source_cluster_info.storage_mode

This controls storage mode for supported storage tiers.

target_cluster_info

Information about cluster attributes after a cluster is updated.

STRUCT(
    "broker_ebs_volume_info" STRUCT(
        "kafka_broker_node_id" VARCHAR,
        "provisioned_throughput" STRUCT(
            "enabled" BOOLEAN,
            "volume_throughput" BIGINT
        ),
        "volume_size_gb" BIGINT
    )[],
    "configuration_info" STRUCT(
        "arn" VARCHAR,
        "revision" BIGINT
    ),
    "number_of_broker_nodes" BIGINT,
    "enhanced_monitoring" VARCHAR,
    "open_monitoring" STRUCT(
        "prometheus" STRUCT(
            "jmx_exporter" STRUCT(
                "enabled_in_broker" BOOLEAN
            ),
            "node_exporter" STRUCT(
                "enabled_in_broker" BOOLEAN
            )
        )
    ),
    "kafka_version" VARCHAR,
    "logging_info" STRUCT(
        "broker_logs" STRUCT(
            "cloud_watch_logs" STRUCT(
                "enabled" BOOLEAN,
                "log_group" VARCHAR
            ),
            "firehose" STRUCT(
                "delivery_stream" VARCHAR,
                "enabled" BOOLEAN
            ),
            "s3" STRUCT(
                "bucket" VARCHAR,
                "enabled" BOOLEAN,
                "prefix" VARCHAR
            )
        )
    ),
    "instance_type" VARCHAR,
    "client_authentication" STRUCT(
        "sasl" STRUCT(
            "scram" STRUCT(
                "enabled" BOOLEAN
            ),
            "iam" STRUCT(
                "enabled" BOOLEAN
            )
        ),
        "tls" STRUCT(
            "certificate_authority_arn_list" VARCHAR[],
            "enabled" BOOLEAN
        ),
        "unauthenticated" STRUCT(
            "enabled" BOOLEAN
        )
    ),
    "encryption_info" STRUCT(
        "encryption_at_rest" STRUCT(
            "data_volume_kms_key_id" VARCHAR
        ),
        "encryption_in_transit" STRUCT(
            "client_broker" VARCHAR,
            "in_cluster" BOOLEAN
        )
    ),
    "connectivity_info" STRUCT(
        "public_access" STRUCT(
            "type" VARCHAR
        ),
        "vpc_connectivity" STRUCT(
            "client_authentication" STRUCT(
                "sasl" STRUCT(
                    "scram" STRUCT(
                        "enabled" BOOLEAN
                    ),
                    "iam" STRUCT(
                        "enabled" BOOLEAN
                    )
                ),
                "tls" STRUCT(
                    "enabled" BOOLEAN
                )
            )
        )
    ),
    "storage_mode" VARCHAR,
    "broker_count_update_info" STRUCT(
        "created_broker_ids" DOUBLE[],
        "deleted_broker_ids" DOUBLE[]
    )
)
Show child fields
target_cluster_info.broker_count_update_info

Describes brokers being changed during a broker count update.

Show child fields
target_cluster_info.broker_count_update_info.created_broker_ids[]
target_cluster_info.broker_count_update_info.deleted_broker_ids[]
target_cluster_info.broker_ebs_volume_info[]
Show child fields
target_cluster_info.broker_ebs_volume_info[].kafka_broker_node_id

The ID of the broker to update.

target_cluster_info.broker_ebs_volume_info[].provisioned_throughput

EBS volume provisioned throughput information.

Show child fields
target_cluster_info.broker_ebs_volume_info[].provisioned_throughput.enabled

Provisioned throughput is enabled or not.

target_cluster_info.broker_ebs_volume_info[].provisioned_throughput.volume_throughput

Throughput value of the EBS volumes for the data drive on each kafka broker node in MiB per second.

target_cluster_info.broker_ebs_volume_info[].volume_size_gb

Size of the EBS volume to update.

target_cluster_info.client_authentication

Includes all client authentication information.

Show child fields
target_cluster_info.client_authentication.sasl

Details for ClientAuthentication using SASL.

Show child fields
target_cluster_info.client_authentication.sasl.iam

Indicates whether IAM access control is enabled.

Show child fields
target_cluster_info.client_authentication.sasl.iam.enabled

Indicates whether IAM access control is enabled.

target_cluster_info.client_authentication.sasl.scram

Details for SASL/SCRAM client authentication.

Show child fields
target_cluster_info.client_authentication.sasl.scram.enabled

SASL/SCRAM authentication is enabled or not.

target_cluster_info.client_authentication.tls

Details for ClientAuthentication using TLS.

Show child fields
target_cluster_info.client_authentication.tls.certificate_authority_arn_list[]
target_cluster_info.client_authentication.tls.enabled

Specifies whether you want to turn on or turn off TLS authentication.

target_cluster_info.client_authentication.unauthenticated

Contains information about unauthenticated traffic to the cluster.

Show child fields
target_cluster_info.client_authentication.unauthenticated.enabled

Specifies whether you want to turn on or turn off unauthenticated traffic to your cluster.

target_cluster_info.configuration_info

Information about the changes in the configuration of the brokers.

Show child fields
target_cluster_info.configuration_info.arn

ARN of the configuration to use.

target_cluster_info.configuration_info.revision

The revision of the configuration to use.

target_cluster_info.connectivity_info

Information about the broker access configuration.

Show child fields
target_cluster_info.connectivity_info.public_access

Public access control for brokers.

Show child fields
target_cluster_info.connectivity_info.public_access.type

The value DISABLED indicates that public access is turned off. SERVICE_PROVIDED_EIPS indicates that public access is turned on.

target_cluster_info.connectivity_info.vpc_connectivity

VPC connectivity access control for brokers.

Show child fields
target_cluster_info.connectivity_info.vpc_connectivity.client_authentication

Includes all client authentication information for VPC connectivity.

Show child fields
target_cluster_info.connectivity_info.vpc_connectivity.client_authentication.sasl

SASL authentication type details for VPC connectivity.

Show child fields
target_cluster_info.connectivity_info.vpc_connectivity.client_authentication.sasl.iam

Details for SASL/IAM client authentication for VPC connectivity.

Show child fields
target_cluster_info.connectivity_info.vpc_connectivity.client_authentication.sasl.iam.enabled

SASL/IAM authentication is on or off for VPC connectivity.

target_cluster_info.connectivity_info.vpc_connectivity.client_authentication.sasl.scram

Details for SASL/SCRAM client authentication for VPC connectivity.

Show child fields
target_cluster_info.connectivity_info.vpc_connectivity.client_authentication.sasl.scram.enabled

SASL/SCRAM authentication is on or off for VPC connectivity.

target_cluster_info.connectivity_info.vpc_connectivity.client_authentication.tls

TLS authentication type details for VPC connectivity.

Show child fields
target_cluster_info.connectivity_info.vpc_connectivity.client_authentication.tls.enabled

TLS authentication is on or off for VPC connectivity.

target_cluster_info.encryption_info

Includes all encryption-related information.

Show child fields
target_cluster_info.encryption_info.encryption_at_rest

The data-volume encryption details.

Show child fields
target_cluster_info.encryption_info.encryption_at_rest.data_volume_kms_key_id

The ARN of the AWS KMS key for encrypting data at rest. If you don't specify a KMS key, MSK creates one for you and uses it.

target_cluster_info.encryption_info.encryption_in_transit

The details for encryption in transit.

Show child fields
target_cluster_info.encryption_info.encryption_in_transit.client_broker

Indicates the encryption setting for data in transit between clients and brokers. The following are the possible values.

TLS means that client-broker communication is enabled with TLS only.

TLS_PLAINTEXT means that client-broker communication is enabled for both TLS-encrypted, as well as plaintext data.

PLAINTEXT means that client-broker communication is enabled in plaintext only.

The default value is TLS_PLAINTEXT.

target_cluster_info.encryption_info.encryption_in_transit.in_cluster

When set to true, it indicates that data communication among the broker nodes of the cluster is encrypted. When set to false, the communication happens in plaintext.

The default value is true.

target_cluster_info.enhanced_monitoring

Specifies which Apache Kafka metrics Amazon MSK gathers and sends to Amazon CloudWatch for this cluster.

target_cluster_info.instance_type

Information about the Amazon MSK broker type.

target_cluster_info.kafka_version

The Apache Kafka version.

target_cluster_info.logging_info

You can configure your MSK cluster to send broker logs to different destination types. This is a container for the configuration details related to broker logs.

Show child fields
target_cluster_info.logging_info.broker_logs
Show child fields
target_cluster_info.logging_info.broker_logs.cloud_watch_logs
Show child fields
target_cluster_info.logging_info.broker_logs.cloud_watch_logs.enabled
target_cluster_info.logging_info.broker_logs.cloud_watch_logs.log_group
target_cluster_info.logging_info.broker_logs.firehose
Show child fields
target_cluster_info.logging_info.broker_logs.firehose.delivery_stream
target_cluster_info.logging_info.broker_logs.firehose.enabled
target_cluster_info.logging_info.broker_logs.s3
Show child fields
target_cluster_info.logging_info.broker_logs.s3.bucket
target_cluster_info.logging_info.broker_logs.s3.enabled
target_cluster_info.logging_info.broker_logs.s3.prefix
target_cluster_info.number_of_broker_nodes

The number of broker nodes in the cluster.

target_cluster_info.open_monitoring

The settings for open monitoring.

Show child fields
target_cluster_info.open_monitoring.prometheus

Prometheus settings.

Show child fields
target_cluster_info.open_monitoring.prometheus.jmx_exporter

Indicates whether you want to turn on or turn off the JMX Exporter.

Show child fields
target_cluster_info.open_monitoring.prometheus.jmx_exporter.enabled_in_broker

Indicates whether you want to turn on or turn off the JMX Exporter.

target_cluster_info.open_monitoring.prometheus.node_exporter

Indicates whether you want to turn on or turn off the Node Exporter.

Show child fields
target_cluster_info.open_monitoring.prometheus.node_exporter.enabled_in_broker

Indicates whether you want to turn on or turn off the Node Exporter.

target_cluster_info.storage_mode

This controls storage mode for supported storage tiers.

vpc_connection_info

Description of the VPC connection for CreateVpcConnection and DeleteVpcConnection operations.

STRUCT(
    "vpc_connection_arn" VARCHAR,
    "owner" VARCHAR,
    "user_identity" STRUCT(
        "type" VARCHAR,
        "principal_id" VARCHAR
    ),
    "creation_time" TIMESTAMP_S
)
Show child fields
vpc_connection_info.creation_time

The time when Amazon MSK creates the VPC Connnection.

vpc_connection_info.owner

The owner of the VPC Connection.

vpc_connection_info.user_identity

Description of the requester that calls the API operation.

Show child fields
vpc_connection_info.user_identity.principal_id

A unique identifier for the requester that calls the API operation.

vpc_connection_info.user_identity.type

The identity type of the requester that calls the API operation.

vpc_connection_info.vpc_connection_arn

The Amazon Resource Name (ARN) of the VPC connection.