Skip to content

aws.kafka.list_clusters_v2

Example SQL Queries

SELECT * FROM
aws.kafka.list_clusters_v2;

Description

Returns a list of all the MSK clusters in the current Region.

Table Definition

Column NameColumn Data Type
cluster_name_filter Input Column

Specify a prefix of the names of the clusters that you want to list. The service lists all the clusters whose names start with this prefix.

VARCHAR
cluster_type_filter Input Column

Specify either PROVISIONED or SERVERLESS.

VARCHAR
_aws_profile Input Column

The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role.

STRUCT(
"type" VARCHAR,
"name" VARCHAR,
"account_id" VARCHAR,
"via_profile_name" VARCHAR,
"assumed_role_arn" VARCHAR,
"organization" STRUCT(
"account_name" VARCHAR,
"id" VARCHAR,
"tags" STRUCT(
"key" VARCHAR,
"value" VARCHAR
)[],
"master_account" STRUCT(
"id" VARCHAR,
"email" VARCHAR
),
"parents" STRUCT(
"type" VARCHAR,
"id" VARCHAR,
"name" VARCHAR,
"tags" STRUCT(
"key" VARCHAR,
"value" VARCHAR
)[]
)[]
)
)
Show child fields
_aws_profile.account_id

The AWS account id

_aws_profile.assumed_role_arn

The ARN of the assumed role

_aws_profile.name

The unique name of the profile.

_aws_profile.organization

Information about this profile's membership in the AWS organization.

Show child fields
_aws_profile.organization.account_name

The name of account speciifed by the organization

_aws_profile.organization.id

The organization id

_aws_profile.organization.master_account
Show child fields
_aws_profile.organization.master_account.email

The organization master account email address

_aws_profile.organization.master_account.id

The organization master account id

_aws_profile.organization.parents[]
Show child fields
_aws_profile.organization.parents[].id

The id of the parent

_aws_profile.organization.parents[].name

The name of the parent

_aws_profile.organization.parents[].tags[]
Show child fields
_aws_profile.organization.parents[].tags[].key
_aws_profile.organization.parents[].tags[].value
_aws_profile.organization.parents[].type

The type of parent can be an organization unit or a root

_aws_profile.organization.tags[]
Show child fields
_aws_profile.organization.tags[].key
_aws_profile.organization.tags[].value
_aws_profile.type

The type of profile, either 'credentials' or 'assumed_role'

_aws_profile.via_profile_name

This IAM role for this profile is assumed by first utilizing another profile with this name to obtain credentials.

_aws_region Input Column

The AWS region to use.

VARCHAR
active_operation_arn

The Amazon Resource Name (ARN) that uniquely identifies a cluster operation.

VARCHAR
cluster_arn

The Amazon Resource Name (ARN) that uniquely identifies the cluster.

VARCHAR
cluster_name

The name of the cluster.

VARCHAR
cluster_type

Cluster Type.

VARCHAR
creation_time

The time when the cluster was created.

TIMESTAMP_S
current_version

The current version of the MSK cluster.

VARCHAR
provisioned

Information about the provisioned cluster.

STRUCT(
"broker_node_group_info" STRUCT(
"broker_az_distribution" VARCHAR,
"client_subnets" VARCHAR[],
"instance_type" VARCHAR,
"security_groups" VARCHAR[],
"storage_info" STRUCT(
"ebs_storage_info" STRUCT(
"provisioned_throughput" STRUCT(
"enabled" BOOLEAN,
"volume_throughput" BIGINT
),
"volume_size" BIGINT
)
),
"connectivity_info" STRUCT(
"public_access" STRUCT(
"type" VARCHAR
),
"vpc_connectivity" STRUCT(
"client_authentication" STRUCT(
"sasl" STRUCT(
"scram" STRUCT(
"enabled" BOOLEAN
),
"iam" STRUCT(
"enabled" BOOLEAN
)
),
"tls" STRUCT(
"enabled" BOOLEAN
)
)
)
),
"zone_ids" VARCHAR[]
),
"current_broker_software_info" STRUCT(
"configuration_arn" VARCHAR,
"configuration_revision" BIGINT,
"kafka_version" VARCHAR
),
"client_authentication" STRUCT(
"sasl" STRUCT(
"scram" STRUCT(
"enabled" BOOLEAN
),
"iam" STRUCT(
"enabled" BOOLEAN
)
),
"tls" STRUCT(
"certificate_authority_arn_list" VARCHAR[],
"enabled" BOOLEAN
),
"unauthenticated" STRUCT(
"enabled" BOOLEAN
)
),
"encryption_info" STRUCT(
"encryption_at_rest" STRUCT(
"data_volume_kms_key_id" VARCHAR
),
"encryption_in_transit" STRUCT(
"client_broker" VARCHAR,
"in_cluster" BOOLEAN
)
),
"enhanced_monitoring" VARCHAR,
"open_monitoring" STRUCT(
"prometheus" STRUCT(
"jmx_exporter" STRUCT(
"enabled_in_broker" BOOLEAN
),
"node_exporter" STRUCT(
"enabled_in_broker" BOOLEAN
)
)
),
"logging_info" STRUCT(
"broker_logs" STRUCT(
"cloud_watch_logs" STRUCT(
"enabled" BOOLEAN,
"log_group" VARCHAR
),
"firehose" STRUCT(
"delivery_stream" VARCHAR,
"enabled" BOOLEAN
),
"s3" STRUCT(
"bucket" VARCHAR,
"enabled" BOOLEAN,
"prefix" VARCHAR
)
)
),
"number_of_broker_nodes" BIGINT,
"zookeeper_connect_string" VARCHAR,
"zookeeper_connect_string_tls" VARCHAR,
"storage_mode" VARCHAR,
"customer_action_status" VARCHAR
)
Show child fields
provisioned.broker_node_group_info

Information about the brokers.

Show child fields
provisioned.broker_node_group_info.broker_az_distribution

The distribution of broker nodes across Availability Zones. This is an optional parameter. If you don't specify it, Amazon MSK gives it the value DEFAULT. You can also explicitly set this parameter to the value DEFAULT. No other values are currently allowed.

Amazon MSK distributes the broker nodes evenly across the Availability Zones that correspond to the subnets you provide when you create the cluster.

provisioned.broker_node_group_info.client_subnets[]
provisioned.broker_node_group_info.connectivity_info

Information about the broker access configuration.

Show child fields
provisioned.broker_node_group_info.connectivity_info.public_access

Public access control for brokers.

Show child fields
provisioned.broker_node_group_info.connectivity_info.public_access.type

The value DISABLED indicates that public access is turned off. SERVICE_PROVIDED_EIPS indicates that public access is turned on.

provisioned.broker_node_group_info.connectivity_info.vpc_connectivity

VPC connectivity access control for brokers.

Show child fields
provisioned.broker_node_group_info.connectivity_info.vpc_connectivity.client_authentication

Includes all client authentication information for VPC connectivity.

Show child fields
provisioned.broker_node_group_info.connectivity_info.vpc_connectivity.client_authentication.sasl

SASL authentication type details for VPC connectivity.

Show child fields
provisioned.broker_node_group_info.connectivity_info.vpc_connectivity.client_authentication.sasl.iam

Details for SASL/IAM client authentication for VPC connectivity.

Show child fields
provisioned.broker_node_group_info.connectivity_info.vpc_connectivity.client_authentication.sasl.iam.enabled

SASL/IAM authentication is on or off for VPC connectivity.

provisioned.broker_node_group_info.connectivity_info.vpc_connectivity.client_authentication.sasl.scram

Details for SASL/SCRAM client authentication for VPC connectivity.

Show child fields
provisioned.broker_node_group_info.connectivity_info.vpc_connectivity.client_authentication.sasl.scram.enabled

SASL/SCRAM authentication is on or off for VPC connectivity.

provisioned.broker_node_group_info.connectivity_info.vpc_connectivity.client_authentication.tls

TLS authentication type details for VPC connectivity.

Show child fields
provisioned.broker_node_group_info.connectivity_info.vpc_connectivity.client_authentication.tls.enabled

TLS authentication is on or off for VPC connectivity.

provisioned.broker_node_group_info.instance_type

The type of Amazon EC2 instances to use for Apache Kafka brokers. The following instance types are allowed: kafka.m5.large, kafka.m5.xlarge, kafka.m5.2xlarge, kafka.m5.4xlarge, kafka.m5.12xlarge, and kafka.m5.24xlarge.

provisioned.broker_node_group_info.security_groups[]
provisioned.broker_node_group_info.storage_info

Contains information about storage volumes attached to MSK broker nodes.

Show child fields
provisioned.broker_node_group_info.storage_info.ebs_storage_info

EBS volume information.

Show child fields
provisioned.broker_node_group_info.storage_info.ebs_storage_info.provisioned_throughput

EBS volume provisioned throughput information.

Show child fields
provisioned.broker_node_group_info.storage_info.ebs_storage_info.provisioned_throughput.enabled

Provisioned throughput is enabled or not.

provisioned.broker_node_group_info.storage_info.ebs_storage_info.provisioned_throughput.volume_throughput

Throughput value of the EBS volumes for the data drive on each kafka broker node in MiB per second.

provisioned.broker_node_group_info.storage_info.ebs_storage_info.volume_size

The size in GiB of the EBS volume for the data drive on each broker node.

provisioned.broker_node_group_info.zone_ids[]
provisioned.client_authentication

Includes all client authentication information.

Show child fields
provisioned.client_authentication.sasl

Details for ClientAuthentication using SASL.

Show child fields
provisioned.client_authentication.sasl.iam

Indicates whether IAM access control is enabled.

Show child fields
provisioned.client_authentication.sasl.iam.enabled

Indicates whether IAM access control is enabled.

provisioned.client_authentication.sasl.scram

Details for SASL/SCRAM client authentication.

Show child fields
provisioned.client_authentication.sasl.scram.enabled

SASL/SCRAM authentication is enabled or not.

provisioned.client_authentication.tls

Details for ClientAuthentication using TLS.

Show child fields
provisioned.client_authentication.tls.certificate_authority_arn_list[]
provisioned.client_authentication.tls.enabled

Specifies whether you want to turn on or turn off TLS authentication.

provisioned.client_authentication.unauthenticated

Contains information about unauthenticated traffic to the cluster.

Show child fields
provisioned.client_authentication.unauthenticated.enabled

Specifies whether you want to turn on or turn off unauthenticated traffic to your cluster.

provisioned.current_broker_software_info

Information about the Apache Kafka version deployed on the brokers.

Show child fields
provisioned.current_broker_software_info.configuration_arn

The Amazon Resource Name (ARN) of the configuration used for the cluster. This field isn't visible in this preview release.

provisioned.current_broker_software_info.configuration_revision

The revision of the configuration to use. This field isn't visible in this preview release.

provisioned.current_broker_software_info.kafka_version

The version of Apache Kafka.

provisioned.customer_action_status

Determines if there is an action required from the customer.

provisioned.encryption_info

Includes all encryption-related information.

Show child fields
provisioned.encryption_info.encryption_at_rest

The data-volume encryption details.

Show child fields
provisioned.encryption_info.encryption_at_rest.data_volume_kms_key_id

The ARN of the AWS KMS key for encrypting data at rest. If you don't specify a KMS key, MSK creates one for you and uses it.

provisioned.encryption_info.encryption_in_transit

The details for encryption in transit.

Show child fields
provisioned.encryption_info.encryption_in_transit.client_broker

Indicates the encryption setting for data in transit between clients and brokers. The following are the possible values.

TLS means that client-broker communication is enabled with TLS only.

TLS_PLAINTEXT means that client-broker communication is enabled for both TLS-encrypted, as well as plaintext data.

PLAINTEXT means that client-broker communication is enabled in plaintext only.

The default value is TLS_PLAINTEXT.

provisioned.encryption_info.encryption_in_transit.in_cluster

When set to true, it indicates that data communication among the broker nodes of the cluster is encrypted. When set to false, the communication happens in plaintext.

The default value is true.

provisioned.enhanced_monitoring

Specifies the level of monitoring for the MSK cluster. The possible values are DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER, and PER_TOPIC_PER_PARTITION.

provisioned.logging_info

Log delivery information for the cluster.

Show child fields
provisioned.logging_info.broker_logs
Show child fields
provisioned.logging_info.broker_logs.cloud_watch_logs
Show child fields
provisioned.logging_info.broker_logs.cloud_watch_logs.enabled
provisioned.logging_info.broker_logs.cloud_watch_logs.log_group
provisioned.logging_info.broker_logs.firehose
Show child fields
provisioned.logging_info.broker_logs.firehose.delivery_stream
provisioned.logging_info.broker_logs.firehose.enabled
provisioned.logging_info.broker_logs.s3
Show child fields
provisioned.logging_info.broker_logs.s3.bucket
provisioned.logging_info.broker_logs.s3.enabled
provisioned.logging_info.broker_logs.s3.prefix
provisioned.number_of_broker_nodes

The number of broker nodes in the cluster.

provisioned.open_monitoring

The settings for open monitoring.

Show child fields
provisioned.open_monitoring.prometheus

Prometheus settings.

Show child fields
provisioned.open_monitoring.prometheus.jmx_exporter

Indicates whether you want to turn on or turn off the JMX Exporter.

Show child fields
provisioned.open_monitoring.prometheus.jmx_exporter.enabled_in_broker

Indicates whether you want to turn on or turn off the JMX Exporter.

provisioned.open_monitoring.prometheus.node_exporter

Indicates whether you want to turn on or turn off the Node Exporter.

Show child fields
provisioned.open_monitoring.prometheus.node_exporter.enabled_in_broker

Indicates whether you want to turn on or turn off the Node Exporter.

provisioned.storage_mode

This controls storage mode for supported storage tiers.

provisioned.zookeeper_connect_string

The connection string to use to connect to the Apache ZooKeeper cluster.

provisioned.zookeeper_connect_string_tls

The connection string to use to connect to the Apache ZooKeeper cluster on a TLS port.

serverless

Information about the serverless cluster.

STRUCT(
"vpc_configs" STRUCT(
"subnet_ids" VARCHAR[],
"security_group_ids" VARCHAR[]
)[],
"client_authentication" STRUCT(
"sasl" STRUCT(
"iam" STRUCT(
"enabled" BOOLEAN
)
)
)
)
Show child fields
serverless.client_authentication

Includes all client authentication information.

Show child fields
serverless.client_authentication.sasl

Details for ClientAuthentication using SASL.

Show child fields
serverless.client_authentication.sasl.iam

Indicates whether IAM access control is enabled.

Show child fields
serverless.client_authentication.sasl.iam.enabled

Indicates whether IAM access control is enabled.

serverless.vpc_configs[]
Show child fields
serverless.vpc_configs[].security_group_ids[]
serverless.vpc_configs[].subnet_ids[]
state

The state of the cluster. The possible states are ACTIVE, CREATING, DELETING, FAILED, HEALING, MAINTENANCE, REBOOTING_BROKER, and UPDATING.

VARCHAR
state_info

State Info for the Amazon MSK cluster.

STRUCT(
"code" VARCHAR,
"message" VARCHAR
)
Show child fields
state_info.code
state_info.message
tags

Tags attached to the cluster.

MAP(VARCHAR, VARCHAR)