Skip to content
AWS for Query.Farm

aws.finspace.get_kx_cluster

Example SQL Queries

SELECT * FROM
aws.finspace.get_kx_cluster
WHERE
"environment_id" = 'VALUE'
AND "cluster_name" = 'VALUE';

Description

Retrieves information about a kdb cluster.

Table Definition

Column NameColumn Data Type
cluster_name Required Input Column

A unique name for the cluster.

VARCHAR
environment_id Required Input Column

A unique identifier for the kdb environment.

VARCHAR
_aws_profile Input Column

The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role.

STRUCT(
    "type" VARCHAR,
    "name" VARCHAR,
    "account_id" VARCHAR,
    "via_profile_name" VARCHAR,
    "assumed_role_arn" VARCHAR,
    "organization" STRUCT(
        "account_name" VARCHAR,
        "id" VARCHAR,
        "tags" STRUCT(
            "key" VARCHAR,
            "value" VARCHAR
        )[],
        "master_account" STRUCT(
            "id" VARCHAR,
            "email" VARCHAR
        ),
        "parents" STRUCT(
            "type" VARCHAR,
            "id" VARCHAR,
            "name" VARCHAR,
            "tags" STRUCT(
                "key" VARCHAR,
                "value" VARCHAR
            )[]
        )[]
    )
)
Show child fields
_aws_profile.account_id

The AWS account id

_aws_profile.assumed_role_arn

The ARN of the assumed role

_aws_profile.name

The unique name of the profile.

_aws_profile.organization

Information about this profile's membership in the AWS organization.

Show child fields
_aws_profile.organization.account_name

The name of account speciifed by the organization

_aws_profile.organization.id

The organization id

_aws_profile.organization.master_account
Show child fields
_aws_profile.organization.master_account.email

The organization master account email address

_aws_profile.organization.master_account.id

The organization master account id

_aws_profile.organization.parents[]
Show child fields
_aws_profile.organization.parents[].id

The id of the parent

_aws_profile.organization.parents[].name

The name of the parent

_aws_profile.organization.parents[].tags[]
Show child fields
_aws_profile.organization.parents[].tags[].key
_aws_profile.organization.parents[].tags[].value
_aws_profile.organization.parents[].type

The type of parent can be an organization unit or a root

_aws_profile.organization.tags[]
Show child fields
_aws_profile.organization.tags[].key
_aws_profile.organization.tags[].value
_aws_profile.type

The type of profile, either 'credentials' or 'assumed_role'

_aws_profile.via_profile_name

This IAM role for this profile is assumed by first utilizing another profile with this name to obtain credentials.

_aws_region Input Column

The AWS region to use.

VARCHAR
auto_scaling_configuration

The configuration based on which FinSpace will scale in or scale out nodes in your cluster.

STRUCT(
    "min_node_count" BIGINT,
    "max_node_count" BIGINT,
    "auto_scaling_metric" VARCHAR,
    "metric_target" DOUBLE,
    "scale_in_cooldown_seconds" DOUBLE,
    "scale_out_cooldown_seconds" DOUBLE
)
Show child fields
auto_scaling_configuration.auto_scaling_metric

The metric your cluster will track in order to scale in and out. For example, CPU_UTILIZATION_PERCENTAGE is the average CPU usage across all the nodes in a cluster.

auto_scaling_configuration.max_node_count

The highest number of nodes to scale. This value cannot be greater than 5.

auto_scaling_configuration.metric_target

The desired value of the chosen autoScalingMetric. When the metric drops below this value, the cluster will scale in. When the metric goes above this value, the cluster will scale out. You can set the target value between 1 and 100 percent.

auto_scaling_configuration.min_node_count

The lowest number of nodes to scale. This value must be at least 1 and less than the maxNodeCount. If the nodes in a cluster belong to multiple availability zones, then minNodeCount must be at least 3.

auto_scaling_configuration.scale_in_cooldown_seconds

The duration in seconds that FinSpace will wait after a scale in event before initiating another scaling event.

auto_scaling_configuration.scale_out_cooldown_seconds

The duration in seconds that FinSpace will wait after a scale out event before initiating another scaling event.

availability_zone_id

The availability zone identifiers for the requested regions. 

VARCHAR
az_mode

The number of availability zones you want to assign per cluster. This can be one of the following

  • SINGLE – Assigns one availability zone per cluster.

  • MULTI – Assigns all the availability zones per cluster.

VARCHAR
cache_storage_configurations

The configurations for a read only cache storage associated with a cluster. This cache will be stored as an FSx Lustre that reads from the S3 store. 

STRUCT(
    "type" VARCHAR,
    "size" BIGINT
)[]
Show child fields
cache_storage_configurations[]
Show child fields
cache_storage_configurations[].size

The size of cache in Gigabytes.

cache_storage_configurations[].type

The type of cache storage. The valid values are:

  • CACHE_1000 – This type provides at least 1000 MB/s disk access throughput.

  • CACHE_250 – This type provides at least 250 MB/s disk access throughput.

  • CACHE_12 – This type provides at least 12 MB/s disk access throughput.

For cache type CACHE_1000 and CACHE_250 you can select cache size as 1200 GB or increments of 2400 GB. For cache type CACHE_12 you can select the cache size in increments of 6000 GB.

capacity_configuration

A structure for the metadata of a cluster. It includes information like the CPUs needed, memory of instances, and number of instances.

STRUCT(
    "node_type" VARCHAR,
    "node_count" BIGINT
)
Show child fields
capacity_configuration.node_count

The number of instances running in a cluster.

capacity_configuration.node_type

The type that determines the hardware of the host computer used for your cluster instance. Each node type offers different memory and storage capabilities. Choose a node type based on the requirements of the application or software that you plan to run on your instance.

You can only specify one of the following values:

  • kx.s.large – The node type with a configuration of 12 GiB memory and 2 vCPUs.

  • kx.s.xlarge – The node type with a configuration of 27 GiB memory and 4 vCPUs.

  • kx.s.2xlarge – The node type with a configuration of 54 GiB memory and 8 vCPUs.

  • kx.s.4xlarge – The node type with a configuration of 108 GiB memory and 16 vCPUs.

  • kx.s.8xlarge – The node type with a configuration of 216 GiB memory and 32 vCPUs.

  • kx.s.16xlarge – The node type with a configuration of 432 GiB memory and 64 vCPUs.

  • kx.s.32xlarge – The node type with a configuration of 864 GiB memory and 128 vCPUs.

cluster_description

A description of the cluster.

VARCHAR
cluster_type

Specifies the type of KDB database that is being created. The following types are available:

  • HDB – A Historical Database. The data is only accessible with read-only permissions from one of the FinSpace managed kdb databases mounted to the cluster.

  • RDB – A Realtime Database. This type of database captures all the data from a ticker plant and stores it in memory until the end of day, after which it writes all of its data to a disk and reloads the HDB. This cluster type requires local storage for temporary storage of data during the savedown process. If you specify this field in your request, you must provide the savedownStorageConfiguration parameter.

  • GATEWAY – A gateway cluster allows you to access data across processes in kdb systems. It allows you to create your own routing logic using the initialization scripts and custom code. This type of cluster does not require a writable local storage.

  • GP – A general purpose cluster allows you to quickly iterate on code during development by granting greater access to system commands and enabling a fast reload of custom code. This cluster type can optionally mount databases including cache and savedown storage. For this cluster type, the node count is fixed at 1. It does not support autoscaling and supports only SINGLE AZ mode.

  • Tickerplant – A tickerplant cluster allows you to subscribe to feed handlers based on IAM permissions. It can publish to RDBs, other Tickerplants, and real-time subscribers (RTS). Tickerplants can persist messages to log, which is readable by any RDB environment. It supports only single-node that is only one kdb process.

VARCHAR
code

The details of the custom code that you want to use inside a cluster when analyzing a data. It consists of the S3 source bucket, location, S3 object version, and the relative path from where the custom code is loaded into the cluster. 

STRUCT(
    "s3_bucket" VARCHAR,
    "s3_key" VARCHAR,
    "s3_object_version" VARCHAR
)
Show child fields
code.s3_bucket

A unique name for the S3 bucket.

code.s3_key

The full S3 path (excluding bucket) to the .zip file. This file contains the code that is loaded onto the cluster when it's started.

code.s3_object_version

The version of an S3 object.

command_line_arguments

Defines key-value pairs to make them available inside the cluster.

STRUCT(
    "key" VARCHAR,
    "value" VARCHAR
)[]
Show child fields
command_line_arguments[]
Show child fields
command_line_arguments[].key

The name of the key.

command_line_arguments[].value

The value of the key.

created_timestamp

The timestamp at which the cluster was created in FinSpace. The value is determined as epoch time in milliseconds. For example, the value for Monday, November 1, 2021 12:00:00 PM UTC is specified as 1635768000000.

TIMESTAMP_S
databases

A list of databases mounted on the cluster.

STRUCT(
    "database_name" VARCHAR,
    "cache_configurations" STRUCT(
        "cache_type" VARCHAR,
        "db_paths" VARCHAR[],
        "dataview_name" VARCHAR
    )[],
    "changeset_id" VARCHAR,
    "dataview_name" VARCHAR,
    "dataview_configuration" STRUCT(
        "dataview_name" VARCHAR,
        "dataview_version_id" VARCHAR,
        "changeset_id" VARCHAR,
        "segment_configurations" STRUCT(
            "db_paths" VARCHAR[],
            "volume_name" VARCHAR,
            "on_demand" BOOLEAN
        )[]
    )
)[]
Show child fields
databases[]
Show child fields
databases[].cache_configurations[]
Show child fields
databases[].cache_configurations[].cache_type

The type of disk cache. This parameter is used to map the database path to cache storage. The valid values are:

  • CACHE_1000 – This type provides at least 1000 MB/s disk access throughput.

databases[].cache_configurations[].dataview_name

The name of the dataview to be used for caching historical data on disk.

databases[].cache_configurations[].db_paths[]
databases[].changeset_id

A unique identifier of the changeset that is associated with the cluster.

databases[].database_name

The name of the kdb database. When this parameter is specified in the structure, S3 with the whole database is included by default.

databases[].dataview_configuration

The configuration of the dataview to be used with specified cluster.

Show child fields
databases[].dataview_configuration.changeset_id

A unique identifier for the changeset.

databases[].dataview_configuration.dataview_name

The unique identifier of the dataview.

databases[].dataview_configuration.dataview_version_id

The version of the dataview corresponding to a given changeset.

databases[].dataview_configuration.segment_configurations[]
Show child fields
databases[].dataview_configuration.segment_configurations[].db_paths[]
databases[].dataview_configuration.segment_configurations[].on_demand

Enables on-demand caching on the selected database path when a particular file or a column of the database is accessed. When on demand caching is True, dataviews perform minimal loading of files on the filesystem as needed. When it is set to False, everything is cached. The default value is False.

databases[].dataview_configuration.segment_configurations[].volume_name

The name of the volume where you want to add data.

databases[].dataview_name

The name of the dataview to be used for caching historical data on disk.

execution_role

An IAM role that defines a set of permissions associated with a cluster. These permissions are assumed when a cluster attempts to access another cluster. 

VARCHAR
initialization_script

Specifies a Q program that will be run at launch of a cluster. It is a relative path within .zip file that contains the custom code, which will be loaded on the cluster. It must include the file name itself. For example, somedir/init.q.

VARCHAR
last_modified_timestamp

The last time that the cluster was modified. The value is determined as epoch time in milliseconds. For example, the value for Monday, November 1, 2021 12:00:00 PM UTC is specified as 1635768000000.

TIMESTAMP_S
release_label

The version of FinSpace managed kdb to run.

VARCHAR
savedown_storage_configuration

The size and type of the temporary storage that is used to hold data during the savedown process. This parameter is required when you choose clusterType as RDB. All the data written to this storage space is lost when the cluster node is restarted.

STRUCT(
    "type" VARCHAR,
    "size" BIGINT,
    "volume_name" VARCHAR
)
Show child fields
savedown_storage_configuration.size

The size of temporary storage in gibibytes.

savedown_storage_configuration.type

The type of writeable storage space for temporarily storing your savedown data. The valid values are:

  • SDS01 – This type represents 3000 IOPS and io2 ebs volume type.

savedown_storage_configuration.volume_name

The name of the kdb volume that you want to use as writeable save-down storage for clusters.

scaling_group_configuration

The structure that stores the capacity configuration details of a scaling group.

STRUCT(
    "scaling_group_name" VARCHAR,
    "memory_limit" BIGINT,
    "memory_reservation" BIGINT,
    "node_count" BIGINT,
    "cpu" DOUBLE
)
Show child fields
scaling_group_configuration.cpu

The number of vCPUs that you want to reserve for each node of this kdb cluster on the scaling group host.

scaling_group_configuration.memory_limit

An optional hard limit on the amount of memory a kdb cluster can use.

scaling_group_configuration.memory_reservation

A reservation of the minimum amount of memory that should be available on the scaling group for a kdb cluster to be successfully placed in a scaling group.

scaling_group_configuration.node_count

The number of kdb cluster nodes.

scaling_group_configuration.scaling_group_name

A unique identifier for the kdb scaling group.

status

The status of cluster creation.

  • PENDING – The cluster is pending creation.

  • CREATING – The cluster creation process is in progress.

  • CREATE_FAILED – The cluster creation process has failed.

  • RUNNING – The cluster creation process is running.

  • UPDATING – The cluster is in the process of being updated.

  • DELETING – The cluster is in the process of being deleted.

  • DELETED – The cluster has been deleted.

  • DELETE_FAILED – The cluster failed to delete.

VARCHAR
status_reason

The error message when a failed state occurs. 

VARCHAR
tickerplant_log_configuration

A configuration to store the Tickerplant logs. It consists of a list of volumes that will be mounted to your cluster. For the cluster type Tickerplant, the location of the TP volume on the cluster will be available by using the global variable .aws.tp_log_path. 

STRUCT(
    "tickerplant_log_volumes" VARCHAR[]
)
Show child fields
tickerplant_log_configuration.tickerplant_log_volumes[]
volumes

A list of volumes attached to the cluster. 

STRUCT(
    "volume_name" VARCHAR,
    "volume_type" VARCHAR
)[]
Show child fields
volumes[]
Show child fields
volumes[].volume_name

A unique identifier for the volume.

volumes[].volume_type

The type of file system volume. Currently, FinSpace only supports NAS_1 volume type.

vpc_configuration

Configuration details about the network where the Privatelink endpoint of the cluster resides.

STRUCT(
    "vpc_id" VARCHAR,
    "security_group_ids" VARCHAR[],
    "subnet_ids" VARCHAR[],
    "ip_address_type" VARCHAR
)
Show child fields
vpc_configuration.ip_address_type

The IP address type for cluster network configuration parameters. The following type is available:

  • IP_V4 – IP address version 4

vpc_configuration.security_group_ids[]
vpc_configuration.subnet_ids[]
vpc_configuration.vpc_id

The identifier of the VPC endpoint.