Skip to content
AWS for Query.Farm

aws.neptunedata.get_propertygraph_stream

Example SQL Queries

SELECT * FROM
aws.neptunedata.get_propertygraph_stream;

Description

Gets a stream for a property graph.

With the Neptune Streams feature, you can generate a complete sequence of change-log entries that record every change made to your graph data as it happens. GetPropertygraphStream lets you collect these change-log entries for a property graph.

The Neptune streams feature needs to be enabled on your Neptune DBcluster. To enable streams, set the neptune_streams DB cluster parameter to 1.

See Capturing graph changes in real time using Neptune streams.

When invoking this operation in a Neptune cluster that has IAM authentication enabled, the IAM user or role making the request must have a policy attached that allows the neptune-db:GetStreamRecords IAM action in that cluster.

When invoking this operation in a Neptune cluster that has IAM authentication enabled, the IAM user or role making the request must have a policy attached that enables one of the following IAM actions, depending on the query:

Note that you can restrict property-graph queries using the following IAM context keys:

See Condition keys available in Neptune IAM data-access policy statements).

Table Definition

Column NameColumn Data Type
commit_num Input Column

The commit number of the starting record to read from the change-log stream. This parameter is required when iteratorType isAT_SEQUENCE_NUMBER or AFTER_SEQUENCE_NUMBER, and ignored when iteratorType is TRIM_HORIZON or LATEST.

BIGINT
encoding Input Column

If set to TRUE, Neptune compresses the response using gzip encoding.

VARCHAR
iterator_type Input Column

Can be one of:

  • AT_SEQUENCE_NUMBER   –   Indicates that reading should start from the event sequence number specified jointly by the commitNum and opNum parameters.

  • AFTER_SEQUENCE_NUMBER   –   Indicates that reading should start right after the event sequence number specified jointly by the commitNum and opNum parameters.

  • TRIM_HORIZON   –   Indicates that reading should start at the last untrimmed record in the system, which is the oldest unexpired (not yet deleted) record in the change-log stream.

  • LATEST   –   Indicates that reading should start at the most recent record in the system, which is the latest unexpired (not yet deleted) record in the change-log stream.

VARCHAR
limit Input Column

Specifies the maximum number of records to return. There is also a size limit of 10 MB on the response that can't be modified and that takes precedence over the number of records specified in the limit parameter. The response does include a threshold-breaching record if the 10 MB limit was reached.

The range for limit is 1 to 100,000, with a default of 10.

BIGINT
op_num Input Column

The operation sequence number within the specified commit to start reading from in the change-log stream data. The default is 1.

BIGINT
_aws_profile Input Column

The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role.

STRUCT(
    "type" VARCHAR,
    "name" VARCHAR,
    "account_id" VARCHAR,
    "via_profile_name" VARCHAR,
    "assumed_role_arn" VARCHAR,
    "organization" STRUCT(
        "account_name" VARCHAR,
        "id" VARCHAR,
        "tags" STRUCT(
            "key" VARCHAR,
            "value" VARCHAR
        )[],
        "master_account" STRUCT(
            "id" VARCHAR,
            "email" VARCHAR
        ),
        "parents" STRUCT(
            "type" VARCHAR,
            "id" VARCHAR,
            "name" VARCHAR,
            "tags" STRUCT(
                "key" VARCHAR,
                "value" VARCHAR
            )[]
        )[]
    )
)
Show child fields
_aws_profile.account_id

The AWS account id

_aws_profile.assumed_role_arn

The ARN of the assumed role

_aws_profile.name

The unique name of the profile.

_aws_profile.organization

Information about this profile's membership in the AWS organization.

Show child fields
_aws_profile.organization.account_name

The name of account speciifed by the organization

_aws_profile.organization.id

The organization id

_aws_profile.organization.master_account
Show child fields
_aws_profile.organization.master_account.email

The organization master account email address

_aws_profile.organization.master_account.id

The organization master account id

_aws_profile.organization.parents[]
Show child fields
_aws_profile.organization.parents[].id

The id of the parent

_aws_profile.organization.parents[].name

The name of the parent

_aws_profile.organization.parents[].tags[]
Show child fields
_aws_profile.organization.parents[].tags[].key
_aws_profile.organization.parents[].tags[].value
_aws_profile.organization.parents[].type

The type of parent can be an organization unit or a root

_aws_profile.organization.tags[]
Show child fields
_aws_profile.organization.tags[].key
_aws_profile.organization.tags[].value
_aws_profile.type

The type of profile, either 'credentials' or 'assumed_role'

_aws_profile.via_profile_name

This IAM role for this profile is assumed by first utilizing another profile with this name to obtain credentials.

format

Serialization format for the change records being returned. Currently, the only supported value is PG_JSON.

VARCHAR
last_event_id

Sequence identifier of the last change in the stream response.

An event ID is composed of two fields: a commitNum, which identifies a transaction that changed the graph, and an opNum, which identifies a specific operation within that transaction:

MAP(VARCHAR, VARCHAR)
last_trx_timestamp_in_millis

The time at which the commit for the transaction was requested, in milliseconds from the Unix epoch.

BIGINT
records

An array of serialized change-log stream records included in the response.

STRUCT(
    "commit_timestamp_in_millis" BIGINT,
    "event_id" MAP(VARCHAR, VARCHAR),
    "data" STRUCT(
        "id" VARCHAR,
        "type" VARCHAR,
        "key" VARCHAR,
        "value" VARCHAR,
        "from" VARCHAR,
        "to" VARCHAR
    ),
    "op" VARCHAR,
    "is_last_op" BOOLEAN
)[]
Show child fields
records[]
Show child fields
records[].commit_timestamp_in_millis

The time at which the commit for the transaction was requested, in milliseconds from the Unix epoch.

records[].data

The serialized Gremlin or openCypher change record.

Show child fields
records[].data.from

If this is an edge (type = e), the ID of the corresponding from vertex or source node.

records[].data.id

The ID of the Gremlin or openCypher element.

records[].data.key

The property name. For element labels, this is label.

records[].data.to

If this is an edge (type = e), the ID of the corresponding to vertex or target node.

records[].data.type

The type of this Gremlin or openCypher element. Must be one of:

  • v1   -   Vertex label for Gremlin, or node label for openCypher.

  • vp   -   Vertex properties for Gremlin, or node properties for openCypher.

  • e   -   Edge and edge label for Gremlin, or relationship and relationship type for openCypher.

  • ep   -   Edge properties for Gremlin, or relationship properties for openCypher.

records[].data.value

This is a JSON object that contains a value field for the value itself, and a datatype field for the JSON data type of that value:

records[].event_id

The sequence identifier of the stream change record.

records[].is_last_op

Only present if this operation is the last one in its transaction. If present, it is set to true. It is useful for ensuring that an entire transaction is consumed.

records[].op

The operation that created the change.

total_records

The total number of records in the response.

BIGINT