Skip to content
AWS for Query.Farm

aws.neptunedata.get_sparql_stream

Example SQL Queries

SELECT * FROM
aws.neptunedata.get_sparql_stream;

Description

Gets a stream for an RDF graph.

With the Neptune Streams feature, you can generate a complete sequence of change-log entries that record every change made to your graph data as it happens. GetSparqlStream lets you collect these change-log entries for an RDF graph.

The Neptune streams feature needs to be enabled on your Neptune DBcluster. To enable streams, set the neptune_streams DB cluster parameter to 1.

See Capturing graph changes in real time using Neptune streams.

When invoking this operation in a Neptune cluster that has IAM authentication enabled, the IAM user or role making the request must have a policy attached that allows the neptune-db:GetStreamRecords IAM action in that cluster.

Note that the neptune-db:QueryLanguage:Sparql IAM condition key can be used in the policy document to restrict the use of SPARQL queries (see Condition keys available in Neptune IAM data-access policy statements).

Table Definition

Column NameColumn Data Type
commit_num Input Column

The commit number of the starting record to read from the change-log stream. This parameter is required when iteratorType isAT_SEQUENCE_NUMBER or AFTER_SEQUENCE_NUMBER, and ignored when iteratorType is TRIM_HORIZON or LATEST.

BIGINT
encoding Input Column

If set to TRUE, Neptune compresses the response using gzip encoding.

VARCHAR
iterator_type Input Column

Can be one of:

  • AT_SEQUENCE_NUMBER   –   Indicates that reading should start from the event sequence number specified jointly by the commitNum and opNum parameters.

  • AFTER_SEQUENCE_NUMBER   –   Indicates that reading should start right after the event sequence number specified jointly by the commitNum and opNum parameters.

  • TRIM_HORIZON   –   Indicates that reading should start at the last untrimmed record in the system, which is the oldest unexpired (not yet deleted) record in the change-log stream.

  • LATEST   –   Indicates that reading should start at the most recent record in the system, which is the latest unexpired (not yet deleted) record in the change-log stream.

VARCHAR
limit Input Column

Specifies the maximum number of records to return. There is also a size limit of 10 MB on the response that can't be modified and that takes precedence over the number of records specified in the limit parameter. The response does include a threshold-breaching record if the 10 MB limit was reached.

The range for limit is 1 to 100,000, with a default of 10.

BIGINT
op_num Input Column

The operation sequence number within the specified commit to start reading from in the change-log stream data. The default is 1.

BIGINT
_aws_profile Input Column

The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role.

STRUCT(
    "type" VARCHAR,
    "name" VARCHAR,
    "account_id" VARCHAR,
    "via_profile_name" VARCHAR,
    "assumed_role_arn" VARCHAR,
    "organization" STRUCT(
        "account_name" VARCHAR,
        "id" VARCHAR,
        "tags" STRUCT(
            "key" VARCHAR,
            "value" VARCHAR
        )[],
        "master_account" STRUCT(
            "id" VARCHAR,
            "email" VARCHAR
        ),
        "parents" STRUCT(
            "type" VARCHAR,
            "id" VARCHAR,
            "name" VARCHAR,
            "tags" STRUCT(
                "key" VARCHAR,
                "value" VARCHAR
            )[]
        )[]
    )
)
Show child fields
_aws_profile.account_id

The AWS account id

_aws_profile.assumed_role_arn

The ARN of the assumed role

_aws_profile.name

The unique name of the profile.

_aws_profile.organization

Information about this profile's membership in the AWS organization.

Show child fields
_aws_profile.organization.account_name

The name of account speciifed by the organization

_aws_profile.organization.id

The organization id

_aws_profile.organization.master_account
Show child fields
_aws_profile.organization.master_account.email

The organization master account email address

_aws_profile.organization.master_account.id

The organization master account id

_aws_profile.organization.parents[]
Show child fields
_aws_profile.organization.parents[].id

The id of the parent

_aws_profile.organization.parents[].name

The name of the parent

_aws_profile.organization.parents[].tags[]
Show child fields
_aws_profile.organization.parents[].tags[].key
_aws_profile.organization.parents[].tags[].value
_aws_profile.organization.parents[].type

The type of parent can be an organization unit or a root

_aws_profile.organization.tags[]
Show child fields
_aws_profile.organization.tags[].key
_aws_profile.organization.tags[].value
_aws_profile.type

The type of profile, either 'credentials' or 'assumed_role'

_aws_profile.via_profile_name

This IAM role for this profile is assumed by first utilizing another profile with this name to obtain credentials.

format

Serialization format for the change records being returned. Currently, the only supported value is NQUADS.

VARCHAR
last_event_id

Sequence identifier of the last change in the stream response.

An event ID is composed of two fields: a commitNum, which identifies a transaction that changed the graph, and an opNum, which identifies a specific operation within that transaction:

MAP(VARCHAR, VARCHAR)
last_trx_timestamp_in_millis

The time at which the commit for the transaction was requested, in milliseconds from the Unix epoch.

BIGINT
records

An array of serialized change-log stream records included in the response.

STRUCT(
    "commit_timestamp_in_millis" BIGINT,
    "event_id" MAP(VARCHAR, VARCHAR),
    "data" STRUCT(
        "stmt" VARCHAR
    ),
    "op" VARCHAR,
    "is_last_op" BOOLEAN
)[]
Show child fields
records[]
Show child fields
records[].commit_timestamp_in_millis

The time at which the commit for the transaction was requested, in milliseconds from the Unix epoch.

records[].data

The serialized SPARQL change record. The serialization formats of each record are described in more detail in Serialization Formats in Neptune Streams.

Show child fields
records[].data.stmt

Holds an N-QUADS statement expressing the changed quad.

records[].event_id

The sequence identifier of the stream change record.

records[].is_last_op

Only present if this operation is the last one in its transaction. If present, it is set to true. It is useful for ensuring that an entire transaction is consumed.

records[].op

The operation that created the change.

total_records

The total number of records in the response.

BIGINT