Skip to content
AWS for Query.Farm

aws.forecast.get_accuracy_metrics

Example SQL Queries

SELECT * FROM
aws.forecast.get_accuracy_metrics
WHERE
"predictor_arn" = 'VALUE';

Description

Provides metrics on the accuracy of the models that were trained by the CreatePredictor operation. Use metrics to see how well the model performed and to decide whether to use the predictor to generate a forecast. For more information, see Predictor Metrics.

This operation generates metrics for each backtest window that was evaluated. The number of backtest windows (NumberOfBacktestWindows) is specified using the EvaluationParameters object, which is optionally included in the CreatePredictor request. If NumberOfBacktestWindows isn't specified, the number defaults to one.

The parameters of the filling method determine which items contribute to the metrics. If you want all items to contribute, specify zero. If you want only those items that have complete data in the range being evaluated to contribute, specify nan. For more information, see FeaturizationMethod.

Before you can get accuracy metrics, the Status of the predictor must be ACTIVE, signifying that training has completed. To get the status, use the DescribePredictor operation.

Table Definition

Column NameColumn Data Type
predictor_arn Required Input Column

The Amazon Resource Name (ARN) of the predictor to get metrics for.

VARCHAR
_aws_profile Input Column

The AWS profile defines the AWS identity used. It can be defined via credentials or by assuming a IAM role.

STRUCT(
    "type" VARCHAR,
    "name" VARCHAR,
    "account_id" VARCHAR,
    "via_profile_name" VARCHAR,
    "assumed_role_arn" VARCHAR,
    "organization" STRUCT(
        "account_name" VARCHAR,
        "id" VARCHAR,
        "tags" STRUCT(
            "key" VARCHAR,
            "value" VARCHAR
        )[],
        "master_account" STRUCT(
            "id" VARCHAR,
            "email" VARCHAR
        ),
        "parents" STRUCT(
            "type" VARCHAR,
            "id" VARCHAR,
            "name" VARCHAR,
            "tags" STRUCT(
                "key" VARCHAR,
                "value" VARCHAR
            )[]
        )[]
    )
)
Show child fields
_aws_profile.account_id

The AWS account id

_aws_profile.assumed_role_arn

The ARN of the assumed role

_aws_profile.name

The unique name of the profile.

_aws_profile.organization

Information about this profile's membership in the AWS organization.

Show child fields
_aws_profile.organization.account_name

The name of account speciifed by the organization

_aws_profile.organization.id

The organization id

_aws_profile.organization.master_account
Show child fields
_aws_profile.organization.master_account.email

The organization master account email address

_aws_profile.organization.master_account.id

The organization master account id

_aws_profile.organization.parents[]
Show child fields
_aws_profile.organization.parents[].id

The id of the parent

_aws_profile.organization.parents[].name

The name of the parent

_aws_profile.organization.parents[].tags[]
Show child fields
_aws_profile.organization.parents[].tags[].key
_aws_profile.organization.parents[].tags[].value
_aws_profile.organization.parents[].type

The type of parent can be an organization unit or a root

_aws_profile.organization.tags[]
Show child fields
_aws_profile.organization.tags[].key
_aws_profile.organization.tags[].value
_aws_profile.type

The type of profile, either 'credentials' or 'assumed_role'

_aws_profile.via_profile_name

This IAM role for this profile is assumed by first utilizing another profile with this name to obtain credentials.

_aws_region Input Column

The AWS region to use.

VARCHAR
auto_ml_override_strategy

The LatencyOptimized AutoML override strategy is only available in private beta. Contact Amazon Web Services Support or your account manager to learn more about access privileges.

The AutoML strategy used to train the predictor. Unless LatencyOptimized is specified, the AutoML strategy optimizes predictor accuracy.

This parameter is only valid for predictors trained using AutoML.

VARCHAR
is_auto_predictor

Whether the predictor was created with CreateAutoPredictor.

BOOLEAN
optimization_metric

The accuracy metric used to optimize the predictor.

VARCHAR
predictor_evaluation_results

An array of results from evaluating the predictor.

STRUCT(
    "algorithm_arn" VARCHAR,
    "test_windows" STRUCT(
        "test_window_start" TIMESTAMP_S,
        "test_window_end" TIMESTAMP_S,
        "item_count" BIGINT,
        "evaluation_type" VARCHAR,
        "metrics" STRUCT(
            "rmse" DOUBLE,
            "weighted_quantile_losses" STRUCT(
                "quantile" DOUBLE,
                "loss_value" DOUBLE
            )[],
            "error_metrics" STRUCT(
                "forecast_type" VARCHAR,
                "wape" DOUBLE,
                "rmse" DOUBLE,
                "mase" DOUBLE,
                "mape" DOUBLE
            )[],
            "average_weighted_quantile_loss" DOUBLE
        )
    )[]
)[]
Show child fields
predictor_evaluation_results[]
Show child fields
predictor_evaluation_results[].algorithm_arn

The Amazon Resource Name (ARN) of the algorithm that was evaluated.

predictor_evaluation_results[].test_windows[]
Show child fields
predictor_evaluation_results[].test_windows[].evaluation_type

The type of evaluation.

  • SUMMARY - The average metrics across all windows.

  • COMPUTED - The metrics for the specified window.

predictor_evaluation_results[].test_windows[].item_count

The number of data points within the window.

predictor_evaluation_results[].test_windows[].metrics

Provides metrics used to evaluate the performance of a predictor.

Show child fields
predictor_evaluation_results[].test_windows[].metrics.average_weighted_quantile_loss

The average value of all weighted quantile losses.

predictor_evaluation_results[].test_windows[].metrics.error_metrics[]
Show child fields
predictor_evaluation_results[].test_windows[].metrics.error_metrics[].forecast_type

The Forecast type used to compute WAPE, MAPE, MASE, and RMSE.

predictor_evaluation_results[].test_windows[].metrics.error_metrics[].mape

The Mean Absolute Percentage Error (MAPE)

predictor_evaluation_results[].test_windows[].metrics.error_metrics[].mase

The Mean Absolute Scaled Error (MASE)

predictor_evaluation_results[].test_windows[].metrics.error_metrics[].rmse

The root-mean-square error (RMSE).

predictor_evaluation_results[].test_windows[].metrics.error_metrics[].wape

The weighted absolute percentage error (WAPE).

predictor_evaluation_results[].test_windows[].metrics.rmse

The root-mean-square error (RMSE).

predictor_evaluation_results[].test_windows[].metrics.weighted_quantile_losses[]
Show child fields
predictor_evaluation_results[].test_windows[].metrics.weighted_quantile_losses[].loss_value

The difference between the predicted value and the actual value over the quantile, weighted (normalized) by dividing by the sum over all quantiles.

predictor_evaluation_results[].test_windows[].metrics.weighted_quantile_losses[].quantile

The quantile. Quantiles divide a probability distribution into regions of equal probability. For example, if the distribution was divided into 5 regions of equal probability, the quantiles would be 0.2, 0.4, 0.6, and 0.8.

predictor_evaluation_results[].test_windows[].test_window_end

The timestamp that defines the end of the window.

predictor_evaluation_results[].test_windows[].test_window_start

The timestamp that defines the start of the window.