Uncertainty Quantification

There are always uncertainties in computational predictions, it could be related to the sparsity of observed instances or the choice of hyperparameters and models. While it is impossible to make accurate point prediction, we can try to minimise prediction error rate by reducing uncertainties. Uncertainty quantification is a salient statistical technique to include uncertainty in the model training process so to provide a more accurate and reliable output for users.

With the NeuralProphet default configuration, a single value is predicted for each individual instance. The prediction output is just a point estimator without any consideration on uncertainties. Prediction intervals instead provide a more accurate estimation by quantifying the uncertainty and a possible value range for every single individual instance.

In this session, we will be introducing you to the two statistical techniques available in NeuralProphet: (1) quantile regression and (2) conformal prediction. These two modules are not mutually exclusive, you may apply both modules on top of any models.

The quantile regression module allows the algorithm to learn only a certain quantile of output variables for each instance. As for the conformal prediction module, it adds a calibration process on top of the model to quantify uncertainties in data for both point estimators and prediction intervals. You can also find more information about the concept quantile regression and conformal prediction by clicking their respective links.

We will illustrate and further elaborate on both quantification modules using the hospital electric load dataset. The dataset has recorded the electricity consumption of a hospital in SF in 2015 by hour.

# much faster using the following code, but may not have the latest upgrades/bugfixes
# pip install neuralprophet

if "google.colab" in str(get_ipython()):
    # uninstall preinstalled packages from Colab to avoid conflicts
    !pip uninstall -y torch notebook notebook_shim tensorflow tensorflow-datasets prophet torchaudio torchdata torchtext torchvision
    !pip install git+https://github.com/ourownstory/neural_prophet.git  # may take a while

import numpy as np
import pandas as pd
from neuralprophet import NeuralProphet, uncertainty_evaluate, set_log_level, set_random_seed

data_location = "http://raw.githubusercontent.com/ourownstory/neuralprophet-data/main/datasets/"
df = pd.read_csv(data_location + "energy/SF_hospital_load.csv")

Data splitting

In NeuralProphet, there is a data splitting function which divide a dataset input into two subsets. You can configure the function by indicating the time-series frequencies and splitting ratio. A list of frequency aliases can be found here.

In our hospital electric load dataset, we will divide the original dataset into training and testing set with a train-test ratio of \(1/16\).

# Create NeuralProphet object
m = NeuralProphet()

# Data spliting function split one time-series dataframe into two
# Configure the hourly frequency by assigning 'H' to parameter freq
# Configure the splitting ratio with a value between 0 and 1 for valid_p
train_df, test_df = m.split_df(df, freq="H", valid_p=1.0 / 16)

INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.989% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column
INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column

After this splitting, we will have 8213 and 547 instances extracted in sequence as training and testing set respectively.

train_df.shape, test_df.shape

((8213, 2), (547, 2))

1. Quantile Regression

By the NeuralProphet forecasting default setting, you are only getting a single output as the point estimator for each instance. The point estimator is calculated based on a single 50th percentile regression. To generate a prediction interval, an NeuralProphet object needs a list of at least an upper and lower quantile pair as the parameter. However, you may create as many quantiles as you wish in a NeuralProphet model.

Back to forecasting our hospital electric load dataset. Assuming we want the true value to be within the estimator’s prediction interval 90% of the time (i.e., 90% confidence level), we create a three-quantiles regression model that outputs the 5th, 50th and 95th percentile values respectively.

# NeuralProphet only accepts quantiles value in between 0 and 1
# Parameter for quantile regression
confidence_lv = 0.9
quantile_list = [round(((1 - confidence_lv) / 2), 2), round((confidence_lv + (1 - confidence_lv) / 2), 2)]

# Create NeuralProphet object with list of quantile as parameter
qr_model = NeuralProphet(quantiles=quantile_list)
qr_model.set_plotting_backend("plotly-static")

Once the quantile regression module is added on a model, NeuralProphet uses the pinball loss (quantile loss) function to assess goodness-of-fit of the trained model. Similiar to how log-likelihood loss function is used for Gaussian linear regression.

Instead of taking the absolute error in, pinball loss function has different error weightings for each different quantile. We usually take an upper quantile exceeding 50th percentile as the upper bound of prediction interval and another lower quantile below 50th percentile as the lower bound of interval. When the actual output lays outside the prediction interval, the loss function assigns a heavier weight for the absolute error and vice versa. We eventually minimise the loss function by adjusting the parameters of the quantile line iteratively.

Let’s see how weighting differs between different lower quantiles (10th vs 25th percentile). Percentile indicates the probability of having the true value below the estimation value. Comparing to the 25th percentile, the 10th percentile has a smaller expected probability of having the true value below the line. When the actual lays outside the prediction interval (i.e., the actual value is smaller than the predicted one), it is more problematic for a 10th percentile to have such error, as it expects 90% of the true value to be above the line, as opposed to the 75% above the line for the 25th percentile.

# Fit the model with the observations
metrics = qr_model.fit(df, freq="H")

# Create a new dataframe for the results
# Including 100 historical values and 30 value points for the future
future = qr_model.make_future_dataframe(df, periods=30, n_historic_predictions=100)

# Perform prediction with the trained models
forecast = qr_model.predict(df=future)

WARNING - (NP.forecaster.fit) - When Global modeling with local normalization, metrics are displayed in normalized scale.
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.989% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.config.init_data_params) - Setting normalization to global as only one dataframe provided for training.
INFO - (NP.utils.set_auto_seasonalities) - Disabling yearly seasonality. Run NeuralProphet with yearly_seasonality=True to override this.
INFO - (NP.config.set_auto_batch_epoch) - Auto-set batch_size to 32
INFO - (NP.config.set_auto_batch_epoch) - Auto-set epochs to 106

INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.989% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.231% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.231% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H

INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column

Prediction made for hospital electric load is visualised below. yhat1 has shown the prediction made at the 50th percentile while the confidence interval has been shaped in light blue. The lower bound is the prediction result of the 10th quantile whereas the upper one is of the 90th.

qr_model.plot(forecast)

2. Conformal prediction

While there are different ways to carry out conformal prediction, NeuralProphet adopts the split conformal prediction, which requires a holdout or calibration set. To carry out split conformal prediction, the dataset has to be split into three distinct sets for training, calibration and testing respectively. An initial prediction interval is created with the base model trained by the training dataset. Uncertainty is quantified by comparing the target variables in the calibration set against the predicted value. The final conformal prediction interval is then formed by adding the quantified uncertainty to both tails of the predicted value.

You can select Naive (or absolute residual) or Conformalized Quantile Regression (or CQR) for the conformal prediction in NeuralProphet. We will be discussing these two options in detail in the following sub section.

Calibration and validation set

At least three subsets (i.e. testing, calibration and testing) are needed in the conformal prediction feature in NeuralProphet. You may choose to opt in a validation subset in this model. If you want to add in a validation subset to train the base model, please make sure the period of the validation subset must be in between of the training and calibration subsets. We will not cover validation procedure in detail here, you can however refer to the Train, Validate and Test procedure tutorial to know how to build up a NeuralProphet model using validation set.

Here, we further divide the calibration set from the training set with a train-calibration ratio of \(1/11\).

# Add calibration set using the data splitting function
train_df, cal_df = m.split_df(train_df, freq="H", valid_p=1.0 / 11)

INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.988% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H

INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column
INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column

After this splitting, we will have 7467, 746 and 547 instances extracted in sequence as training, calibration and testing set respectively.

train_df.shape, cal_df.shape, test_df.shape

((7467, 2), (746, 2), (547, 2))

Base model training

You can build any NeuralProphet model you deemed as fit as the base model. The calibration process in conformal prediction would be later added on the base model to quantify the uncertainty in our final estimation.

We are interested to know how conformal prediction affects different models. Back to our example, we will be comparing the conformal prediction results between a simple quantile regression and a complex 4-layer autoregression model in our illustration here. You can refer to the quantile regression session above and the Autoregression standalone tutorial material for the logic and applications of the features.

# Parameter for autoregression
# Predict the value in the next hours based on the last three daysin an-hour steps
n_lags = 3 * 24

# Create a simple quantile regression model
cp_model1 = NeuralProphet(quantiles=quantile_list)
cp_model1.set_plotting_backend("plotly-static")

# Create a 4-layer autoregression model as the base
cp_model2 = NeuralProphet(
    yearly_seasonality=False,
    weekly_seasonality=False,
    daily_seasonality=False,
    n_lags=n_lags,
    ar_layers=[32, 32, 32, 32],
    learning_rate=0.003,
    quantiles=quantile_list,
)
cp_model2.set_plotting_backend("plotly-static")

After finish configuring the model, we fit the model with the train set. If you have further split the training dataset into training and validation, you can either (i) concatenate the two datasets in one dataset for training or (ii) assign the training and validation datasets as two separated parameters.

# Feed the training subset in the configured NeuralProphet models
# Configure the hourly frequency by assigning 'H' to parameter freq
set_random_seed(0)
metrics1 = cp_model1.fit(train_df, freq="H")
set_random_seed(0)
metrics2 = cp_model2.fit(train_df, freq="H")

WARNING - (NP.forecaster.fit) - When Global modeling with local normalization, metrics are displayed in normalized scale.
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.987% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.config.init_data_params) - Setting normalization to global as only one dataframe provided for training.
INFO - (NP.utils.set_auto_seasonalities) - Disabling yearly seasonality. Run NeuralProphet with yearly_seasonality=True to override this.
INFO - (NP.config.set_auto_batch_epoch) - Auto-set batch_size to 32
INFO - (NP.config.set_auto_batch_epoch) - Auto-set epochs to 111
WARNING - (NP.config.set_lr_finder_args) - Learning rate finder: The number of batches (234) is too small than the required number                     for the learning rate finder (246). The results might not be optimal.

WARNING - (NP.forecaster.fit) - When Global modeling with local normalization, metrics are displayed in normalized scale.
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.987% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.config.init_data_params) - Setting normalization to global as only one dataframe provided for training.
INFO - (NP.config.set_auto_batch_epoch) - Auto-set batch_size to 32
INFO - (NP.config.set_auto_batch_epoch) - Auto-set epochs to 111

We use the fitted base model to forecast both the point prediction and the quantile regression prediction intervals for the testing dataset.

# Perform estimation for the testing data with the trained model
forecast1 = cp_model1.predict(test_df)[n_lags:]
forecast2 = cp_model2.predict(test_df)[n_lags:]

INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.817% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.817% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H

INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.817% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.818% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H

INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column

Option 1: Naive Conformal Prediction

After training the base model, we then carry out the calibration process using the naive module. The steps are outlined as follow: i. predict the output value of the instances within the calibration set; ii. calculate absolute residual by comparing the actual and predicted value for each observation in the calibration set; iii. sort all absolute residual in ascending order; iv. finds the quantified uncertainty (\(\hat{q}\)) with the desired confidence level; and v. use the quantified uncertainty (\(\hat{q}\)) to make the final prediciton intervals.

Going back to our example, we need to denote the parameter value for calibration set, significant level (alpha) for conformal prediction on top of the pre-trained models above.

# Parameter for naive conformal prediction
method = "naive"
alpha = 1 - confidence_lv

# Enable conformal predict on the pre-trained models
# Evaluate parameter is optional, refer to the "Evaluate Performance" session below
naive_forecast1 = cp_model1.conformal_predict(
    test_df,
    calibration_df=cal_df,
    alpha=alpha,
    method=method,
    plotting_backend="plotly-static",
    show_all_PI=True,
)
naive_forecast2 = cp_model2.conformal_predict(
    test_df,
    calibration_df=cal_df,
    alpha=alpha,
    method=method,
    plotting_backend="plotly-static",
    show_all_PI=True,
)

INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.866% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.866% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H

INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.817% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.817% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H

INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column

Data type cannot be displayed: application/vnd.plotly.v1+json

INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.866% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.866% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H

INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.817% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.818% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H

INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column

Data type cannot be displayed: application/vnd.plotly.v1+json

The plots above demonstrate how quantified uncertainty (\(\hat{q}\)) change with different confidence level (1-alpha).

Below are the \(\hat{q}\) values for each model. As they are all positive, you will see the naive interval extending beyond that of simple QR. The weaker model has a large gap between these two intervals since it has a larger \(\hat{q}\) value, while the bounds are shifted much less for the complex model.

naive_forecast1

	ds	y	yhat1	yhat1 - qhat1	yhat1 + qhat1	trend	season_weekly	season_daily	yhat1 5.0%	yhat1 95.0%
0	2015-12-09 06:00:00	803.410865	988.585815	827.912791	1149.258840	1031.427246	-35.461987	-7.379447	909.404419	1087.498291
1	2015-12-09 07:00:00	868.264194	1089.076294	928.403269	1249.749319	1031.444336	-43.485428	101.117432	994.053711	1189.946045
2	2015-12-09 08:00:00	975.752982	1146.444092	985.771067	1307.117117	1031.461304	-51.675148	166.657928	1057.043823	1236.407227
3	2015-12-09 09:00:00	983.268943	1155.003296	994.330271	1315.676321	1031.478271	-59.986122	183.511185	1078.018799	1243.008301
4	2015-12-09 10:00:00	1095.825986	1145.466553	984.793528	1306.139578	1031.495239	-68.370621	182.341919	1073.590576	1236.537354
...	...	...	...	...	...	...	...	...	...	...
542	2015-12-31 20:00:00	845.563081	833.224609	672.551584	993.897634	1040.634766	-103.066475	-104.343666	757.201721	912.330200
543	2015-12-31 21:00:00	827.530521	795.515076	634.842051	956.188101	1040.651733	-94.571983	-150.564713	711.605408	872.624634
544	2015-12-31 22:00:00	829.256300	771.299255	610.626230	931.972280	1040.668701	-85.883865	-183.485596	677.626831	873.617920
545	2015-12-31 23:00:00	813.937205	762.211975	601.538950	922.885000	1040.685791	-77.059326	-201.414413	669.592285	869.896973
546	2016-01-01 00:00:00	815.588584	765.104858	604.431834	925.777883	1040.702759	-68.155449	-207.442413	675.509155	861.297852

547 rows × 10 columns

naive_qhat1 = naive_forecast1.iloc[-1]["yhat1"] - naive_forecast1.iloc[-1]["yhat1 - qhat1"]  # cp_model1
naive_qhat2 = naive_forecast2.iloc[-1]["yhat1"] - naive_forecast2.iloc[-1]["yhat1 - qhat1"]  # cp_model2
naive_qhat1, naive_qhat2

(160.67302487812503, 20.35876320312491)

We can then plot the predicted intervals (5th, 50th and 95th percentile values) to compare the performance of the models. The quantile regression prediction intervals are annoted in blue while the conformal prediction intervals with the quantified uncertainty included are denoted in red.

With the same quantile parameter, the simple model has a much wider quantile regression prediction interval (in blue) comparing to the complex model. The same happens to the conformal prediction intervals (in red), where the weaker model has a wider width, as such it can capture more actual values than the quantile regression prediction intervals.

# Date range shown in the plots (optional)
cutoff = 7 * 24

fig1 = cp_model1.highlight_nth_step_ahead_of_each_forecast(1).plot(
    naive_forecast1[-cutoff:], plotting_backend="plotly-static"
)
fig2 = cp_model2.highlight_nth_step_ahead_of_each_forecast(1).plot(
    naive_forecast2[-cutoff:], plotting_backend="plotly-static"
)

WARNING - (NP.forecaster.plot) - highlight_forecast_step_n is ignored since auto-regression not enabled.

Option 2: Conformalized Quantile Regression

In Conformalized Quantile Regression, or the cqr module, the method runs as follows: i. calculate non-conformity scores as the differences between data points from the calibration dataset and their nearest prediction quantile, which provides a measure of how well the data fits the current quantile regression model. The non-conformity scores are negative for data points within the quantile regression interval and positive if they are outside the interval; ii. sort list of non-conformity scores; iii. find the value of \(\hat{q}\) such that the portion of scores in the list greater than \(\hat{q}\) is equal to that error rate; and iv. adjust the quantiles from the regression model by an amount (\(\hat{q}\)).

There are two scenarios for what the CQR model means based on the value of \(\hat{q}\). If the one-sided prediction interval width adjustment is positive, then CQR extends beyond the QR intervals, as it deems the QR interval to be too confident. Conversely, if the one-sided prediction interval width adjustment is negative, then CQR contracts the QR intervals, as it deems the QR interval to be too conservative.

# Parameter for Conformalized Quantile Regression
method = "cqr"

# Enable conformal predict on the pre-trained models
# Evaluate parameter is optional, refer to the "Evaluate Performance" session below
cqr_forecast1 = cp_model1.conformal_predict(
    test_df, calibration_df=cal_df, alpha=alpha, method=method, plotting_backend="plotly-static"
)
cqr_forecast2 = cp_model2.conformal_predict(
    test_df, calibration_df=cal_df, alpha=alpha, method=method, plotting_backend="plotly-static"
)

INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.866% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.866% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H

INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.817% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.817% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H

INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column

Data type cannot be displayed: application/vnd.plotly.v1+json

INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.866% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.866% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H

INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.817% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H
INFO - (NP.df_utils._infer_frequency) - Major frequency H corresponds to 99.818% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - H

INFO - (NP.df_utils.return_df_in_original_format) - Returning df with no ID column

Data type cannot be displayed: application/vnd.plotly.v1+json

Below are the \(\hat{q}\) values for each model. As they are all positive, you will see the CQR interval extending beyond that of simple QR. cp_model1 has a large gap between these two intervals since it has a larger \(\hat{q}\) value, while the bounds are shifted much less for cp_model2.

We again plot the prediction intervals to examine how this CQR method affects the result.

fig1 = cp_model1.highlight_nth_step_ahead_of_each_forecast(1).plot(
    cqr_forecast1[-cutoff:], plotting_backend="plotly-static"
)
fig2 = cp_model2.highlight_nth_step_ahead_of_each_forecast(1).plot(
    cqr_forecast2[-cutoff:], plotting_backend="plotly-static"
)

WARNING - (NP.forecaster.plot) - highlight_forecast_step_n is ignored since auto-regression not enabled.

Evaluate Performance

We are using interval width and miscoverage rate as the performance metrics. - interval_width: The average prediction interval, or q_hat multiplied by 2 because it is static or non-adaptive, this is also knowns as the efficiency metric. - miscoverage_rate: The actual miscoverage error rate on the OOS test set, this is also knowns as the validity metric.

The smaller the metrics are, the better the models are performing.

Let’s test with the models we trained above. We first build a dataframe with two rows for the two models we trained and then we will concatenate the Naive and CQR performance metrics in the dataframe for the evaluation.

# Create evaluation dataframe skeleton with the 2 models (denoted as m1 and m2)
models = ["m1", "m2"]
eval_df = pd.DataFrame(models, columns=pd.MultiIndex.from_tuples([("model", "", "")]))

We are aggregating the performance metrics for naive and put them in the evaluation dataframe skeleton at this session. The performance metrics are calculated when you have turned on the evaluate parameter in the conformal prediction feature.

# Extract the naive performance metrics from their respective forecast datasets
naive_eval1 = uncertainty_evaluate(naive_forecast1)
naive_eval2 = uncertainty_evaluate(naive_forecast2)

# Aggregate the naive performance metrics for m1 and m2
naive_evals = [naive_eval1, naive_eval2]
naive_eval_df = pd.concat(naive_evals).reset_index(drop=True)

# Extract the cqr performance metrics from their respective forecast datasets
cqr_eval1 = uncertainty_evaluate(cqr_forecast1)
cqr_eval2 = uncertainty_evaluate(cqr_forecast2)

# Aggregate the cqr performance metrics for m1 and m2
cqr_evals = [cqr_eval1, cqr_eval2]
cqr_eval_df = pd.concat(cqr_evals).reset_index(drop=True)

Lastly, we concatenate the naive and cqr evaluation dataframes and then compare how the models are performing with the naive and cqr prediction.

# Concatenate the naive and cqr evaluation dataframes
eval_df = pd.concat([naive_eval_df, cqr_eval_df], axis=1, keys=["naive", "cqr"])
eval_df

	naive		cqr
	yhat1		yhat1
	interval_width	miscoverage_rate	interval_width	miscoverage_rate
0	321.346050	0.107861	309.400519	0.107861
1	40.717526	0.065263	41.712111	0.063158

Performance of the trained model above:

• This notebook is only using single forecast timestep models, hence we would only have yhat1 as the point estimator.

• Across all three uncertainty prediction methods, the complex autoregression model (m2) has a smaller interval_width and miscoverage_rate than the simpler model (m1). As such, we can conclude that the more complex the model is, the better the data are fitted and the more accurate the predictions are made.

• Both Naive and CQR conformal prediction methods significantly outperforms vanilla QR in terms of miscoverage_rate. This shows that the vanilla QR is overconfident in its quantile range. The interval_width should be broadened further to converge its actual miscoverage_rate (on out-of-sample test set) to the specified alpha at 0.1.

• The interval_width doubles the quantified uncertainties (qhat1) for the Naive conformal predictions, as such you will be seeing symmetrical prediction intervals.

• When we are only looking at the simple quantile regression model (m1), CQR is more preferable. Though it has the same miscoverage_rate as Naive, it still has a narrower prediction interval_width.

• As for the complex model (m2), Naive has a slightly better interval_width, but CQR has a slightly better miscoverage_rate. You may want to feed the model with more data to determine which method more preferable.