Sparse Autoregression

Here we fit NeuralProphet to data with 5-minute resolution (daily temperatures at Yosemite). This is a continuation of the tutorial notebook Tutorial 4: Auto regression. While tutorial 4 covers autoregression in general, this notebook focus on sparsity.

if "google.colab" in str(get_ipython()):
    # uninstall preinstalled packages from Colab to avoid conflicts
    !pip uninstall -y torch notebook notebook_shim tensorflow tensorflow-datasets prophet torchaudio torchdata torchtext torchvision
    !pip install git+https://github.com/ourownstory/neural_prophet.git # may take a while
    #!pip install neuralprophet # much faster, but may not have the latest upgrades/bugfixes

import pandas as pd
from neuralprophet import NeuralProphet, set_log_level

set_log_level("ERROR")

data_location = "https://raw.githubusercontent.com/ourownstory/neuralprophet-data/main/datasets/"
df = pd.read_csv(data_location + "yosemite_temps.csv")

Sparsifying the AR coefficients

The autoregression component of NeuralProphet is defined as a AR-Net (paper, github). Thus, we can set ar_reg to a number greater zero, if we like to induce sparsity in the AR coefficients.

However, fitting a model with multiple components and regularizations can be harder to fit and in some cases you may need to take manual control over the training hyperparameters.

We will start by setting regularization to 0.1

m = NeuralProphet(
    n_lags=6 * 12,
    n_forecasts=3 * 12,
    n_changepoints=0,
    weekly_seasonality=False,
    daily_seasonality=False,
    learning_rate=0.01,
    ar_reg=0.1,
)
m.set_plotting_backend("plotly-static")
metrics = m.fit(df, freq="5min")  # validate_each_epoch=True, plot_live_loss=True

m.plot_parameters()

m = m.highlight_nth_step_ahead_of_each_forecast(1)
m.plot_parameters()

m = m.highlight_nth_step_ahead_of_each_forecast(36)
m.plot_parameters()

Further reducing the non-zero AR-coefficents

By setting the ar_reg higher, we can further reduce the number of non-zero weights. Here we set it to 1

m = NeuralProphet(
    n_lags=6 * 12,
    n_forecasts=3 * 12,
    n_changepoints=0,
    daily_seasonality=False,
    weekly_seasonality=False,
    learning_rate=0.01,
    ar_reg=1,
)
m.set_plotting_backend("plotly-static")
metrics = m.fit(df, freq="5min")

m.plot_parameters()

m = m.highlight_nth_step_ahead_of_each_forecast(1)
m.plot_parameters()

m = m.highlight_nth_step_ahead_of_each_forecast(36)
m.plot_parameters()

Extreme sparsity

The higher we set ar_reg, the fewer non-zero weiths are fitted by the model. Here we set it to 10, which should lead to a single non-zero lag.

Note: Extreme values can lead to training instability.

m = NeuralProphet(
    n_lags=6 * 12,
    n_forecasts=3 * 12,
    n_changepoints=0,
    daily_seasonality=False,
    weekly_seasonality=False,
    learning_rate=0.01,
    ar_reg=10,
)
m.set_plotting_backend("plotly-static")
metrics = m.fit(df, freq="5min")

m.plot_parameters()

m = m.highlight_nth_step_ahead_of_each_forecast(1)
m.plot_parameters()

m = m.highlight_nth_step_ahead_of_each_forecast(36)
m.plot_parameters()