Hyperparameter Optimization for Forecasting

Hyperparameter optimization searches over model, feature, and pipeline choices to minimize a validation objective. In forecasting, the objective must be computed under time-respecting evaluation. Searching directly on final backtest results leads to optimistic estimates.

Core concepts

An objective function maps a candidate configuration to a scalar loss, such as validation WAPE, MAE, or a business-weighted metric. A search space defines allowed values for hyperparameters. A trial is one sampled configuration and its result. A study is the collection of trials and metadata.

A sampler proposes new configurations. A pruner stops weak trials early. A timeout limits runtime. Persistent storage allows studies to resume after interruption and supports auditability.

What can be tuned

Forecasting search spaces can include model hyperparameters, lag sets, rolling-window lengths, feature transformations, target transformations, model family, ensemble strategy, ensemble size, neural architecture, learning rate, batch size, dropout, and maximum training steps.

The search space should be structured. For example, tree depth matters only for tree models, and hidden-layer sizes matter only for neural models. Treating all options as one flat set of independent parameters creates invalid combinations and wastes trials.

Nested evaluation risks

If a model is tuned on the same final backtest period used for reporting, the reported estimate is biased upward. This is a form of adaptive overfitting: the search process has learned idiosyncrasies of the evaluation period.

Practical alternatives include inner validation within each training fold, a separate tuning period, nested backtesting, conservative reuse of global hyperparameters, and final evaluation on untouched forecast origins.

Conceptually:

training history
    |
    +-- inner validation for tuning
    |
    +-- backtest forecasts for final comparison

For ensembles:

backtest forecasts
        |
        +-- ensemble-training portion
        |
        +-- ensemble-evaluation portion

Practical search design

Start with a small, defensible baseline search. Tune the highest-impact parameters first: lag set, seasonal lags, learning rate, tree complexity, number of estimators, and target transformation. Add neural architecture or ensemble choices only after the validation protocol is stable.

For large panels, tune global hyperparameters on representative partitions rather than every series. For expensive neural models, use pruners and fixed validation windows to control compute. For boosted trees, early stopping can reduce overfitting and training time, but the early-stopping validation data must respect time.

Practical guidance

• Define the metric and aggregation level before launching the search.
• Keep a simple baseline outside the optimization process.
• Use time-respecting validation for every trial.
• Store trial configuration, random seeds, data cutoffs, metric results, and code version.
• Avoid tuning on the final evaluation period.
• Prefer structured conditional search spaces over large flat spaces.

Common failure modes

• Selecting the best configuration from the final reported backtest.
• Letting preprocessing fit on all data before trial evaluation.
• Comparing trials that used different forecast origins.
• Optimizing one metric and deploying for another cost function.
• Running many expensive trials without checking whether naive baselines are already competitive.

Connections

Forecast hyperparameter search must sit inside backtesting or rolling-origin validation. It affects machine learning forecasting, forecast ensembling, and the final forecast error metrics.

References

• Hyndman & Athanasopoulos, FPP3: Time series cross-validation
• scikit-learn TimeSeriesSplit

Nav

Section — Time-Series Forecasting

← Forecast Ensembling Forecasting System Design →