Rolling Origin Validation

Rolling-origin validation evaluates a forecaster by moving the training cutoff forward through history. At cutoff $t_i$, the model is trained only on observations up to $t_i$ and evaluated on one or more future horizons:

$${\hat{y}}_{t_i+h|t_i},h=1,\dots ,H.$$

This mirrors the production forecasting contract: each forecast can use the past and any covariates genuinely known at forecast creation time, but not outcomes or derived statistics from the future.

There are two common variants. Expanding-window validation keeps the first training date fixed and adds more history at each cutoff. Sliding-window validation keeps a fixed lookback length and drops older observations, which can help when concept drift in forecasting is expected. Multi-horizon validation records errors separately by horizon because a model that is strong at $h=1$ may be weak at $h=14$.

Fold scheme	Training window at each cutoff	Use when
Expanding window	first date fixed, history grows	history stays relevant; more data helps
Sliding window	fixed-length lookback, older data dropped	drift makes old data misleading
Multi-horizon	either window, errors kept per horizon $h$	short- and long-horizon accuracy differ

Leakage usually enters outside the split object. Lag features, rolling means, scalers, target encoders, hyperparameter search, and imputation must be fitted inside each fold whenever they depend on observed targets. That is why rolling origin is the fold design behind backtesting, not a complete evaluation system by itself.

Rolling validation produces the error table consumed by forecast error metrics, forecast evaluation, and hyperparameter optimization for forecasting. The useful output is not a single score but a matrix by cutoff, horizon, segment, and model, so failures around holidays, cold-start series, or long horizons are visible.

The cutoffs should also match the production retraining policy. A model retrained every night can use denser origins than a model retrained monthly. When labels arrive late, evaluation must delay scoring until outcomes are genuinely available.

References

• Hyndman & Athanasopoulos, FPP3: Time series cross-validation
• scikit-learn TimeSeriesSplit

Nav

Section — Time-Series Forecasting

← Backtesting Forecast Error Metrics →