{"id":"temporal-forecast-validation","name":"Forecasts are tested forward in time","shortName":"Forecasts are tested forward in time","role":"Horizon, rolling origin, seasonal baseline, probabilistic intervals and leakage.","description":"A forecast is a prediction made at a time using only information available then. Its horizon determines how far ahead the target lies. Training and evaluation must preserve that availability boundary: randomly mixing future and past observations can leak information into a test that appears predictive. Rolling-origin evaluation advances the cutoff and tests later horizons, while simple seasonal or last-value baselines provide a meaningful comparison. Probabilistic forecasts add intervals or distributions whose coverage and calibration require evaluation. Chronos supplies primary context for learning time-series representations through scaling and quantization. The worked numbers here compare explicit forecasts; they do not run Chronos or establish a forecasting advantage on a real dataset.\n\nThe application-contract experience fits an actual five-coefficient ridge model to hours 1–23. Each rolling origin uses its latest observation, then recursively feeds predictions through the selected horizon. Later observations begin a new forecast but do not update model coefficients. A persistence baseline uses the exact same targets. A future-label intervention produces zero error while explicitly violating information availability.","mechanism":["A forecast origin defines the available observations and the later target horizon.","Training, normalization and model selection use only the permitted earlier information.","Predictions are compared with a declared simple baseline at the same origins and horizons.","Error and interval coverage are reported with sample size and dependence limits, rather than implied certainty."],"learningObjective":"Preserve time availability and compare forecasts and interval coverage on the same declared horizons.","engineeringExample":"At one origin, future values are [12,15,13]. A candidate predicts [10,14,16], so absolute errors [2,1,3] give mean absolute error 2. A last-value baseline of 11 predicts [11,11,11], giving errors [1,4,2] and mean 7/3. The candidate is better on this tiny three-target example, not necessarily on other origins. If two of the three realized values lie inside a stated 90% interval, observed coverage is 2/3; three possibly dependent targets cannot establish long-run 90% calibration.\n\nWith a +4-unit shift beginning at hour 32, the default one-hour forecast scores 24 target rows. Its mean absolute error is about 3.26680 synthetic load units, while persistence achieves 1.10411 on the same rows. More fitting machinery does not guarantee the better forecast. At an eight-hour horizon the number of scored origins is 17, and the separately matched baseline must be reported again.","misconception":"A low error after a random time split can reflect leakage rather than predictive information.","check":{"question":"Why must a normalization fit exclude future evaluation values?","options":["Because otherwise the forecast uses information unavailable at its origin","Because future data has no units","Because all time series must be constant"],"answer":0,"explanation":"A forecast’s evidence boundary is what was available when it was issued."},"checks":[],"prerequisites":["training-data","evaluation"],"related":["tool-loop","evaluation","coding-agent-patch-evidence"],"specs":[{"label":"Mechanism boundary","value":"Could the forecast have been made using information available then?","context":"All forecasts precede their targets; values are synthetic. A small set of correlated horizons is not a broad accuracy or calibration estimate."},{"label":"Evidence type","value":"Authored worked example","context":"Worked explanation with a bounded numeric or state example. This lesson does not claim an executing implementation of an external model or platform."}],"interfaces":[{"name":"Declared input and state","direction":"in","signal":"A forecast origin defines the available observations and the later target horizon."},{"name":"Checked outcome","direction":"out","signal":"Preserve time availability and compare forecasts and interval coverage on the same declared horizons."}],"tradeoff":"Longer horizons and changing regimes can degrade accuracy; narrower intervals look precise but must retain evaluated coverage.","whyAI":"Forecasting applications need temporal evidence discipline in addition to a model capable of processing sequences.","sourceIds":["automation-chronos","automation-sklearn-leakage"],"sources":[{"title":"Chronos: Learning the Language of Time Series","url":"https://arxiv.org/abs/2403.07815v3"},{"title":"scikit-learn · Common pitfalls and data leakage","url":"https://scikit-learn.org/stable/common_pitfalls.html#data-leakage"}],"science":[{"title":"Forecasts are tested forward in time · explicit relationship","equation":"Mean absolute error averages the absolute forecast error over the declared positive number of horizons.","explanation":"At one origin, future values are [12,15,13]. A candidate predicts [10,14,16], so absolute errors [2,1,3] give mean absolute error 2. A last-value baseline of 11 predicts [11,11,11], giving errors [1,4,2] and mean 7/3. The candidate is better on this tiny three-target example, not necessarily on other origins. If two of the three realized values lie inside a stated 90% interval, observed coverage is 2/3; three possibly dependent targets cannot establish long-run 90% calibration.","assumptions":"All forecasts precede their targets; values are synthetic. A small set of correlated horizons is not a broad accuracy or calibration estimate."}],"evidenceNotes":"This is a worked explanation, not an executing external platform, model or robot. The quantities and fixtures are authored. Source scope is recorded in the automation learning review manifest; no source benchmark or private system behavior is claimed. All forecasts precede their targets; values are synthetic. A small set of correlated horizons is not a broad accuracy or calibration estimate.\n\nExecutable ridge and recursive forecasting use original deterministic hourly data. They are not a real demand forecast, independent-sample confidence estimate or neural forecasting benchmark.","level":"model","branch":"Models become evaluated applications","subbranch":"Forecasts are tested forward in time","scale":"Logical state, task evidence and declared units; no physical assembly scale","kind":"system","workstream":"W11","visualFamily":"software-flow","labId":"library","labScope":"The worked example explains this mechanism. General numerical labs are a reference; they do not execute this lesson’s full mechanism.","history":[],"workedTrace":{"actual":[12,15,13],"forecast":[10,14,16],"baseline":[11,11,11],"mae":2,"baselineMae":2.3333333333333335,"covered":2,"intervalCases":3}}