Live demo
A weekly demand forecast, and the first honest question about it.
12 SKUs, two years of weekly history. The first year calibrates, the second is held out. The forecaster ships a 90% interval derived from its own Gaussian error model. Counting how often demand actually landed inside that interval takes one line of code, and it is the line almost nobody runs.
The model is not broken. Its point forecast is fine. What is wrong is the assumption behind the band: it treats error as Gaussian with variance proportional to volume, and real demand error is skewed and grows faster than that. Split conformal replaces that assumption with the residuals you already have. Scaling residuals by the sd the model itself reports, before taking the quantile, gets 91.3%, and that version widens where volume is high rather than everywhere equally.
Measured, not assumed
It is wrong at every level, and wrong by more the further out you go.
The 99% interval is the one planners lean on for the cases that hurt, and it is the one furthest from its promise. That is the signature of a parametric tail assumption meeting a distribution with real skew in it.
| Nominal level | Model interval covers | Conformal covers | Conformal, scaled | Shortfall | Mean half-width |
|---|---|---|---|---|---|
| 80.0% | 47.3% | 79.8% | 79.5% | 32.7 pts missing | 22 u → 46 u |
| 90.0% | 58.5% | 90.7% | 91.3% | 31.5 pts missing | 28 u → 70 u |
| 95.0% | 65.5% | 96.2% | 95.8% | 29.5 pts missing | 33 u → 88 u |
| 99.0% | 76.9% | 99.2% | 99.5% | 22.1 pts missing | 44 u → 184 u |
Conformal gives marginal coverage: over all SKU-weeks pooled, at least 1 minus alpha. It does not promise the same coverage inside every SKU. A constant width over-covers the small SKUs and under-covers the largest one, which is why the scaled variant exists. Neither is conditional coverage, and no method gives you that for free. Being explicit about which guarantee you bought is most of the value here.
One SKU, held out
The weeks the band was supposed to contain.
Every dot is a week of real demand against the interval that was published before it happened. Red dots are the weeks the planner was told would not occur.
The conformal band is not a different forecast. It is the same line with an honest width around it, and the width came from counting residuals rather than from assuming a shape. Switch to a low-volume SKU and the two bands converge, which is exactly where the model assumption was closest to true.
What it costs
Stock to the 95% upper bound and count the weeks you ran out.
A planner sizing safety stock takes the top of the interval and holds to it. A two-sided 95% band should be breached on the high side well under 5% of weeks. Here is what each band actually delivered across 624 held-out SKU-weeks.
Conformal did not make the forecast better and it did not save anyone money on inventory. It carries 2.7 times the cover, because that is the width a 95% service level on this demand actually requires. The model interval was cheaper only in the sense that a smaller number written on a plan is cheaper than the stock it fails to buy: 16.7% of weeks short instead of 3.8%. The value here is not a narrower band. It is that the number on the plan and the outcome in the warehouse finally agree, so the service level becomes something you choose rather than something you discover.