Always-valid canaries for model upgrades
Your canary is a t-test you run every morning
A prompt or model upgrade goes to 5% of traffic and someone checks the dashboard once a day. Checking daily at alpha 0.05 is not alpha 0.05. Canary checks after every batch and keeps the false-halt rate where you think it is.
The problem
Peeking daily turns a 5% error rate into something far worse
The rollout runbook says: watch task success on the canary for two weeks, halt if the difference is significant. The engineer runs the test each morning on the cumulative traffic. Each look is a fresh chance for noise to cross the line, and fourteen looks compound. Good upgrades get rolled back on a bad Tuesday. Bad upgrades get promoted because the day they were checked happened to be lucky. Nobody sees the inflation because each individual test looks fine.
The insight
A test you can check after every batch is the only kind a rollout can use
Rollouts are sequential by nature. Traffic arrives in batches, and the question is asked after each one. A fixed-horizon test is valid once, at a sample size chosen in advance, which is not how anyone runs a canary. The mixture sequential probability ratio test gives a p-value that stays valid no matter how often you look, so "halt as soon as it is significant" becomes a rule with a guarantee instead of a bug. On identical arms it halts at the rate it promises. On a real regression it stops as soon as the evidence is in.
Mixture SPRT over paired control and canary outcomes with a normal mixing prior at the regression size worth halting for, evaluated after every batch, with the daily z-test on the same stream as the baseline.
How it works
Four steps, no data science team
The canary takes its share of requests. Each batch pairs canary outcomes with control outcomes from the same window.
The always-valid p-value moves with each batch. There is no schedule to respect and no penalty for looking.
Roll back on a negative crossing, promote on a positive one, hold otherwise. The false-halt rate is alpha whether you look once or a hundred times.
The decision, the evidence at that moment, and the traffic it took, attached to the rollout.
Who it is for
The engineer who owns model and prompt upgrades
Teams shipping prompt and model upgrades behind a traffic split, where the rollback decision is currently a person, a dashboard and a morning t-test.
Pricing
- –Always-valid p-value
- –Decision and evidence
- –A/A calibration report
- –Flag integration
- –Automatic halt
- –Per-segment canaries
- –Slack decision
- –Multi-metric
- –Self-hosted
- –SSO
- –Audit export
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| LaunchDarkly, Statsig, Eppo rollouts | Feature flags with experiment analysis attached. | Built for product experiments on a fixed horizon. Progressive rollouts get the same fixed-horizon test, checked whenever someone opens the page. |
| Argo Rollouts, Flagger | Promote or abort a rollout on a metric query. | The analysis step is a threshold on a Prometheus query, evaluated on an interval. No error rate is promised, and none is delivered. |
| Vendor eval dashboards | Plot canary quality against control. | A plot with a confidence band is a fixed-horizon test in a costume. It inflates the same way when read daily. |
| A dashboard and a t-test each morning | What everyone does today. | Fourteen looks at alpha 0.05 is not alpha 0.05. The inflation is invisible from inside any single morning. |
The guarantee holds when canary and control traffic are exchangeable, which a random split gives you and a sticky-by-customer split may not. The demo pairs one control request per canary request and leaves the rest of the control arm unused, which is conservative. Statsig and Eppo have sequential tests on their roadmaps and could point them at rollouts in a quarter. It fails if teams decide their upgrades are obviously better and stop measuring, or if the flag vendors ship the same test first.
Market
Priced against the experimentation and release-engineering budget teams already carry
Every team running prompt or model upgrades behind a traffic split ships one every week or two. Two thousand teams at the Team tier is $12M ARR. The Scale tier is priced against one bad promotion.