Live demo
200 A/A canaries in. 6 out.
200 canaries where the upgrade changes nothing, each watched for 14 days. The morning t-test halts 44 of them. The sequential test halts 6, which is what alpha 0.05 promised. Then the same test on two upgrades that do change something.
| Rollout | Truth | Decision | Canary requests | Estimate | Always-valid p | Daily test crossed at |
|---|---|---|---|---|---|---|
| prompt-v42 | prompt-v42, truly 3 points worse | roll back | 1,500 | -4.7pt | 0.012 | 500 |
| prompt-v43 | prompt-v43, truly 2 points better | promote | 4,000 | +2.4pt | 0.030 | 1,750 |
| Canary requests | Requests seen | Estimate | Daily p | Always-valid p |
|---|---|---|---|---|
| 250 | 5,000 | +2.8pt | 0.369 | 1.000 |
| 500 | 10,000 | -5.0pt | 0.033 | 0.395 |
| 750 | 15,000 | -2.9pt | 0.125 | 0.805 |
| 1,000 | 20,000 | -4.5pt | 0.006 | 0.118 |
| 1,250 | 25,000 | -4.2pt | 0.005 | 0.090 |
| 1,500 | 30,000 | -4.7pt | 0.000 | 0.012 |
Same traffic, same 14 looks. The daily test crossed on prompt-v42 earlier than the sequential one did, and it also crosses on 22.0% of nothing. A test that fires early on noise is not a fast test.
The guarantee holds when requests are split at random. A sticky-by-customer split, or a canary that only sees one region, breaks it. Each batch here pairs one control request with each canary request and leaves the rest of the control arm unused, which is conservative: the 5.0% slice is the bottleneck, not the control.