Canary

The canary decision

What the rollout gets told, and what it can trust

A decision with the evidence at the moment it was made, the traffic it cost, and the error rate it carries. No dashboard, no morning ritual.

canary status --rollout prompt-v42fail
rollout        prompt-v42
traffic        5.0% canary
requests       30,000 seen  (1,500 canary, 1,500 paired control, 6 batches)
task success   control 86.5%   canary 81.8%   -4.7pt
evidence       always-valid p = 0.012  (alpha 0.05, checked after every batch)

decision       ROLL BACK

guarantee      a canary that changes nothing is halted at most 5.0% of the time,
               however often this status is read
for contrast   a daily t-test on the same traffic halts 22.0% of neutral upgrades
               (44 of 200 A/A canaries, 14 looks each)