Lesson two, running live
200 A/A experiments in. 72 out.
Two hundred experiments where the treatment is identical to the control. Every “win” below is false by construction.
Checked every morning
36.0%
Always-valid bound
1.0%
Nominal false-positive rate is 5.0%. Peeking daily turns that into 36.0% without anyone changing a single line of analysis code.
| # | The mistake | The fix |
|---|---|---|
| 1 | Your eval cannot see what you are looking for | Power and minimum detectable effect |
| 2 | Checking every morning is not free | Always-valid sequential testing |
| 3 | Twelve slices is twelve tests | Benjamini-Hochberg |
| 4 | Your confidence score is not a probability | Conformal prediction |
| 5 | The top of your leaderboard is noise | Empirical Bayes shrinkage |
| 6 | A percentage throws away when it happened | Survival analysis |