The verdict
What replaces the green checkmark
Not a number that turns green. An interval that is honest at every moment you look at it, and a stopping recommendation you are actually licensed to follow.
E
checkout-button-copyKeep runningKeep running. This is not a win.
- Observed lift
- +7.50 pts
- Naive p-value
- 0.0466
- Always-valid p-value
- 0.470
- Always-valid interval
- −4.99 pts to +19.99 pts
- Observations so far
- 120
Your dashboard turned green here. Run this to full sample and the effect settles at +2.50 pts. There was never anything in this experiment — both arms were generated from the same rate.
E
onboarding-checklistShip itStop and ship. Valid early stop.
- Observed lift
- +3.78 pts
- Always-valid p-value
- 0.0418
- Always-valid interval
- +0.07 pts to +7.49 pts
- Observations used
- 1560
- Fixed-horizon plan
- 4000 — stopped 61% early
The interval excludes zero under a statistic that is valid for exactly this stopping rule. Stopping here is not a shortcut, it is the method working as designed.
Why not just tell people to stop peeking
Because it has been the standard advice for twenty years and the behaviour has not changed. People check things they care about. The workable fix is to make checking safe rather than to make it forbidden.