The lesson
What lands in the inbox
One idea, one demonstration, one thing to change on Monday. No course, no upsell.
Lesson 02
Checking every morning is not free
Experiments
200
True effect
zero
Naive wins
72
Sequential
2
We ran 200 experiments where the treatment was byte-identical to the control. Every win is false by construction. Checking each morning and stopping at significance produced 72 winners — 36.0%, against a nominal 5.0%.
The same data under an always-valid bound produced 2. You do not have to stop looking. You have to look with the right threshold.