The calculators
4 configurations in. 3 out.
Four eval and test configurations, scored against a three-point effect. Two of them cannot see it.
| Configuration | n | Smallest visible | Power | n needed | |
|---|---|---|---|---|---|
| A 40-case eval | 40 | 11.6pp | 10.8% | 599 | underpowered |
| A 200-case eval | 200 | 5.2pp | 36.7% | 599 | underpowered |
| A 900-case eval | 900 | 2.4pp | 93.0% | 599 | usable |
| A conversion A/B test | 5,000 | 1.6pp | 41.6% | 12,857 | underpowered |
A forty-case eval cannot see a three-point regression, and it will still print a number every time you run it. That number is not evidence of anything, which is the entire reason this page is free.