The answer
Ten seconds, no signup
What you came for, without an account, an email gate or a product tour.
underpoweredA 40-case evaln=40 · power 10.8%
Smallest regression visible: 11.6pp. To see 3pp at 80% power you need 599 cases.
underpoweredA 200-case evaln=200 · power 36.7%
Smallest regression visible: 5.2pp. To see 3pp at 80% power you need 599 cases.
usableA 900-case evaln=900 · power 93.0%
Smallest regression visible: 2.4pp. Enough to detect the 3pp effect you care about.
underpoweredA conversion A/B testn=5,000 · power 41.6%
Smallest regression visible: 1.6pp. To see 1pp at 80% power you need 12,857 cases.