A/B tests that survive being watched.
You checked the experiment every morning. That broke it.
A p-value is only valid if you looked once, at a sample size you fixed in advance. Everybody looks daily and stops when it turns green. Experiment Guard replaces the p-value with one that stays valid no matter how often you check.
Two hundred A/A experiments with no effect in any of them, simulated live in your browser.
The problem
The stopping rule everybody uses invalidates the test everybody runs.
Standard significance testing assumes a single look at a sample size fixed before you start. Real practice is the opposite: a dashboard, checked continuously, and a decision to stop the moment the result crosses the line. That is many shots at the same threshold, and you stop precisely on the shot that crosses. On 200 simulated A/A tests the realised false-positive rate is 36%, not 5%. A large share of shipped wins are noise that got booked as lift.
The insight
The fix is not discipline. It is a different p-value.
The standard advice is to stop peeking, pre-register the sample size, and wait. Nobody does this, and telling growth teams to stop watching their experiments has failed as a strategy for twenty years. The better move is to accept the behaviour and change the statistic underneath it. A mixture sequential probability ratio test produces a p-value that is valid under continuous monitoring by construction, which means stopping as soon as it crosses is a legitimate rule rather than a bug. You keep the workflow and get the guarantee back.
A mixture SPRT with a normal mixing prior over plausible effects, producing always-valid p-values and confidence sequences that hold under optional stopping. Reference: Johari, Koomen, Pekelis and Walsh, "Peeking at A/B Tests" (2017). The mixing variance is set near the effect size you would actually act on, which is the one parameter that matters and the one this makes explicit.
How it works
Four steps, no data science team
Assignments and conversions from your existing tooling. No change to how experiments are run or assigned.
Computed continuously alongside the naive one, so you can see the gap on your own live tests rather than taking it on faith.
Stop for a win, stop for futility, or keep running, with a confidence sequence that is honest at every point you look at it.
Sequential monitoring on the metrics you must not break, with rollback fired on statistically resolved harm rather than on a scary-looking morning.
Who it is for
The team whose wins keep not compounding
Growth and product engineering teams running enough concurrent experiments that a false-win rate in the tens of percent is quietly steering the roadmap. Usually the team that has already noticed shipped wins failing to show up in the aggregate numbers.
Pricing
- –mSPRT implementation
- –Peeking simulator
- –No account required
- –Always-valid monitoring on every test
- –Confidence sequences in your dashboard
- –Futility stopping
- –Guardrail metrics
- –Automatic rollback
- –Self-hosted
- –Custom priors per metric
- –Review with a statistician
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Optimizely / Statsig / Eppo | Full experimentation platforms. The serious ones do implement sequential testing. | Sold as a platform replacement, priced accordingly, and adopting one means moving your assignment layer. Most teams already have a home-grown setup they are not going to replace over a statistics concern. |
| Home-grown experiment tooling | What the majority of teams actually run. | Almost universally a fixed-horizon t-test on a dashboard that gets checked daily. The failure is invisible from inside because a false win looks exactly like a real one. |
| GrowthBook / open source | Open-source platforms, some with Bayesian or sequential options. | Sequential support tends to be an option you have to know to turn on, which selects for teams who already understood the problem. |
| A statistician saying stop peeking | Correct, free, and ignored. | Twenty years of evidence that behaviour does not change. Better to fix the statistic than the human. |
The most honest read is that this is a feature of an experimentation platform rather than a standalone company, and that the platforms that would buy it are the ones already most likely to have built it. The wedge that could make it a company is being the layer that sits on top of whatever a team already runs, rather than asking them to migrate. If teams will not adopt a statistics layer without also adopting a platform, this becomes an acquisition rather than a business.
Market
Sold beside existing tooling, not instead of it
Experimentation platforms run $2,000 to $20,000 a month at mid-market and up. This sits alongside a home-grown setup at a fraction of that, aimed at the much larger population of teams who run experiments without buying a platform. Three thousand teams at the Team tier is $18M ARR, and the free simulator is the thing that convinces them they have a problem.