Statistical power for LLM evals.
Your eval set cannot see the regression you are looking for.
Forty cases, 82% to 79%, and the team spends a day bisecting a prompt. That difference is noise, and the eval could never have told you otherwise. EvalPower computes what your eval can actually resolve, before you trust it.
Two real eval runs with a known injected regression. Everything computed live from the per-case outcomes.
The problem
Everyone reports the number. Almost nobody reports the uncertainty on it.
Eval suites emit a pass rate. Two versions, two pass rates, and the difference gets treated as a fact. But a pass rate is an estimate from a sample, and its standard error depends on how many cases you ran and how much the judge disagrees with itself. Without that, a team cannot tell a real regression from sampling noise, so they chase phantoms and ship real regressions in the same week.
The insight
Report significance and you are answering the wrong question first.
Significance testing answers "given what I observed, could this be chance?" That is the second question. The first is "if a regression I care about existed, would this eval set have found it?" That is statistical power. A team with an underpowered eval that reports significance correctly still ships regressions, because their eval quietly fails to reject almost every real effect. And the specific structure of evals makes this worse than the textbook case: the same cases run through both versions, so the correct test is paired, and the judge itself contributes variance that no number of extra cases can remove.
Paired bootstrap confidence intervals and sign-flip permutation tests instead of an unpaired t-test on two means. Power and minimum-detectable-effect computed for the paired design. Variance decomposed into model-driven and judge-driven components from repeated judgements on identical outputs, which yields an effective sample size. Benjamini-Hochberg control across slices, because scoring twelve categories separately is twelve tests.
How it works
Four steps, no data science team
Per-case outcomes from your existing harness, or an export from whatever platform you run. It needs the case-level scores, not the summary. The pairing is where the power comes from.
Minimum detectable effect at your current size, power for the regression size you actually care about, and the number of cases required to close the gap.
Repeated judgements on identical outputs split observed variance into real quality differences and judge self-disagreement, and convert that into an effective sample size.
A CI check that blocks on a statistically resolved regression and stays quiet on noise, with the interval and the power printed in the PR so the decision is auditable.
Who it is for
The engineer who has to decide whether to ship
Teams with an eval suite they do not fully trust. Usually the ones who have already argued about whether a three-point move was real, and had no way to settle it.
Pricing
- –Power and MDE calculator
- –Paired bootstrap CLI
- –Public, no account
- –CI gate with PR annotations
- –Judge-variance tracking over time
- –Per-slice FDR control
- –Eval-set health monitoring
- –Self-hosted
- –Custom estimators
- –SSO and audit log
- –Design review with a statistician
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Braintrust | Eval automation that analyses statistical significance and can block merges. | Answers whether an observed difference is significant. Does not tell you beforehand that your eval set was never capable of resolving the effect, and does not separate judge noise from model behaviour. |
| Langfuse / LangSmith | Eval running, tracing, and score dashboards. | Reports the score. Uncertainty on the score is left to the reader. |
| A statistician on the team | Exactly right, and the actual substitute at large labs. | Does not scale below the companies that can hire one, which is most of the market. |
| Spreadsheets and intuition | What most teams do today: eyeball the delta, argue, pick one. | Systematically overconfident on small samples in both directions. |
The real risk is that this is a feature, not a company. Braintrust already owns the eval run and could add power analysis in a sprint. The counter is that the wedge is a free, public, genuinely useful calculator that spreads on its own, and that the durable product is not the significance test but the judge-variance tracking, which requires repeated-judgement data nobody currently collects, and which gets more valuable the longer it runs.
Market
Attached to the eval budget, which every team building on models now has
Teams already pay $200 to $2,000 a month for eval platforms. This is a smaller line item beside that, sold to the same buyer, with a free tier that reaches every engineer who has ever squinted at a three-point delta. Five thousand teams at the Team tier is $18M ARR.