Live demo
Two eval runs. One of them is lying to you.
Same 40 cases, two model versions. The scoreboard shows a five point drop. Every number below is computed in your browser from the actual per-case outcomes — a paired permutation test and a bootstrap interval, not a t-test on two means.
The null here is sign-flipping each case’s difference, which is the correct null for paired data: if the two versions were equivalent, whether any individual case got better or worse is a coin flip.
A team reading only the scoreboard files a bug, bisects the prompt, and burns a day. The test says the two versions are not distinguishable on this evidence. The eval is not wrong — it is underpowered, which is a different and fixable problem.
The same regression, resolved
There really was a three point regression
Identical injected effect, identical generator, 600 cases instead of 40. The effect was always there. The first eval simply could not see it — and overstated it while failing to confirm it, which is what small samples do.
The null here is sign-flipping each case’s difference, which is the correct null for paired data: if the two versions were equivalent, whether any individual case got better or worse is a coin flip.
The calculator
What can your eval set actually see?
Run this before you trust a number, not after you have already shipped a fix for a difference that was never there.
The part nobody measures
How much of your eval movement is the judge?
Score the same 180 model outputs three times with the same judge. Nothing about the outputs changed between passes, so every bit of spread within a row is the judge disagreeing with itself. That is measurement error, and it sets a hard floor on what your eval can resolve no matter how many cases you add.
A 180-case eval scored by this judge carries about as much information as 102 cases scored perfectly. The fix is not always more cases — averaging several judge passes per case, or tightening the rubric until it stops disagreeing with itself, buys back the same resolution far more cheaply. You cannot make that trade without measuring the split first.