The CI gate
What lands on the pull request
Two outcomes, and the difference between them is the product. One says the eval is not capable of answering the question. The other blocks the merge with an interval attached.
🟡evalpower / regression-checkNeutralGitHub Checks
Eval is underpowered for this comparison
Observed: -5.0 pts on 40 cases 95% CI: -12.5 pts to 0.0 pts (covers zero) p = 0.490 (paired permutation, 5,000 sign flips) Smallest effect this eval can resolve: 14.3 pts To detect a 3.0 pt regression at 80% power you need ~913 cases
Not blocking. This eval cannot distinguish the two versions, in either direction. Treat the delta as unmeasured rather than as evidence of no change.
🔴evalpower / regression-checkFailingGitHub Checks
Statistically resolved regression — blocking
Observed: -3.0 pts on 600 cases 95% CI: -5.2 pts to -0.8 pts (excludes zero) p = 0.0060 (paired permutation) Power for a 3.0 pt effect at this size: > 80%
Blocking. The regression is resolved at this sample size and the interval excludes no-change. Override with a documented reason if shipping anyway.
Why the first one is the valuable half
Every eval tool can block on a big regression. Almost none will tell you that the green check you just got was meaningless. A passing underpowered eval is worse than no eval, because it manufactures confidence — and that is the failure this gate is built to catch.