Eval gates that check every slice
The aggregate passed. Three customer slices did not.
Your eval gate compares one pass rate against another. A candidate that gains a point on English and loses eight on German enterprise clears it, and you hear about it from the German enterprise customer. Slices tests every slice, corrected, so the gate fails on the right three and stays quiet on the rest.
The problem
One number cannot fail for the right reason
The gate everyone runs compares the aggregate pass rate of baseline and candidate. Two big slices improve, three small ones fall off a cliff, and the average moves up by less than half a point. The obvious fix, checking every slice, runs a dozen tests per merge and fails clean candidates so often that the team turns it off within a month. So slices go unchecked and customers become the eval.
The insight
Per-slice checking is a multiple-testing problem, and the test has to respect the pairing
Both model versions ran the same cases. The per-case difference is the quantity with the least noise in it, and a test on two independent means throws that away. Eval scores are bounded and mostly 0/1, so the normal approximation behind a t-test is a poor fit too. A paired bootstrap on the differences handles both. Then twelve slices are twelve tests, and Benjamini-Hochberg is the difference between a gate that fails on three real regressions and one that fails on three real regressions plus one imaginary one every other merge.
Paired bootstrap confidence interval and sign-flip permutation p-value per slice on per-case score differences, then Benjamini-Hochberg across slices at q < 0.05, reporting only slices whose corrected q clears the bar with a negative point estimate.
How it works
Four steps, no data science team
Slices reads per-case scores for baseline and candidate, keyed by case id and slice. Any harness that writes a row per case works.
Each slice gets a bootstrap interval and a permutation p-value on its own paired differences. Rubric fractions and pass-fail bits go through the same test.
BH across everything tested in the run. A slice that would have failed uncorrected and did not survive is listed as suppressed, with its numbers, not hidden.
The check prints each regressed slice with its delta, interval and adjusted p, and the aggregate line that would have passed. That is what the reviewer reads.
Who it is for
The engineer who owns the eval gate
Teams with an eval suite in CI and customers who do not all look alike. Usually found the week after a slice regression shipped and a customer noticed first.
Pricing
- –Per-slice paired bootstrap
- –BH across slices
- –Suppressed list
- –CLI and GitHub check
- –Run history per slice
- –Per-slice power report
- –Slack on gate failure
- –Custom alpha per suite
- –Self-hosted
- –SSO
- –Customer-defined slices
- –Audit export
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Braintrust, Langfuse, LangSmith experiment comparison | Show baseline and candidate scores side by side, with a breakdown by tag or metadata. | The breakdown is a table of means. No pairing, no interval, no correction across the rows, so the reader has to decide which cells are real, and does so by eye. |
| promptfoo | Runs the eval in CI and fails on a threshold. | The threshold is on the aggregate pass rate, or on a per-test assertion. There is no per-slice comparison with an error rate attached. |
| A pass-rate threshold in CI | Fail if the candidate is more than X points below baseline. | X is a guess, the comparison ignores the pairing, and it is one number. It cannot fail on the German enterprise slice because it never looks at it. |
| Compare the aggregate | What almost everyone does today. | Passes any candidate whose wins outweigh its losses on average. The losses are the customers who leave. |
Any of the eval platforms could add a paired bootstrap and BH to their comparison view in a sprint; the method is public and the code is short. The defensible part is being the gate, in CI, on the pull request, rather than a view someone opens later. For this to fail, teams would have to stop slicing their evals, or slice them so thinly that twenty-case slices cannot fail with confidence and the gate reports "cannot tell" on every merge, which is honest and unsatisfying. The paired bootstrap assumes cases within a slice are exchangeable and independent. A suite built by templating one prompt into fifty variants violates that, and the p-values come out optimistic. The demo reports the rubric share and the per-slice size for that reason.
Market
Priced against the CI budget, next to the test runner and the coverage tool
Every team with an eval suite in CI and more than one customer segment is a candidate, and the first slice regression that reaches a customer is the buying event. Four hundred teams at the Team tier is $1.9M ARR. The Scale tier is priced for teams whose enterprise customers notice before the dashboard does.