Domain eval suites that can actually see a regression
Your eval suite says 96% and cannot detect anything
A suite where almost everything passes has stopped discriminating between versions. Eval Kits ships domain suites built to sit where changes are visible, with the statistics to prove it.
The problem
Every vertical AI company builds the same bad eval suite
It starts as forty cases somebody wrote in an afternoon, grows to a hundred and forty, and reaches a pass rate near the ceiling because the failing cases got fixed and nobody added harder ones. At that point it cannot detect a regression of any size, and the team is making ship decisions on it every week without knowing that.
The insight
Five checks fail almost every suite in the wild, and four of them are arithmetic
Can it see the effect you care about, how much of its movement is the judge disagreeing with itself, is every slice big enough to report, is it saturated at a ceiling or floor, and is it scoring the same case twice. None of that needs judgement — it needs power analysis, a variance decomposition and a duplicate check. The scorecard is what makes the rebuild worth paying for, because it says exactly what is broken before anyone writes a case.
Paired-design power and minimum detectable effect, variance decomposition into model and judge components from repeated judgements, per-slice sufficiency, saturation detection, and normalised duplicate identification.
How it works
Four steps, no data science team
Five checks, a number out of ten, and the specific reason for each mark.
Contract extraction, claims coding, financial reconciliation. The domain sets the case library.
Enough cases per slice to support the claims you want to make, with repeated judgements built in.
Re-scored on every change, so it cannot silently drift back to the ceiling.
Who it is for
The founder or lead who signs off on shipping a model change
Vertical AI companies whose product is a model output in a specific domain, and whose eval suite was written by whoever had time.
Pricing
- –Five-check scorecard
- –Power analysis
- –Judge-noise decomposition
- –Domain case library
- –Maintained and versioned
- –Continuous scoring
- –Slice sufficiency alerts
- –Custom construction
- –Your data, your slices
- –Quarterly rebuild
- –Direct support
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Braintrust, Langfuse, LangSmith | Platforms for running and tracking evals. | They run whatever suite you give them and report the number. None tells you the suite cannot detect anything. |
| Public benchmarks | Standardised suites for general capability. | Not your domain, not your slices, and contaminated in ways nobody can rule out. |
| Hiring a domain expert | A lawyer or coder writes the cases. | Produces good cases and no statistical structure. Domain knowledge is half the problem; sufficiency is the other half. |
| The suite you already have | Forty cases and a spreadsheet. | Free, already integrated, and saturated at the ceiling. |
This is closer to a content business than a software one: the value sits in the maintained case library, which is expensive to build, expensive to keep current, and requires domain expertise per vertical that a solo generalist does not have. Picking one vertical and going deep is the only viable version, and picking the wrong one costs a year. The scorecard is genuinely differentiated and is also the free tier, which means the paid product rests on the part that is hardest for us to defend.
Market
On the eval budget, which every vertical AI company now has and none of them spends well
Bottom-up by vertical: a few hundred serious contract-AI companies, then the same again in claims and reconciliation. Real, and it compounds only if the library compounds.