Two weeks to an eval suite you can trust
You have 62 test cases and you know that is not enough
Everyone knows their eval suite is thin. Nobody has two spare weeks to fix it. This is those two weeks, at a fixed price, ending in a suite that can see the regression you care about.
The problem
The eval suite is the first thing built and the last thing maintained
It starts as whatever cases were on hand during the first demo. It grows when something breaks. Nobody ever sits down and asks how many cases each slice needs to support the claims being made — because that question requires power analysis and the person who would run it is shipping features.
The insight
Sizing a suite is arithmetic, and skipping the arithmetic is why every suite is the wrong size
How many cases you need per slice follows directly from the pass rate, the effect you want to detect and how correlated your two arms are. It is a formula. The reason suites are thin is not that the formula is hard, it is that nobody applies it — so cases get added by anecdote until someone feels better, which produces a suite that is simultaneously too small to detect anything and too large to maintain.
Required-n per slice from a paired-design power calculation, judge-noise decomposition from repeated judgements to set the effective sample size, saturation and duplicate detection, then construction against those targets.
How it works
Four steps, no data science team
Five checks on the current suite. Usually uncomfortable and always specific.
Cases per slice, computed from the effect you actually care about.
Cases written, repeated judgements wired in, slices balanced.
Running in your CI, with the scorecard re-run so you can see the difference.
Who it is for
The engineering lead who owns model quality
Series A and B teams whose product depends on model output quality and whose eval suite has not kept up. The shortest path from a warm introduction to an invoice.
Pricing
- –Five-check scorecard
- –Required-n per slice
- –Written findings
- –Case construction
- –Slice balancing
- –Repeated judgements
- –CI integration
- –Handover re-score
- –Quarterly refresh
- –Continuous scoring
- –Saturation alerts
- –Direct access
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Eval platforms | Braintrust, Langfuse and friends run your suite. | They are the pipes. Somebody still has to write the cases and size the suite, and that is the part nobody has time for. |
| Data labelling vendors | Will produce cases at volume. | Volume without statistical structure. A thousand cases in the wrong slices is not better than sixty in the right ones. |
| Doing it internally next quarter | The plan on every roadmap. | Has been next quarter for three quarters. This is a real competitor and it usually wins. |
| Eval Kits | Our own productised domain suites. | Cheaper and not yours. The build-out exists for teams whose slices are specific enough that a library will not fit. |
This is the most commoditisable thing in the document — case generation gets cheaper every quarter, and the labelling vendors are moving toward it. The defensible part is the sizing and the judge-noise decomposition, which is a small fraction of the delivered work and the entire reason it is worth paying for. Treat the window as short, use it to learn which slices recur across customers, and turn that into Eval Kits before the window closes.
Market
On the same eval budget as Eval Kits, at the engagement end rather than the subscription end
Twelve build-outs a year is $192k plus retainers. Its real value is that it is the fastest route from an introduction to a paying customer, and it feeds the product it will eventually be replaced by.