Eval pass rates corrected for judge bias
Your judge passes a third of the failures. Your pass rate is wrong.
Teams grade agent outputs with an LLM judge and report the judged pass rate as if it were the true one. A lenient judge turns a 62% agent into a 70% agent and nobody corrects for it. Judge does, with an interval, from the small labelled set you already have.
The problem
The number on the eval dashboard is the judge's opinion, not the agent's pass rate
An LLM judge is a classifier with a sensitivity and a false-positive rate, and a reported pass rate is what that classifier says about the outputs, not what a human would say. Everyone knows the judge is imperfect. Almost nobody applies the correction, because the correction is a 1978 epidemiology result that never made it into the eval tooling. The result is a pass rate that is too high by an amount nobody has measured, and a v2 comparison that reads as a wash when v2 is worse.
The insight
A judge is a diagnostic test, and diagnostic tests have had a correction since 1978
When a test with known sensitivity and specificity is run over a population, the apparent positive rate is a linear function of the true one, and the line has been known for fifty years. Rogan and Gladen solved it for the true rate. The only inputs are the judge's true-pass rate and false-pass rate on a small human-labelled sample, which every eval team already has and uses only to eyeball. The correction also tells you when the judge is too weak to correct: below a separation of 0.2 the estimator divides by nothing and Judge says so rather than printing a number.
Rogan-Gladen misclassification correction: corrected = (judged rate - FPR) / (TPR - FPR), with TPR and FPR estimated from a labelled calibration set and a seeded bootstrap over both the calibration set and the judged set for the interval, plus the same correction on a two-variant difference.
How it works
Four steps, no data science team
Three hundred outputs, graded by a human. You probably have these already from writing the rubric.
Judge computes the judge's true-pass and false-pass rates on that slice and reports whether they are far enough apart to support a correction.
Every judged pass rate is corrected and carries an interval that reflects both the calibration sample and the eval sample.
Two prompt variants scored by the same judge get a corrected difference. That is the number the ship decision rests on.
Who it is for
Whoever puts their name on the eval number
Teams that run an LLM judge over agent outputs and put the pass rate in a weekly update, a launch review, or a customer report.
Pricing
- –Judge TPR and FPR
- –Corrected pass rate
- –Separation check
- –Corrected difference per PR
- –Calibration drift alerts
- –Per-slice correction
- –Report export
- –Multiple judges per suite
- –Calibration set sizing
- –SSO and audit log
- –API access
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Braintrust, LangSmith, Langfuse judge scorers | Run an LLM judge over traces and chart the pass rate. | They report the judge's number as the number. None of them estimate the judge's error rates or correct for them. |
| Arize Phoenix evals | Pre-built judge templates with published benchmark accuracy. | Benchmark accuracy is on someone else's data. The correction needs the judge's error rates on your outputs, and then needs to be applied. |
| Human labelling vendors | Label everything with people. | Correct and expensive. Three hundred labels plus the correction recovers most of what three thousand labels would tell you. |
| Report the judged number | What almost everyone does today. | Biased by an amount nobody has measured, in a direction that flatters the agent, and it compresses every variant comparison toward zero. |
The eval platforms could add this in a sprint; the formula is one line and the calibration set already exists in their systems. The defensible part is small: the interval that accounts for both samples, the refusal to correct a weak judge, and the corrected difference for comparisons. The method assumes the calibration slice is drawn from the same distribution as the judged set. If the judge is harsher on one queue and the labelled slice under-samples that queue, the corrected number is off, and the demo shows the per-queue diagnostic for exactly that reason. It also assumes the judge's error rates are stable across the two variants being compared, which fails if v2 changes the output style enough to change how the judge reads it. This fails as a company if teams decide a flattering number is fine as long as it is consistent week to week.
Market
Priced against the eval budget, which every team shipping an agent already has, and against the cost of the labelling it replaces
Every team running an LLM judge in CI is a candidate, and the first time a corrected number contradicts a launch review is the forcing event. Fifteen hundred teams at the Team tier is $9M ARR; the Scale tier is priced against a labelling vendor invoice.