Live demo
A support agent, 2,000 tickets, and one number nobody can justify.
Each ticket carries the confidence the agent reported and whether a human later found the answer correct. Every number below is computed in your browser from those outcomes. Nothing is a stored result.
That gap is not the problem on its own. Plenty of useful scores are miscalibrated. The problem is what happens next: the team writes escalate if confidence < 0.90 into a config, and from that moment on nobody in the building can say what error rate the agent is delivering to customers.
What the number actually buys
Five plausible thresholds. Five different error rates.
Measured on 2,000 held-out tickets. Nothing here was available at the moment somebody picked 0.90, because it requires labelled outcomes and a calculation nobody ran.
| Threshold | Answered | Escalated | Wrong answers | Error rate delivered |
|---|---|---|---|---|
| 0.80 | 1,937 | 3.1% | 201 | 10.4% |
| 0.85 | 1,881 | 5.9% | 169 | 9.0% |
| 0.90shipped | 1,743 | 12.8% | 103 | 5.9% |
| 0.95 | 1,349 | 32.6% | 31 | 2.3% |
| 0.99 | 270 | 86.5% | 0 | 0.0% |
The inversion
Choose the error rate. The threshold is an output.
Split conformal never tries to fix the confidence score. It reads a threshold off 800 labelled tickets and holds your budget on the answers given. Everything below is then measured on 2,000 tickets the calibrator never saw.
The guarantee is finite-sample, not asymptotic, and it is stated on data the threshold never saw. At a 2.0% budget the calibration slice comes in at 1.9% by construction and the held-out traffic lands at 2.2%. The distance between those two is sampling noise on 800 labelled tickets, and it shrinks as you label more. It is not a tuning knob.
| Error budget | Threshold | Escalation | Held-out error rate | Humans per 1,000 tickets |
|---|---|---|---|---|
| 0.5% | 0.968 | 50.5% | 0.6% | 505 |
| 1.0% | 0.963 | 44.7% | 1.0% | 447 |
| 2.0% | 0.953 | 34.6% | 2.2% | 346 |
| 5.0% | 0.916 | 17.1% | 4.6% | 171 |
| 10.0% | 0.846 | 5.6% | 9.2% | 56 |
Nothing on this table is a modelling choice. It is the same agent and the same score read at five different budgets, and the last column is the staffing question a support director can answer without knowing what conformal prediction is.
The failure that ends the argument
The model got better. The deployment got worse.
Same agent, same queues, new base model. Accuracy went from 87.0% to 89.9%. The upgrade also made the model more confident about everything, which nobody measures, so the 0.90 threshold stopped filtering. Nothing in the config changed and nothing alerted.
Mean reported confidence moved from 0.949 to 0.977, so almost everything now clears 0.90. The team shipped an accuracy improvement and increased customer-facing errors by 61 tickets in the same window.
The threshold moved a long way because it is an output of the data rather than a constant. The error rate did not move, because that is the thing being held fixed. This is also the only view in which the upgrade can be priced: the budget is unchanged and it now costs 34.0% of traffic instead of 34.6%.
It does not make the agent more accurate, and it does not decide anything about the escalated tickets. It only guarantees the error rate among answers the agent gives on its own, and only while live traffic resembles the calibration slice. When the ticket mix moves, the slice has to be refreshed. A stale calibration set is the one way this method lies to you, which is why the threshold is re-derived on a rolling window rather than pinned.