The artifact
Not a notification. A decision on every request.
This sits inline in the agent loop and returns one of two things before any answer reaches a customer. What makes it different from the constant it replaces is that every record carries the threshold in force and the budget that threshold was derived from, so a decision from six months ago can still be explained.
Realised error rate on the last 2,000 answered tickets: 2.2%. The threshold it replaced, 0.9, was delivering 5.9% on the same traffic.
| Ticket | Queue | Score | Decision |
|---|---|---|---|
T-01080 Charged twice for the November invoice | Billing | 0.9528 | Answer |
T-00968 Exchange sent the wrong size twice | Returns | 0.9527 | Answer |
T-02021 Password reset email never arrives | Account access | 0.9526 | Answer |
T-00884 Rate limit hit at a quarter of the documented ceiling | Integrations | 0.9526 | Answer |
T-01053 Return label expired before pickup | Returns | 0.9526 | Answer |
T-01606 SSO login loops back to the sign-in page | Account access | 0.9526 | Escalate |
T-02028 Return label expired before pickup | Returns | 0.9525 | Escalate |
T-02091 Cannot remove a seat from the team plan | Account access | 0.9524 | Escalate |
The last answer and the first escalation are 0.0000 apart on a score that runs from zero to one. That is not a meaningful distinction between two tickets, and pretending otherwise would be dishonest. A line has to go somewhere. The only claim being made is that this line holds the 2.0% budget on held-out traffic and the old one did not.
{
"ticket": "T-01606",
"queue": "Account access",
"decision": "escalate",
"score": 0.9526,
"threshold": 0.9526,
"policy": {
"method": "split_conformal_risk_control",
"target_error_rate": 0.02,
"calibration_n": 800,
"calibration_error_rate": 0.0191,
"expected_escalation_rate": 0.3462
},
"guarantee": "error rate among answered tickets held at or below target while live traffic is exchangeable with the calibration set"
}Any framework can branch on a number. What no framework gives you is the ability to answer, months later, why this particular customer got an automated answer and what error rate the company had accepted at the time. That record is what turns an agent deployment into something a risk function will sign off on, and it falls out of the method rather than being bolted on.