Earn agent autonomy with evidence, not a week
"It has been clean for a week" is not evidence about a 2% error rate
Teams decide when an agent may act without a human by feel. Ladder runs a sequential test on every outcome against the rung's error ceiling, promotes the day the evidence clears the bar, and demotes the day it stops clearing it, with a guarantee attached to both.
The problem
A week of clean days at 80 actions a day tells you almost nothing
A reviewer reads ten of the day's eighty actions. Seven days with no error in that sample feels like proof. At a 3% error rate, over budget for unsupervised action, a ten-a-day sample produces seven clean days in a row nine times out of ten. Applied to all eighty labels, the same rule almost never promotes anyone, including an agent that is genuinely fine. And once promoted, nobody re-checks after the model update.
The insight
The decision is sequential, so the test has to be
Nobody decides autonomy at a fixed sample size chosen in advance. They look every day and act when it looks good, which is exactly the situation where a fixed-horizon test lies. A mixture sequential probability ratio test on the error rate is valid at every look: the likelihood ratio in favour of "under the ceiling" is a supermartingale under the null, so the chance it ever crosses 1/alpha is at most alpha, no matter how often you check or when you stop. The same machinery runs in the other direction for demotion, and it restarts at every model update, because the test assumes the thing being measured is the same thing throughout.
One-sample mixture SPRT on a Bernoulli error rate against a fixed ceiling, alternatives mixed over a half-normal prior on the log-odds shift, decided at ln(1/alpha) by Ville's inequality; one test per direction, evidence reset on every deployment.
How it works
Four steps, no data science team
Suggest only, act with approval, act unsupervised. Each carries the error ceiling an agent must be shown below to climb off it.
Every action gets a pass or fail from whatever ground truth you have: a reversal, an escalation, a downstream check, a thumbs-down.
The test runs on every outcome. The day the evidence clears ln(1/alpha), the agent moves up, and the policy document says which day and why.
A fresh test watches the new rung and restarts on every deployment. When the rate is shown above the bar it cleared, the agent goes back down the same day.
Who it is for
Whoever signs off on letting the agent act unsupervised
Teams letting an agent take real actions, refunds, edits, deploys, sends, who have to decide how much of that happens without a human in the loop.
Pricing
- –Three rungs
- –Sequential test on every outcome
- –Promotion and demotion days
- –Enforced rungs
- –Deploy-triggered resets
- –Autonomy policy document
- –Slack on every rung change
- –Custom ladders
- –Per-decision audit trail
- –SSO and SCIM
- –Self-hosted
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Human-in-the-loop platforms | Route agent actions to a human for approval. | They implement the rung. They have no opinion on when an agent should move between rungs, so the decision stays with a gut feeling and a calendar. |
| Feature-flag percentages | Roll autonomy out to a growing share of traffic. | A percentage is a dial, not a decision. Nothing in a flag system tests the error rate against a ceiling or says when to turn the dial back down. |
| Runbooks and gut feel | A written policy that says "promote after a clean week". | The rule is uninformative at the volume a reviewer can read and unreachable at the volume the agent runs. Either way it is not a test of the rate the policy names. |
| "It has been clean for a week" | What almost everyone does today. | Promotes nine in ten over-budget agents on a review sample, never re-checks after a model update, and carries no statement about what could go wrong. |
The HITL platforms and the agent frameworks own the enforcement point and could add a sequential test in a sprint. The defensible part is the discipline around resets and labels, not the SPRT. The test assumes outcomes are exchangeable between resets: an agent whose error rate drifts with the day of the week, or an upstream model that changes behind the same API name, violates that, and the demotion latency without a reset is weeks rather than days, which the demo shows. It also assumes the labels are honest; an outcome label that only arrives when a customer complains undercounts errors, and a test on undercounted errors promotes with false confidence. If teams decide a rung is a product decision rather than a measurement, this becomes a chart nobody acts on.
Market
Priced against the review headcount that currently sits between the agent and the action
Every team that lets an agent act is going to have to decide how much of that happens unsupervised, and the first incident after a promotion is the forcing event. Two thousand teams at the Team tier is $12M ARR; the Scale tier is priced against the reviewer it replaces.