Cost routing with a stated error rate
Everyone routes to the cheap model on a hunch
Short prompts to the small one, hard ones to the large one, thresholds picked in a meeting. Router picks the cheapest model that clears your quality bar with a coverage guarantee, and tells you what the guarantee costs.
The problem
A routing rule with no measured error rate is just a way to lose quality quietly
Inference is the largest line on the bill, so every team writes a router. None of them can say what it costs in quality, because the rule was chosen by intuition and the failures land on the small fraction of requests nobody samples. The result is a saving you can measure and a regression you cannot.
The insight
Routing is an abstention problem wearing a cost hat
The question "is this model good enough for this request" is the same shape as "is this agent confident enough to act", and it has the same answer. Split conformal takes a calibration set and returns a lower bound on quality that holds at whatever coverage you ask for, with no assumption about the score being calibrated. Router picks the cheapest model whose bound clears your bar, so the saving comes with a number attached rather than a hope.
One-sided split-conformal lower bounds per model from a graded calibration set, with the finite-sample rank correction; cheapest eligible model selected at the requested coverage, and the realised bar-hit rate checked against the promise on held-out traffic.
How it works
Four steps, no data science team
A few hundred requests scored once, per model. The same set you would use for an eval.
Quality at least 0.8, at least 90% of the time. Both are business decisions, not tuning knobs.
Cheapest model whose conformal bound clears the bar. Everything else goes up a tier.
Realised hit rate against the stated coverage, continuously, so drift shows up before the bill does.
Who it is for
Whoever gets asked why the bill went up
Teams with meaningful inference spend and a quality bar they can articulate. Usually after a finance conversation about the model line.
Pricing
- –Calibration analysis
- –Conformal bounds per model
- –Projected saving
- –Live routing
- –Coverage monitoring
- –Automatic re-calibration
- –Spend reporting
- –Self-hosted proxy
- –Per-workload policy
- –SSO
- –Committed-spend reporting
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| OpenRouter, LiteLLM, Portkey | Gateways that route across providers with fallback and caching. | They route on availability, price and hand-written rules. None attaches a measured error rate to the routing decision, which is the only thing that makes a saving defensible. |
| Not Diamond, Martian and routing startups | Learned routers that predict the best model per request. | Closest competitor, and they optimise a predicted quality score. A prediction is not a guarantee; conformal is the difference between "we think this is fine" and "this holds 90% of the time". |
| Prompt caching and batching | Reduce cost without touching model choice. | Complementary and orthogonal. Neither answers which model to use. |
| A rule in a config file | Route by prompt length or customer tier. | Free, already deployed, and unable to tell you what it costs in quality. |
The guarantee holds only while live traffic resembles the calibration set, and traffic mix shifts whenever the product does — so the staleness detector is load-bearing, exactly as it is for Abstain. Commercially the gateways are one feature away: adding a calibration step to LiteLLM is not hard, and they already sit in the request path with the traffic. The wedge is that doing conformal correctly and explaining why it holds is a statistics argument, and the failure mode of a naive implementation is a guarantee that quietly does not hold. That is a thin moat and an honest one.
Market
A share of the inference bill, which is the fastest-growing line in most AI budgets
Any team spending five figures a month on inference saves multiples of the subscription. A thousand teams at Team pricing is $9.6M ARR, and Scale is priced against a percentage of committed spend.