Which segments are really worse
Your error rate is 4%. Which segments are actually worse?
Group an agent's errors by tier, language, region and channel and the dashboard shows a dozen segments at eight to fifteen percent, most of them with forty requests. The team chases the loud ones and misses the two that are genuinely broken. Uneven shrinks, tests and corrects, and hands back the two.
The problem
A group-by sorted by rate is a list of the smallest segments
Every small segment has a wide sampling distribution, so the top and bottom of any rate table belong to whoever has the fewest requests. At forty requests and a four percent process, four errors is a normal week and reads as ten percent. The team opens the table when an enterprise customer complains, sorts by error rate, and spends the week on Voice and SMS while the segment that is actually failing sits in the middle of the list at nine percent on eighteen hundred requests.
The insight
Shrink first, then test, then correct
James-Stein showed in 1961 that pulling each group's estimate toward the population mean, in proportion to how little data it has, lowers total error. Always, for three or more groups. Fit the prior from the data and a forty-request segment moves almost all the way to the population; an eighteen-hundred-request segment barely moves. That alone reorders the table correctly. Then each segment is tested against the pooled rate, and because twenty-four tests is twenty-four chances to be fooled, Benjamini-Hochberg decides which of them are findings. What survives is short, and it is the right two.
Beta-binomial empirical Bayes with the prior fitted by moments from the pooled rate, a two-sided binomial test of each segment against the pooled rate, and Benjamini-Hochberg across segments at q < 0.05, reporting only segments whose shrunk rate sits above the population.
How it works
Four steps, no data science team
Requests and errors per segment for the window. Uneven reads the same table your dashboard renders; no traces required.
The prior strength is estimated from the spread beyond sampling noise and reported in requests, so you can see how much a forty-request segment was discounted and why.
Each segment against the pooled rate, then BH across all of them. The output names the segments above the population that survive, and counts the ones that did not.
A one-page review with raw, shrunk and adjusted p per real segment, and the population rate as the bar. Forwardable to the customer success lead who asked.
Who it is for
The engineer who owns the agent's quality and gets asked why enterprise customers complain
Teams running an agent in production across enough customers to have segments, and enough enterprise accounts that someone asks why theirs is worse.
Pricing
- –Shrinkage with fitted prior
- –BH across segments
- –Segment review page
- –Trace connectors
- –Weekly review by email
- –Segment history
- –Alert when a real segment appears
- –Per-customer segments
- –Self-hosted
- –SSO
- –Export to your warehouse
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Langfuse, Braintrust dashboard group-bys | Break scores and error rates down by any metadata field. | The breakdown is raw rates in a sortable table. No shrinkage, no test, no correction, so the sort order is the sample-size order. |
| Datadog, Grafana breakdowns | Error rate by tag, with alert thresholds per series. | A threshold on a raw rate fires on every small segment every few weeks. Teams answer by raising it until it fires on nothing. |
| Fairlearn-style toolkits | Disaggregated metrics across groups, built for classifier fairness audits. | Built for a fixed test set with known labels, not a week of production traffic with sample sizes spanning two orders of magnitude, and they report the disparity without asking whether it is noise. |
| Sort the group-by by error rate | What everyone does today. | Sorting by a rate ranks by sample size. The first screen is the tiny segments; the real ones are on the second. |
The observability vendors could add shrinkage to a group-by in a sprint, and one of them probably will once a customer asks for it. The defensible part is the review, delivered, with the noise counted, rather than another column in a table. For this to fail, teams would have to stop cutting by segment, or their segments would have to overlap so heavily (every request is in a tier and a language and a region) that the segments are far from independent. BH holds under positive dependence, but the prior treats segments as exchangeable, which is wrong when one dimension is systematically different from the others. Errors within a segment are modelled as independent draws; a retry storm from one customer is one incident counted forty times, and the test will call that a segment. The demo shows the prior strength and the population rate so those assumptions are visible.
Market
Priced against the observability budget, as the review layer on top of the group-by that already exists
Every team with an agent in production and enterprise customers who compare notes is a candidate, and the first "why is ours worse" escalation is the buying event. Two hundred teams at the Team tier is $1.1M ARR; five hundred is $2.7M. The Scale tier is priced for per-customer segments, where the count of segments is the count of accounts.