Live demo
24 segments in. 2 out.
One agent, 7 days, 23,240 requests and 1,045 errors, cut by tier, language, region and channel. The population rate is 4.5%. Sorted by raw rate, the top of the table is the segments with the fewest requests.
| Segment | Requests | Errors | Raw | Shrunk | 95% | q | ||
|---|---|---|---|---|---|---|---|---|
| lang:nl | 50 | 6 | 12.0% | 7.8% | 2.9% to 12.7% | 0.084 | noise | |
| channel:sms | 45 | 5 | 11.1% | 7.2% | 2.4% to 12.1% | 0.129 | noise | |
| lang:ja | 400 | 44 | 11.0% | 10.1% | 7.4% to 12.8% | <0.001 | real | |
| tier:enterprise | 1,800 | 169 | 9.4% | 9.2% | 7.9% to 10.5% | <0.001 | real | |
| lang:it | 55 | 5 | 9.1% | 6.6% | 2.2% to 11.1% | 0.267 | noise | |
| region:af-south | 40 | 3 | 7.5% | 5.7% | 1.2% to 10.1% | 0.575 | ||
| region:ap-south | 250 | 13 | 5.2% | 5.1% | 2.6% to 7.5% | 0.792 | ||
| channel:voice | 40 | 2 | 5.0% | 4.7% | 0.6% to 8.7% | 0.916 | ||
| lang:fr | 180 | 8 | 4.4% | 4.5% | 1.9% to 7.0% | 0.973 | ||
| region:us-east | 3,100 | 135 | 4.4% | 4.4% | 3.6% to 5.1% | 0.844 | ||
| region:eu-west | 1,200 | 52 | 4.3% | 4.3% | 3.2% to 5.5% | 0.897 | ||
| channel:api | 900 | 38 | 4.2% | 4.2% | 3.0% to 5.5% | 0.844 | ||
| channel:email | 120 | 5 | 4.2% | 4.3% | 1.4% to 7.2% | 0.916 | ||
| tier:free | 5,000 | 204 | 4.1% | 4.1% | 3.5% to 4.6% | 0.339 | ||
| lang:en | 4,200 | 158 | 3.8% | 3.8% | 3.2% to 4.3% | 0.104 | ||
| tier:pro | 1,500 | 53 | 3.5% | 3.6% | 2.7% to 4.5% | 0.246 | ||
| channel:web | 2,600 | 91 | 3.5% | 3.5% | 2.8% to 4.2% | 0.085 | ||
| lang:es | 700 | 24 | 3.4% | 3.5% | 2.2% to 4.8% | 0.345 | ||
| channel:slack | 500 | 17 | 3.4% | 3.5% | 2.0% to 5.0% | 0.437 | ||
| lang:pt | 90 | 3 | 3.3% | 3.8% | 0.8% to 6.8% | 0.792 | ||
| lang:de | 320 | 8 | 2.5% | 2.8% | 1.2% to 4.5% | 0.254 | ||
| lang:ko | 42 | 1 | 2.4% | 3.7% | 0.1% to 7.2% | 0.762 | ||
| region:sa-east | 60 | 1 | 1.7% | 3.1% | 0.1% to 6.2% | 0.497 | ||
| region:ca-central | 48 | 0 | 0.0% | 2.6% | 0.0% to 5.5% | 0.319 |
The dashboard rule caught 2 of the 2 real segments and added 3 that were noise, 3 of them with 60 requests or fewer. Shrinkage moves each of those most of the way back to the population; the test and the correction remove what is left.
Assumptions, stated. Errors within a segment are treated as independent draws, so a retry storm from one customer would read as a segment. The prior treats segments as exchangeable, and the pooled rate includes the bad segments, which makes the bar slightly conservative. Against the rates injected into this scenario, the raw table carries a total squared error of 0.0185 and the shrunk table 0.0046. That reduction is the James-Stein result, and it holds every week.