Capacity planning with extreme value theory.
Your p99.9 is a six-week event, not a rare one.
Teams size capacity off the worst thing in their retention window and then get surprised at a rate that feels random. It is not random. Tail fits the shape of your latency tail and tells you what a year actually contains.
Ninety days of hourly peak latency for five services, drawn from tails with known shape parameters. Every fit runs in your browser.
The problem
A percentile of a sample cannot see past the sample.
The p99.9 of 90 days of data is, mechanically, close to the worst thing that happened in 90 days. It carries no information about the tail beyond it, because there is none in the sample to carry. So the capacity plan gets built on roughly the worst day anyone remembers, the plan holds until it does not, and the review calls the outage unprecedented. It was not unprecedented. It was unobserved, which is a different word.
The insight
Tails have a mathematical form, so you can estimate what you have not seen.
Above a high enough threshold, exceedances of almost any distribution converge to a Generalised Pareto. That is a theorem, not a modelling choice, and it means the tail can be fitted from the exceedances you do have and then extrapolated past them. The whole answer lives in one number. A shape parameter above zero means the distribution is heavy-tailed: far worse events are not merely possible, they are expected on a long enough horizon. Below zero means there is a ceiling and you are probably already near it. Insurance prices catastrophe risk this way and hydrology sizes flood defences this way. Infrastructure engineering, which has the same problem, almost never does.
Peaks-over-threshold with a Generalised Pareto fitted by probability-weighted moments, which stays stable at the exceedance counts a few months of data actually give you where maximum likelihood does not converge. Return levels at operational horizons, inverted to give the recurrence interval of whatever level you are currently provisioned at. Latency converted to capacity by Little’s Law, so the output is a replica count rather than a chart.
How it works
Four steps, no data science team
Hourly peaks per service from your metrics store. Ninety days is enough. It needs the samples, not the pre-aggregated percentiles, because the percentiles are the thing that threw the tail away.
One shape parameter per service, with the exceedance count and threshold shown so you can see what the fit is standing on. Services split into the ones with a ceiling and the ones without.
Return levels at a month, a quarter, a year, three years. And the inverse: how often the level you are provisioned at today gets exceeded.
Arrival rate times latency is in-flight requests, and in-flight requests divided by per-replica concurrency is machines. The capacity artifact goes into the quarterly plan with the arithmetic attached.
Who it is for
The SRE or platform owner who signs off the capacity plan
Teams that have already had the incident where the graph went somewhere it had never been, and whose capacity plan is still a percentile with a multiplier bolted on.
Pricing
- –GPD fit and return levels
- –CSV or Prometheus input
- –Public, no account
- –Continuous refits as data arrives
- –Shape-parameter drift alerts
- –Capacity artifacts for the quarterly plan
- –Declustering for correlated exceedances
- –Self-hosted
- –Multi-service dependency tails
- –SSO and audit log
- –Review with a statistician
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Datadog, Grafana, Honeycomb | Compute and chart p50 through p99.9 across every service you have. | Every number they show is a summary of what happened. None of them extrapolates, and none of them will tell you that your p99.9 is a six-week event. |
| Autoscalers and HPA | React to load by adding capacity. | React is the operative word. Scaling has a lead time measured in minutes and a tail event does not wait. Autoscaling also needs a target, and the target is usually a percentile. |
| A headroom multiplier in a spreadsheet | Take the p99, double it, call it the plan. The actual state of the art in most companies. | The multiplier is a guess at the shape parameter. Sometimes 2× is far too little and sometimes it wastes half the fleet, and the spreadsheet cannot tell you which. |
| Load and chaos testing | Push the system to a chosen level and see what breaks. | Answers what happens at a level. Does not answer which level, which is the question the capacity plan is actually asking. |
Two ways this dies. First, exceedances in real infrastructure cluster: one bad deploy produces forty consecutive bad hours, and treating those as forty independent observations inflates the shape parameter and over-provisions the fleet. Declustering is a real engineering problem and getting it wrong once is enough for a capacity owner to stop trusting the tool. Second, and more likely: the number is right and nobody can act on it. A platform lead who reads the 1-in-365 level, cannot get the spend approved, and quietly goes back to the percentile is a churned customer regardless of whether the math was correct.
Market
A line item inside the observability and capacity budget every infrastructure org already has
Companies large enough to have a named capacity owner already spend six figures a year on observability. This sits beside that at a fraction of the price and is sold to the same person. Two thousand teams on the Team tier is roughly $10M ARR, before the larger accounts that have hundreds of services and a formal capacity review.