Find the node group that got slower.
One pool got 28% slower. The fleet average moved 0.7%.
Identical SKUs do not stay identical. A partial driver rollout, a degrading NVLink, a rack that runs hot, and one node group starts returning fewer tokens per second per dollar while every liveness check stays green. Silicon watches efficiency per pool instead of per fleet and names the group and the day.
Fourteen fleet series, one degrading pool, detection computed live in your browser, including the fleet average that never moved.
The problem
Nothing is down, so nothing alerts.
Fleet monitoring was built to answer whether a node is alive. It answers that well. It has no opinion on whether a node that is alive is delivering the throughput you are paying for, because the two questions have different denominators. Aggregate tokens per second across the fleet hides a bad pool by construction, and the pool itself never misses a heartbeat, so the degradation runs until someone happens to compare invoices against throughput by hand.
The insight
Efficiency is a per-pool question. Averaging is what hides it.
Tokens per second per dollar is the only number that says whether the fleet is working, and it is exactly the number that dilutes fastest. One pool in forty can lose a third of its throughput and land inside the daily variation of the fleet aggregate, so the more nodes you add the harder the problem gets to see. The fix is to stop aggregating and test every pool separately, which immediately creates the second problem: forty pools times six metrics is 240 tests, and at α=0.05 with no correction that is twelve false alarms every day. Testing per pool only works if the multiple-comparisons problem is handled, and handling it is most of the engineering.
CUSUM on day-over-day percentage changes of each pool-level efficiency metric, Bayesian Online Changepoint Detection (Adams & MacKay 2007) for abrupt shifts, and Benjamini-Hochberg FDR control across every pool × metric combination in the run. Percentage changes rather than raw levels, so a pool that is simply serving more requests does not read as a regression.
How it works
Four steps, no data science team
DCGM for the cards, your inference server metrics for tokens and requests, your cloud billing export for the rate. No agent, no new instrumentation.
Tokens per second, HBM bandwidth utilisation, SM occupancy, thermal throttle events, ECC corrections, and cost per million tokens, sliced by pool, SKU, driver version, and model served.
Each series runs through CUSUM and BOCD. Whatever crosses is filtered by Benjamini-Hochberg across the whole run, so what survives survives knowing how many pools you tested.
The changepoint has a date. Your driver rollouts, firmware pushes, and rack work have dates. The alert lines them up and shows the healthy peer pools next to the bad one.
Who it is for
The platform engineer who owns the fleet and the bill
Teams running their own inference on owned or reserved capacity, past the point where one person knows every node by name. Usually 200 to 20,000 accelerators, a platform team of 3 to 15, and a finance conversation about cost per token that nobody can currently win.
Pricing
- –Full detection engine
- –14-day history
- –Slack alerts
- –One fleet
- –Unlimited pools and slices
- –13-month history
- –Driver and firmware correlation
- –Cost per million tokens by pool
- –PagerDuty and webhook
- –Self-hosted, air-gapped option
- –Custom efficiency signals
- –SSO and audit log
- –Detection tuning support
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| NVIDIA DCGM plus Grafana | The metrics themselves, on dashboards you build. | It is the data layer, not the decision layer. Someone still has to notice that one pool has drifted, on a dashboard with forty lines on it. |
| Datadog / New Relic GPU monitoring | Ingests DCGM, gives you thresholds and anomaly widgets. | Thresholds are per metric and static. A pool that drops 28% below its own history but stays above the fleet floor never crosses one. |
| Run:ai / Kubernetes schedulers | Placement, quota, and utilisation of the fleet. | Optimises whether the cards are busy. A throttled card is busy. Utilisation goes up when efficiency goes down. |
| Cloud cost tools | Spend by account, project, and instance type. | Denominated in dollars per hour, not dollars per token. Both a healthy pool and a degraded pool cost exactly the same per hour, which is the entire problem. |
The honest failure mode: for most fleets the answer is a shrug. Real inference traffic is heterogeneous, so a pool can genuinely serve longer prompts than its neighbours for a week and produce a true changepoint that nobody caused and nobody can fix. If half the alerts resolve as workload mix, the platform team mutes the channel and the product is over. The defence is to normalise on work done rather than requests, and to require several independent signals in the same pool to move together, since a mix shift moves token rate without touching throttle counts or ECC corrections. That will not save us if a customer routes by content type in a way we cannot see, and we should say so before they buy rather than after.
Market
Priced against wasted accelerator hours, not per seat
A 500-accelerator fleet at typical reserved pricing runs several million dollars a year. One pool of forty degrading by a third for six weeks is a five-figure loss on its own, and it recurs. Two thousand fleets at the Fleet tier is $21M ARR, and the number of companies operating their own inference capacity rather than renting an API is growing every quarter.