Live demo

14 series in. 5 alerts out.

A support-triage agent, 60 days of traces, 14 series across three customer tiers. On day 45 the team shipped v2.14.0, which rewrote the tool-selection preamble. Task volume never moved. Nothing errored. Find the incident.

Every number on this page is computed in your browser right now, by the same CUSUM and Bayesian changepoint code that runs in production. Nothing here is a screenshot.

The comparison that matters

Against the rule most teams actually ship

The standard configuration is a percentage or sigma threshold against a trailing mean, evaluated per metric. Run it over the exact same 14 series and count what lands in the channel.

Naive rule
88
alerts sent · alert if |Δ| > 20% vs 7-day mean
Of those, false
43
fired before anything was wrong
This engine
5
alerts sent, all correct
Noise reduction
94%
fewer messages, same catch

The false-alarm count is not a judgement call. The incident was injected at a known index, so every naive alert before that index is wrong by construction.

What the engine found

Detection run

Detection run5 alerts14 series tested · 5 fired · 0 suppressed by BH at FDR=0.05
MetricTrendBaselineNowChangeConfidenceMethod
Tool-call error rate
tier=enterprise
2.06%9.90%+380%99.9%CUSUM
Retry rate
tier=enterprise
4.31%13.7%+217%99.9%CUSUM
Steps per task
tier=enterprise
5.1649.246+79.0%99.9%CUSUM
Cost per task
tier=enterprise
$0.04$0.10+132%99.9%CUSUM
p95 latency
tier=enterprise
3.08s5.73s+86.2%99.9%CUSUM

Show your work

Tool-call error rate · tier=enterprise

Baseline
2.06%
After changepoint
9.90%
Change
+380%
Confidence
99.9%
CUSUM (Page 1954)
The series60 daysdashed line = baseline mean · red = detected changepoint
1.13%6.33%11.5%changepointJun 29Aug 27
The CUSUM accumulatorsmall deviations stack slowly; a real shift stacks fast
h = 0.565 (decision boundary)4.990

Baseline mean μ = 0.0130 and σ = 0.1414, both computed from the first two thirds of the day-over-day percentage changes. The slack k = 0.0707 is half a sigma, and the decision boundary h = 0.5654 is four. The accumulator runs on percentage changes rather than raw levels so a healthy growing series cannot drift across the boundary on its own.

What the naive rule did on this same series

alert if |Δ| > 20% vs 7-day mean fired 6 times across this one metric. The statistical pipeline sent one message, on the day the regime actually changed.

What lands in Slack

One message, with the work already done

A
AgentSREAPP#eng-alerts
🔴 Tool-call error rate jumped +380%tier=enterprise
Baseline
2.06%
Now
9.90%
Confidence
99.9%
Started
Aug 13
Agent
support-triage
Deploy
v2.14.0
Shipped
Aug 12, 14:22 PT
Traces
312,880
Likely cause · The changepoint lands within 40 minutes of deploy v2.14.0, which rewrote the tool-selection preamble. Error and retry rates rose only on tier=enterprise, whose conversations carry a longer tool manifest — consistent with the model losing the correct tool in a longer list. Task volume is unchanged, so this would not appear on a throughput dashboard.
  1. 1.Diff the tool-selection preamble between v2.13.4 and v2.14.0.
  2. 2.Replay the 40 flagged enterprise traces against v2.13.4 to confirm.
  3. 3.If confirmed, roll back the preamble and keep the rest of v2.14.0.
AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected

The written cause is generated only after the statistics confirm the change. The model never decides whether something is an anomaly — it explains one that has already been established. Getting that order backwards is how these products hallucinate.