Drift Radar — live demo · Sixty prototypes, two engines

Live demo

14 series in. 5 alerts out.

A frozen 1,400-task suite, run three times daily at temperature zero against three vendors’ pinned models for 60 days. On day 44 one vendor changed something behind an unchanged version string. The overall benchmark score barely moved. Find it.

Every number on this page is computed in your browser right now, by the same CUSUM and Bayesian changepoint code that runs in production. Nothing here is a screenshot.

The comparison that matters

Against the rule most teams actually ship

The standard configuration is a percentage or sigma threshold against a trailing mean, evaluated per metric. Run it over the exact same 14 series and count what lands in the channel.

Naive rule

alerts sent · alert if |Δ| > 2.5σ vs 7-day mean

Of those, false

fired before anything was wrong

This engine

alerts sent, all correct

Noise reduction

89%

fewer messages, same catch

The false-alarm count is not a judgement call. The incident was injected at a known index, so every naive alert before that index is wrong by construction.

What the engine found

Detection run

Detection run5 alerts14 series tested · 5 fired · 0 suppressed by BH at FDR=0.05

Metric	Baseline	Now	Change	Confidence	Method
JSON schema adherence vendor-a / pinned-2026-05	97.3%	89.4%	−8.1%	99.9%	CUSUM
Tool argument validity vendor-a / pinned-2026-05	95.8%	88.6%	−7.5%	99.9%	CUSUM
Long-context recall @ 100k vendor-a / pinned-2026-05	88.2%	79.9%	−9.5%	99.9%	CUSUM
Mean output tokens vendor-a / pinned-2026-05	416.27	552.31	+32.7%	99.9%	CUSUM
p50 latency vendor-a / pinned-2026-05	890ms	654ms	−26.5%	99.9%	CUSUM

Show your work

JSON schema adherence · vendor-a / pinned-2026-05

Baseline

97.3%

After changepoint

89.4%

Change

−8.1%

Confidence

99.9%

CUSUM (Page 1954)

The series60 daysdashed line = baseline mean · red = detected changepoint

The CUSUM accumulatorsmall deviations stack slowly; a real shift stacks fast

Baseline mean μ = -0.0002 and σ = 0.0054, both computed from the first two thirds of the day-over-day percentage changes. The slack k = 0.0027 is half a sigma, and the decision boundary h = 0.0217 is four. The accumulator runs on percentage changes rather than raw levels so a healthy growing series cannot drift across the boundary on its own.

What the naive rule did on this same series

alert if |Δ| > 2.5σ vs 7-day mean fired 4 times across this one metric, including 2 before anything was actually wrong. The statistical pipeline sent one message, on the day the regime actually changed.

What lands in Slack

One message, with the work already done

Drift RadarAPP#eng-alerts

🔴 JSON schema adherence dropped −8.1%vendor-a / pinned-2026-05

Baseline

97.3%

Now

89.4%

Confidence

99.9%

Started

Aug 12

Suite

1,400 frozen tasks

Model

vendor-a, pinned

Runs

3× daily, temp 0

Version string

unchanged

Likely cause · Five signals on the same pinned model shift within one day of each other, and none of the other tracked models move. Output length rose 34% while p50 latency fell 26% — faster and more verbose at once is the signature of a serving-side change rather than task drift, most consistent with a quantisation or routing change behind an unchanged version string. The vendor published no changelog entry.

1.Pin to the dated snapshot if the vendor offers one.
2.Re-run your own eval suite against the affected capabilities before the next release.
3.Tighten JSON parsing and retry logic on the two tool-calling paths that depend on schema adherence.

AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected

The written cause is generated only after the statistics confirm the change. The model never decides whether something is an anomaly — it explains one that has already been established. Getting that order backwards is how these products hallucinate.