The alert
The message that ends the argument
Not “it feels worse.” A date, a capability, a magnitude, and an interval — the kind of thing you can put in a vendor support ticket and have taken seriously.
D
Drift RadarAPP#eng-alerts🔴 JSON schema adherence dropped −8.1%vendor-a / pinned-2026-05
Baseline
97.3%
Now
89.4%
Confidence
99.9%
Started
Aug 12
Suite
1,400 frozen tasks
Model
vendor-a, pinned
Runs
3× daily, temp 0
Version string
unchanged
Likely cause · Five signals on the same pinned model shift within one day of each other, and none of the other tracked models move. Output length rose 34% while p50 latency fell 26% — faster and more verbose at once is the signature of a serving-side change rather than task drift, most consistent with a quantisation or routing change behind an unchanged version string. The vendor published no changelog entry.
- 1.Pin to the dated snapshot if the vendor offers one.
- 2.Re-run your own eval suite against the affected capabilities before the next release.
- 3.Tighten JSON parsing and retry logic on the two tool-calling paths that depend on schema adherence.
AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected
D
Drift RadarAPP#eng-alerts🔴 Tool argument validity dropped −7.5%vendor-a / pinned-2026-05
Baseline
95.8%
Now
88.6%
Confidence
99.9%
Started
Aug 12
Suite
1,400 frozen tasks
Model
vendor-a, pinned
Runs
3× daily, temp 0
Version string
unchanged
Likely cause · Five signals on the same pinned model shift within one day of each other, and none of the other tracked models move. Output length rose 34% while p50 latency fell 26% — faster and more verbose at once is the signature of a serving-side change rather than task drift, most consistent with a quantisation or routing change behind an unchanged version string. The vendor published no changelog entry.
- 1.Pin to the dated snapshot if the vendor offers one.
- 2.Re-run your own eval suite against the affected capabilities before the next release.
- 3.Tighten JSON parsing and retry logic on the two tool-calling paths that depend on schema adherence.
AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected
On being fair to the provider
The suite is frozen, the temperature is zero, the models are pinned, and the methodology is public. That is what makes a finding checkable rather than an accusation — and it is also what makes it useful to the provider, who often has less visibility into downstream capability regressions than the customers do.