The alert
The whole product is this message
The vendor will not send it, and your own monitoring has no reason to. This arrives on the day their behaviour changed, with the endpoint, the date, and the before and after distributions already attached.
C
Contract DriftAPP#eng-alerts🔴 p95 latency jumped +102%vendor=payments
Baseline
385ms
Now
777ms
Confidence
99.9%
Started
Aug 10
Vendor
payments
Endpoint
POST /v2/charges
Detected
Aug 10, from your own call logs
Calls
4.1M
Likely cause · Four independent signals on vendor=payments break at the same index and nowhere else. Latency doubled, one 4xx subcode went from 1.4% to 5.1%, settlement_id started arriving null on about 15% of responses, and mean payload size fell 39%, which is consistent with a nested object no longer being expanded by default. The 5xx rate never moved, so the vendor status page shows a green month.
- 1.Pull ten flagged responses and diff them field by field against ten from Aug 5.
- 2.Make settlement_id a required field in your parser so the null path stops silently writing empty rows.
- 3.Send the vendor the changepoint date, the subcode, and the payload diff. That is a support ticket they cannot close as "no known issues".
AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected
C
Contract DriftAPP#eng-alerts🔴 Null rate, settlement_id jumped +2285%vendor=payments
Baseline
0.62%
Now
14.7%
Confidence
99.9%
Started
Aug 10
Vendor
payments
Endpoint
POST /v2/charges
Detected
Aug 10, from your own call logs
Calls
4.1M
Likely cause · Four independent signals on vendor=payments break at the same index and nowhere else. Latency doubled, one 4xx subcode went from 1.4% to 5.1%, settlement_id started arriving null on about 15% of responses, and mean payload size fell 39%, which is consistent with a nested object no longer being expanded by default. The 5xx rate never moved, so the vendor status page shows a green month.
- 1.Pull ten flagged responses and diff them field by field against ten from Aug 5.
- 2.Make settlement_id a required field in your parser so the null path stops silently writing empty rows.
- 3.Send the vendor the changepoint date, the subcode, and the payload diff. That is a support ticket they cannot close as "no known issues".
AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected
Why four signals and not one
One metric moving on one vendor is a bad week. Four independent signals breaking at the same index on the same vendor, and nowhere else across thirteen other series, is a deploy. The correlation is what turns a finding into a support ticket the vendor has to answer.