Pipeline Sentinel

The alert

The whole product is this message

The dbt run this night was green. Nothing failed, nothing retried, nothing paged. This is the message that arrives anyway, naming the model, the column, and the night the distribution changed.

P
Pipeline SentinelAPP#eng-alerts
🔴 Null rate jumped +1913%fct_orders.promo_code
Baseline
0.61%
Now
12.3%
Confidence
99.9%
Started
Aug 12
Model
fct_orders
Changepoint
Aug 12
dbt tests failing
0
Column checks in run
6,140
Likely cause · Four checks on fct_orders changed regime on the same night and no other model moved. promo_code went from effectively never null to null about one row in eight, row count rose about a third while the distinct customer_id ratio fell by the matching amount, and the model landed later. A null spike plus a volume rise plus a cardinality ratio drop, all confined to one model, is a source-shape change feeding a join fan-out rather than a load problem. Every dbt test passed on this run.
  1. 1.Diff the payments vendor source schema against the contract as of Aug 11.
  2. 2.Check the fct_orders join grain against order_lines for duplicate keys.
  3. 3.Recompute the last 15 days of revenue reporting before anyone quotes it.
  4. 4.Add a not_null threshold test on promo_code so this becomes a hard failure, not a drift.
AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected
P
Pipeline SentinelAPP#eng-alerts
🔴 Row count jumped +32.8%fct_orders
Baseline
184.7k
Now
245.4k
Confidence
99.9%
Started
Aug 12
Model
fct_orders
Changepoint
Aug 12
dbt tests failing
0
Column checks in run
6,140
Likely cause · Four checks on fct_orders changed regime on the same night and no other model moved. promo_code went from effectively never null to null about one row in eight, row count rose about a third while the distinct customer_id ratio fell by the matching amount, and the model landed later. A null spike plus a volume rise plus a cardinality ratio drop, all confined to one model, is a source-shape change feeding a join fan-out rather than a load problem. Every dbt test passed on this run.
  1. 1.Diff the payments vendor source schema against the contract as of Aug 11.
  2. 2.Check the fct_orders join grain against order_lines for duplicate keys.
  3. 3.Recompute the last 15 days of revenue reporting before anyone quotes it.
  4. 4.Add a not_null threshold test on promo_code so this becomes a hard failure, not a drift.
AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected
Why 4 messages and not 15

Every one of the 15 column checks in this run was tested by both detectors. 4 cleared the confidence floor, and Benjamini-Hochberg then set the alert budget across the whole run rather than per check. At real warehouse scale that is thousands of tests a night, where per-check thresholds produce a steady drip of alarms and the tool gets muted.

The ordering rule

Statistics decide whether something happened. The model only explains what already cleared that bar, and only ever sees a confirmed changepoint. Feeding raw column profiles to a model and asking “does this look wrong?” is how these products end up confidently wrong.