Pipeline Sentinel — live demo · Sixty prototypes, two engines

Live demo

15 series in. 4 alerts out.

A warehouse on dbt, 60 days of nightly runs, 15 checks across four models plus the run itself. On day 44 an upstream vendor changed a source schema. The run stayed green, every test passed, and the numbers downstream started being wrong. Find the night it happened.

Every number on this page is computed in your browser right now, by the same CUSUM and Bayesian changepoint code that runs in production. Nothing here is a screenshot.

The comparison that matters

Against the rule most teams actually ship

The standard configuration is a percentage or sigma threshold against a trailing mean, evaluated per metric. Run it over the exact same 15 series and count what lands in the channel.

Naive rule

alerts sent · alert if |Δ| > 2.5σ vs 7-day mean

Of those, false

fired before anything was wrong

This engine

alerts sent, all correct

Noise reduction

92%

fewer messages, same catch

The false-alarm count is not a judgement call. The incident was injected at a known index, so every naive alert before that index is wrong by construction.

What the engine found

Detection run

Detection run4 alerts15 series tested · 4 fired · 0 suppressed by BH at FDR=0.05

Metric	Baseline	Now	Change	Confidence	Method
Null rate fct_orders.promo_code	0.61%	12.3%	+1913%	99.9%	CUSUM
Row count fct_orders	184.7k	245.4k	+32.8%	99.9%	CUSUM
Freshness lag (min) fct_orders	24.14	79.52	+229%	99.9%	CUSUM
Distinct customer_id ratio fct_orders.customer_id	0.706	0.538	−23.7%	99.9%	CUSUM

Show your work

Null rate · fct_orders.promo_code

Baseline

0.61%

After changepoint

12.3%

Change

+1913%

Confidence

99.9%

CUSUM (Page 1954)

The series60 daysdashed line = baseline mean · red = detected changepoint

The CUSUM accumulatorsmall deviations stack slowly; a real shift stacks fast

Baseline mean μ = 0.0130 and σ = 0.2008, both computed from the first two thirds of the day-over-day percentage changes. The slack k = 0.1004 is half a sigma, and the decision boundary h = 0.8032 is four. The accumulator runs on percentage changes rather than raw levels so a healthy growing series cannot drift across the boundary on its own.

What the naive rule did on this same series

alert if |Δ| > 2.5σ vs 7-day mean fired 5 times across this one metric, including 4 before anything was actually wrong. The statistical pipeline sent one message, on the day the regime actually changed.

What lands in Slack

One message, with the work already done

Pipeline SentinelAPP#eng-alerts

🔴 Null rate jumped +1913%fct_orders.promo_code

Baseline

0.61%

Now

12.3%

Confidence

99.9%

Started

Aug 12

Model

fct_orders

Changepoint

Aug 12

dbt tests failing

Column checks in run

6,140

Likely cause · Four checks on fct_orders changed regime on the same night and no other model moved. promo_code went from effectively never null to null about one row in eight, row count rose about a third while the distinct customer_id ratio fell by the matching amount, and the model landed later. A null spike plus a volume rise plus a cardinality ratio drop, all confined to one model, is a source-shape change feeding a join fan-out rather than a load problem. Every dbt test passed on this run.

1.Diff the payments vendor source schema against the contract as of Aug 11.
2.Check the fct_orders join grain against order_lines for duplicate keys.
3.Recompute the last 15 days of revenue reporting before anyone quotes it.
4.Add a not_null threshold test on promo_code so this becomes a hard failure, not a drift.

AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected

The written cause is generated only after the statistics confirm the change. The model never decides whether something is an anomaly — it explains one that has already been established. Getting that order backwards is how these products hallucinate.