The alert
A date, a channel, and a region of the curve
The useful part of a retention alert is not that retention moved. It is which part of the curve moved, on which cohorts, starting when, and which adjacent signals moved with it. That is the difference between a debate in a growth meeting and a diff against last week’s landing page.
R
Retention CliffAPP#eng-alerts🔴 Hours to second session jumped +94.9%channel=paid_social
Baseline
29.022
Now
56.574
Confidence
99.9%
Started
Aug 9
Channel
paid_social
Cohorts
signup date ≥ Aug 9
Changed
creative set 14 + new LP
Users
38,410
Likely cause · Every signal that moved belongs to paid_social cohorts starting Aug 9. The shape of the break is diagnostic: day-7 activation and week-2 retention fell together, while sessions per active user dropped and time to second session nearly doubled. That is a first-session problem, not a long-run value problem, and it points at what the user landed on rather than at the product they eventually got to. Blended month-3 retention is unchanged because these cohorts are 18 days old, so month-3 is not observable for them yet.
- 1.Diff the paid_social landing page and creative set against the Aug 8 version.
- 2.Compare first-session event streams for cohorts on either side of Aug 9.
- 3.Check whether the new creative changed the promise the landing page pays off.
AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected
R
Retention CliffAPP#eng-alerts🔴 Week-2 retention dropped −37.3%channel=paid_social
Baseline
41.1%
Now
25.8%
Confidence
99.9%
Started
Aug 9
Channel
paid_social
Cohorts
signup date ≥ Aug 9
Changed
creative set 14 + new LP
Users
38,410
Likely cause · Every signal that moved belongs to paid_social cohorts starting Aug 9. The shape of the break is diagnostic: day-7 activation and week-2 retention fell together, while sessions per active user dropped and time to second session nearly doubled. That is a first-session problem, not a long-run value problem, and it points at what the user landed on rather than at the product they eventually got to. Blended month-3 retention is unchanged because these cohorts are 18 days old, so month-3 is not observable for them yet.
- 1.Diff the paid_social landing page and creative set against the Aug 8 version.
- 2.Compare first-session event streams for cohorts on either side of Aug 9.
- 3.Check whether the new creative changed the promise the landing page pays off.
AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected
What is not in the message
Blended month-3 retention. It has no finding, and in this scenario it could not have one: the affected cohorts are eighteen days old, so their month-3 number does not exist yet. The metric everyone reports on is structurally the last one to know, which is the reason to watch the curve instead of the summary.