The alert
The whole product is this message
The alternative is a cost dashboard, and cost dashboards get opened during the week the invoice lands. This arrives on the day the number moved, with the route named and the prompt version already lined up against it.
S
SpendGuardAPP#eng-alerts🔴 Cost per request jumped +155%route=document-summarize
Baseline
$0.02
Now
$0.05
Confidence
99.9%
Started
Aug 12
Route
document-summarize
Model
gpt-5.1-mini
Change
prompt v31 to v32, Aug 12 09:41 PT
Requests
11.4M
Likely cause · The changepoint lands the same morning prompt v32 shipped to document-summarize. Input tokens per request went from 1.8k to 4.1k, the prompt cache hit rate collapsed because the shared prefix changed, and retries rose as longer contexts started hitting the output cap. Nothing moved on the other three routes. document-summarize is about 3% of this account’s spend, so the whole regression adds roughly 5% to the bill against underlying growth of 13% a month.
- 1.Diff prompt v31 against v32 on document-summarize, specifically the few-shot block and the early-exit condition.
- 2.Move the static preamble back to the front of the prompt so the provider prefix cache hits again.
- 3.Take the unit numbers to the prompt review, not the invoice: 2.6x cost per request, 2.2x input tokens, cache hit rate 47% down to 20%.
AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected
S
SpendGuardAPP#eng-alerts🔴 Input tokens per request jumped +123%route=document-summarize
Baseline
1839.27
Now
4094.44
Confidence
99.9%
Started
Aug 12
Route
document-summarize
Model
gpt-5.1-mini
Change
prompt v31 to v32, Aug 12 09:41 PT
Requests
11.4M
Likely cause · The changepoint lands the same morning prompt v32 shipped to document-summarize. Input tokens per request went from 1.8k to 4.1k, the prompt cache hit rate collapsed because the shared prefix changed, and retries rose as longer contexts started hitting the output cap. Nothing moved on the other three routes. document-summarize is about 3% of this account’s spend, so the whole regression adds roughly 5% to the bill against underlying growth of 13% a month.
- 1.Diff prompt v31 against v32 on document-summarize, specifically the few-shot block and the early-exit condition.
- 2.Move the static preamble back to the front of the prompt so the provider prefix cache hits again.
- 3.Take the unit numbers to the prompt review, not the invoice: 2.6x cost per request, 2.2x input tokens, cache hit rate 47% down to 20%.
AcknowledgeFalse positiveCUSUM · q < 0.0001 · BH-corrected
Why it is not the spend chart
Total spend went up over this window, and it would have gone up without the regression. There is no threshold on that series that separates the two, which is why the detector is pointed at the ratio instead of the total.