Drift detection for agents in production.
Your agent got worse three weeks ago. Nothing alerted.
Agents degrade quietly. A prompt edit, a model update, a tool that starts timing out on one customer segment. Throughput never moves, so no dashboard turns red. AgentSRE watches production traces statistically and names the deploy that did it.
Fourteen real series, one injected incident, detection computed live in your browser.
The problem
Evals catch what you thought to test. Production is everything else.
An offline eval suite runs on cases you wrote down, scored by a judge, before release. Production traffic is unlabeled, unbounded, and changes underneath you. There is no ground truth to score against, so most teams simply stop measuring after launch, and find out about degradation from a support ticket.
The insight
You do not need labels to know the behaviour changed.
Every observability vendor in this category needs a score before it can tell you anything: a judge, a rubric, a labeled set. That is why they all live in the eval loop and stop at the deploy boundary. But an agent that starts picking the wrong tool changes its own behaviour in ways you can measure without ever knowing whether an answer was right: it retries more, takes more steps, spends more tokens, runs longer. Those signals are unlabeled, free, and already in your traces. Detecting a sustained shift in them is a changepoint problem, and changepoint detection has been solved since 1954.
CUSUM on day-over-day percentage changes for sustained shifts, Bayesian Online Changepoint Detection (Adams & MacKay 2007) for abrupt regime changes, and Benjamini-Hochberg FDR control across every metric × slice × agent-version combination, because testing 400 combinations at α=0.05 without correction produces 20 false alarms per run.
How it works
Four steps, no data science team
OpenTelemetry GenAI semantic conventions, or an export from whatever you already run. No new instrumentation, no labels, no judge.
Tool-call error rate, retry rate, steps per task, tokens and cost per task, latency percentiles, refusal rate, and embedding drift on inputs and outputs, sliced by customer tier, route, model, and agent version.
Each series runs through CUSUM and BOCD. Whatever crosses is then filtered by Benjamini-Hochberg across the whole run, so the alerts that survive are the ones that survive knowing how many tests you ran.
The changepoint has a timestamp. Your deploys have timestamps. The alert correlates them and links the traces on both sides of the line.
Who it is for
The engineer who owns the agent at 2am
Teams running agents in production past the demo stage. Usually 3 to 40 engineers, already paying for tracing, already burned once by a silent regression they found out about from a customer.
Pricing
- –Full detection engine
- –7-day history
- –Slack alerts
- –One agent
- –Unlimited agents and slices
- –90-day history
- –Deploy correlation
- –PagerDuty + webhook
- –Per-slice detection
- –Self-hosted option
- –Custom signals
- –SSO and audit log
- –Detection tuning support
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Braintrust | Eval automation with significance testing and merge blocking in CI. | Needs labeled eval cases. Stops at the deploy boundary. Nothing runs against unlabeled production traffic. |
| Langfuse / LangSmith | Tracing, dashboards, and manual review of production traces. | Stores and displays. You still have to notice the change yourself, on a dashboard nobody opens daily. |
| Arize Phoenix | ML observability with embedding drift and traditional model monitoring. | Built for feature-vector drift in classical ML. Agent behaviour is a step sequence, not a feature vector, and the alerting is threshold-based. |
| Datadog / Grafana | Infrastructure monitoring with anomaly detection add-ons. | Watches the process, not the reasoning. HTTP 200 with a wrong tool call looks perfectly healthy. |
The honest failure mode: an existing observability vendor ships label-free changepoint detection as a feature. Langfuse and Braintrust already own the ingestion pipe, so they would not have to earn the data. The defence is to be the engine rather than the dashboard: integrate into whatever people already run instead of asking them to switch, and be materially better at the detection itself rather than marginally better at everything.
Market
Priced off the tracing budget, not a new line item
Teams running agents in production already pay $200–$2,000/month for tracing. This sits beside that spend, not instead of it. Ten thousand companies at the Team tier is $48M ARR, and the number of companies with agents in production is growing faster than the number with models in production ever did.