Deterministic replay for agent tests
Your agent test suite passes 61% of the time on unchanged code
So the team set CI to retry three times and stopped reading the failures. Replay records every model call once and replays it pinned, which turns a failing test back into information.
The problem
A retry is a decision to stop learning from your tests
Agent suites are nondeterministic because the model is, so a red build means nothing and everybody knows it. The rational response is to retry until green, and the cost of that response is that a genuine regression now looks exactly like the flake you have been ignoring for a month.
The insight
You cannot measure flake with the sample size CI gives you
Run a suite five times, see one failure, and the honest estimate of the flake rate is not 20% — it is somewhere between 1% and 65%. Every team that eyeballs this gets it wrong in the same direction, because a step that has not flaked yet reads as deterministic. Replay runs the suite enough times to separate the steps that are genuinely stable from the ones that have merely been lucky, and reports the Wilson interval rather than a point estimate that cannot support the decision being made on it.
Per-step output hashing across repeated runs, with distinct-output counts, Shannon entropy and Wilson score intervals on the flake rate; suite-level pass probability compounded across steps, and pinned replay to eliminate model nondeterminism from the test path.
How it works
Four steps, no data science team
Every model call and tool result from a clean run, hashed and stored.
Repeat the suite, per-step, and get an interval rather than a guess.
Tests run against the recording, so a failure means the code changed.
When the model or prompt changes, the diff against the old recording is the review.
Who it is for
The engineer who maintains the agent test suite
Teams with an agent test suite they have stopped trusting. Usually the ones who added retries and then noticed a real bug had been retried away.
Pricing
- –Per-step flake rates
- –Wilson intervals
- –Suite pass probability
- –Record and replay
- –Recording diffs on model change
- –CI integration
- –Flake trend tracking
- –Self-hosted recordings
- –Retention policy
- –SSO
- –Priority support
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| VCR, nock and HTTP replay libraries | Record and replay HTTP traffic in tests. | The right idea, built before agents. They do not understand a step boundary, cannot diff a recording semantically, and say nothing about how flaky you were to begin with. |
| LangSmith, Braintrust, Langfuse | Trace and evaluate agent runs. | They observe production and score outputs. Neither makes your test suite deterministic, which is a different job. |
| CI flake detection | Quarantine tests that fail intermittently. | Built for flake caused by timing and shared state. It quarantines the test rather than removing the nondeterminism, which for an agent suite means quarantining everything. |
| Retry three times | What the team already does. | Free, effective at making the build green, and the reason nobody notices a real regression. |
This is a feature more than a company, and the honest version of the pitch says so — it is the sort of thing an agent framework ships in a minor release. It is also genuinely useful today, which makes it a good wedge and a poor destination: the strongest version of this is Replay as part of AgentSRE rather than a separate purchase. The technical risk is recording drift, because a pinned recording slowly stops resembling live behaviour, and a suite that passes against a stale recording is worse than a flaky one.
Market
Priced off the CI budget, where a per-suite tool is an easy approval
Every team with an agent in CI, which is most teams past the demo. Low price point and short sales cycle; better as a wedge into a reliability platform than as the platform.