Tickets the support bot actually removed
The vendor says 35% deflected. Volume fell on every queue that month
A support agent launched on one queue and the dashboard reports the share of conversations it resolved. Ticket volume also fell, on that queue and on the seven it never touched. Deflect measures how many tickets the bot made go away against the queues that got no bot.
The problem
"Resolved by AI" counts conversations the bot touched, not tickets that went away
The vendor dashboard counts a conversation as resolved when the customer stops replying after the bot answers. Many of those would have ended the same way with a help-centre article, or with nobody. Meanwhile the number the head of support is asked to defend is the ticket count, and the ticket count moved for reasons that have nothing to do with the bot: the season, a product fix, a pricing change. Before/after on the treated queue takes credit for all of it. At renewal, nobody in the room can say what the bot did.
The insight
The queues the bot never touched are the counterfactual, if they tracked before launch
Nobody randomised the launch, so this is not an A/B test. But seven queues went through the same season, the same product changes and the same customers without a bot. A weighted blend of them that tracked the billing queue for six months before launch is a credible statement of what billing would have done after. The gap between billing and that blend is the deflection. The season cancels because it hit the blend too. And if the blend did not track before launch, the honest answer is that there is no control, and Deflect says that instead of a number.
Synthetic control with simplex-constrained donor weights fitted on the pre-launch weeks, reported only when pre-period RMSE clears a stated tolerance, with a difference-in-differences permutation p-value deciding whether anything happened and a placebo refit on an untreated queue checking that the method finds nothing where nothing was done.
How it works
Four steps, no data science team
Weekly volume per queue from the helpdesk. No conversation transcripts, no bot logs.
A blend of the untreated queues that tracks the treated one on the pre-launch weeks. The tracking error is the first number reported.
A permutation test says whether the queue moved against its control at all. Only then does the gap become a deflection estimate and a tickets-per-week number.
The same fit on a queue that got no bot. It has to find nothing, and the memo shows that it did.
Who it is for
The head of support or CX ops who bought the agent and has to defend the renewal
Support and CX organisations that bought an AI agent for one queue, are three to six months in, and have a renewal or an expansion decision coming.
Pricing
- –Synthetic control estimate
- –Pre-period fit check
- –Placebo on one queue
- –Weekly refresh
- –All queues as donors
- –Cost per deflected ticket
- –Renewal memo export
- –Staggered-launch designs
- –Cross-brand donor pools
- –SSO
- –Finance-ready export
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Intercom Fin, Zendesk AI, Decagon, Sierra | The agent, and a dashboard reporting the share of conversations it resolved. | The vendor grades its own work, and resolution is defined as the customer going quiet. None of them compare the queue to the queues they are not on. |
| BI tools | Ticket volume over time, sliced by queue. | Show the drop. Cannot say how much of it is the bot, because they do not build a counterfactual and were never asked to. |
| Support analytics platforms | Deflection rate, CSAT, handle time, by channel. | Deflection is computed as help-centre views that did not become tickets, which is a different quantity and equally uncontrolled. |
| Before/after on the queue | What almost everyone does today. | Takes credit for the season, the product fix and the pricing change. Overstates in a good month and understates in a bad one, and cannot tell you which. |
Synthetic control needs untreated queues that tracked the treated one before launch, and a company that launched the bot on every queue at once has no control and gets no number. That is the honest outcome and also a product that sometimes ships nothing. The method is thirty years old and any analytics vendor could add it; the defensible part is refusing to report when the fit is bad, which a vendor grading its own agent will never do. If heads of support decide the vendor number is good enough to renew on, this is a memo nobody commissions.
Market
Priced against the agent contract it evaluates, which is typically five to twenty times larger
Every support organisation running an AI agent on a subset of queues is a candidate, and the renewal is the forcing event. Two thousand support organisations at the Team tier is $14M ARR; Scale is priced against the contract it decides.