Vendor says 76%, reality says 41%: the support bot deflection rates we actually measured

Every support automation vendor now leads with a resolution rate. The numbers on the pricing pages and benchmark write-ups cluster between 60% and 80%, and the highest figure we collected while researching this study was 76%. Procurement decisions, renewals, and headcount plans are being made on those numbers, and on the in-product dashboards that echo them. Almost nobody audits either against the one dataset that cannot flatter itself: the ticket queue.

So we did. Over the last 14 months we measured 14 production support bots across client engagements: five Intercom Fin deployments, four Zendesk AI agents, and five custom RAG bots we built and operate. Every bot had been live for at least 90 days. Every bot was measured over the same window, against Zendesk and Intercom ticket exports rather than the bot's own dashboard. The combined sample covers just over 180,000 bot conversations.

The external numbers frame why this matters. Vendor benchmark material such as the ROI write-ups published at fin.ai anchors buyer expectations at the top of that 60 to 80% band. Industry cost roundups like The Stacc's analysis put a bot-handled routine query at $0.50 to $0.70 against $20 to $25 for a human agent. And Gartner projects that agentic AI will autonomously resolve 80% of common service issues by 2029. The direction of travel is real. The current claims are where we wanted numbers of our own.

From the raw data we computed three metrics used throughout this report: Measured Deflection Rate (MDR), Cost Per Resolution (CPR) and Escalation Quality Score (EQS). This is the same honesty exercise we ran on AI-generated codebases: not an argument against the category, an argument against running procurement on unaudited numbers. The median bot in this sample pays for itself comfortably. It just does about half of what its dashboard says it does.

Methodology and data sources

The raw fields we captured per bot:

• Platform, Intercom Fin / Zendesk AI agent / custom RAG build.
• Tenure and volume, months in production and monthly bot conversation volume from conversation logs.
• Corpus profile, article count, median article age, and whether a ticket-to-KB editorial loop exists.
• Dashboard-reported resolution, the number the bot claims for itself, taken from the vendor or in-house dashboard, unadjusted.
• Measured deflection, computed from ticket exports, help-centre GA4 events, and conversation logs, as defined below.
• Escalation traces, what crossed to a human, with how much context, and what happened after.
• Cost data, platform invoices, model and infrastructure spend, and our own billed build and maintenance hours.

A conversation counts as deflected only if the bot delivered an answer and the same user made no human contact on the same issue within 7 days, matched across channels by account id first and by fuzzy subject match second. A conversation abandoned mid-flow is not a deflection, whatever the dashboard says. The rubric was applied by a single reviewer (the same engineer across all 14 bots) to keep the coding consistent. Project owners gave consent for anonymised inclusion. The dataset preserves platform attribution but no client-level identifiers.

Measurement windows were aligned per bot: 90 consecutive days ending inside the same quarter, chosen to avoid launch spikes and seasonal troughs where the owner flagged them. GA4 help-centre events were used for one specific job: catching users who abandoned the widget and went hunting through self-serve surfaces before eventually emailing, which is channel-switching behaviour neither the bot dashboard nor a naive ticket count will attribute correctly. Post-handoff CSAT comes from the existing survey tooling in each client's Zendesk or Intercom instance, filtered to conversations that started with the bot, so the satisfaction figures describe the escalation experience specifically rather than support in general.

One scope note: we did not attempt to re-measure any vendor's marketing claim directly. Each bot is compared against its own dashboard, which is the number its owner was actually relying on.

The three metrics: MDR, CPR and EQS

1. Measured Deflection Rate (MDR)

MDR = Conversations with no human contact on the same issue within 7 days / Total bot conversations

The headline metric, and deliberately stricter than any vendor definition we have seen. It demands an answer delivered and 7 days of silence across every support channel, not just the widget. It is the number that actually maps to tickets your team did not handle.

2. Cost Per Resolution (CPR)

CPR = Monthly all-in bot cost / Monthly measured resolutions

All-in means platform fees, model and infrastructure spend, amortised build cost, and the maintenance hours someone bills for corpus and eval upkeep. The denominator is measured resolutions, not claimed ones. Both adjustments move the number, and both move it in the same direction: up.

3. Escalation Quality Score (EQS)

EQS = 100 x (0.4 x Context transfer rate + 0.3 x First-touch resolution after handoff + 0.3 x (1 - Repeat contact rate))

A 0 to 100 composite of the three things that make a handoff good: did the human agent receive the full context, did they resolve on first touch, and did the customer come back anyway. As finding 4 shows, this turned out to predict deflection itself, which surprised us.

Finding 1: The dashboard says 64.5%, the tickets say 41.2%

There are three layers to the headline number, and buyers routinely conflate them. The marketing layer advertises up to 76%. The dashboard layer, the number in the product your CFO screenshots for the board deck, reported a median of 64.5% across our sample. The measured layer, computed from ticket exports, came in at a median MDR of 41.2%. Marketing to dashboard is a benchmarks-versus-your-workload gap. Dashboard to measured is a definitions gap, and it is the one you can actually fix.

Take the median dashboard figure against the median measured figure: 23.3 points of daylight. Across the sample, that daylight decomposes into three mechanisms:

1. Silence counted as success (11.4 points). The user stops replying, the conversation times out, the dashboard books a resolution. Our conversation-log review found the largest share of these were mid-flow abandonments: the user gave up, often to open a ticket elsewhere or to churn quietly.
2. Repeat contact within 7 days (7.6 points). The bot answered, the user said thanks, and then filed a ticket on the same issue two days later. The dashboard has no reason to connect the two events. A ticket export matched on account id does.
3. Channel switching (4.3 points). The user abandoned the widget and emailed support directly, or replied to an old thread. Invisible to the bot, visible in the queue.

Two per-bot observations worth pulling out. The four healthy custom bots (A to D) show gaps of 8 to 12 points, not because custom is magic but because we configure their dashboards to count resolution the strict way, so there is less inflation to strip. And bot N, the worst performer in the entire study at 22%, is also a custom build. Its problem was not the pipeline. We will come back to it in the corpus section, because it is the most instructive failure in the dataset.

None of this requires malice to explain. A dashboard that waited 7 days and reconciled against the ticket system before booking a resolution would be slower, more expensive to build, and would print a smaller number than every competitor's. The incentives all point one way, so the generous definition wins, the same way "works on my machine" wins until production traffic arrives. Our position is simply that the buyer should hold the strict number, because the buyer is the one converting it into headcount plans and renewal decisions.

Finding 2: Deflection is a property of the query class, not the bot

The spread is enormous: 68% on password and account access, 9% on account-specific data questions. The pattern is not about model quality. Bots deflect well where the answer is a stable, documented fact that applies to everyone (how do I reset my password, what does this line on the invoice mean). They deflect badly where the answer requires reading the customer's own data, exercising judgment, or taking an action with consequences. Refunds are the extreme case: a 12% MDR against a 41% dashboard claim, because users nodding politely at policy text and then emailing a human is precisely the behaviour dashboards misread as resolution.

The practical consequence: your achievable deflection is mostly decided before the vendor demo, by your ticket mix. A developer-tool company drowning in error triage will never see a password-reset company's numbers, whatever it buys. Before signing anything, classify a month of tickets into these seven classes, multiply by the MDR column, and you have a defensible forecast. Pooled across our sample the volume-weighted mean works out to 39.7%, sitting close to the 41.2% per-bot median; when a vendor forecast lands 25 points above that, the burden of proof is theirs.

To make it concrete: a developer-tool company whose queue runs 30% error triage, 25% how-to, 20% configuration, 10% account-specific questions, 10% billing and 5% password resets multiplies out to an expected MDR of about 34%. That is not a failed deployment. That is the arithmetic of a hard ticket mix, and it is knowable before a single vendor call. The same arithmetic also tells you where to invest: moving error triage from 18% to 30% (better logs in the corpus, structured error-code articles, retrieval tuned for exact codes) is worth more to that company than any general model upgrade, because that class is where its volume lives.

Finding 3: The per-resolution cost maths, done on measured resolutions

Per-resolution pricing sounds like perfect alignment: you pay only when the bot succeeds. The catch is who defines success. If the vendor bills $0.99 on its own resolution definition, and that definition runs at the median inflation we measured (64.5 claimed against 41.2 measured), your effective price is $0.99 multiplied by the inflation ratio: about $1.55 per resolution that genuinely kept a ticket out of the queue. Not a scandal, still excellent against $20 to $25 for a human. But it is 57% more than the list price, and it compounds across every forecast built on it.

The custom side has its own honest accounting to do, and most build-versus-buy posts skip it. A production-grade RAG bot is not a weekend project: chunking, hybrid retrieval, reranking, grounding with citations, an eval suite, and the escalation integration land around $32,000 of build for the deployments in our sample. Amortise over 24 months, add $500 a month of infrastructure and model spend at the configuration that actually works (more on that curve below), add $400 a month of corpus and eval maintenance, and the all-in monthly cost is $2,233. Divide by measured resolutions and the curve is brutal at low volume: $3.72 at 600 a month. It crosses the vendor's effective $1.55 at roughly 1,450 measured resolutions a month, and keeps falling: $0.93 at 2,400, $0.56 at 4,000.

A caution on optimising the wrong direction: the cheapest resolution is not always the right one. Refunds and cancellations sit at a 12% MDR partly because they should. A customer heading for the exit is a retention conversation, and handing that to a policy quoting bot to save $20 is how you convert a save opportunity into a chargeback. The per-resolution maths in Chart 3 applies to the query classes a bot handles well; the classes it handles badly belong in the escalation budget, priced as relationship work rather than as tickets.

So the decision is a volume threshold, not a philosophy. Below about 1,000 measured resolutions a month, rent. Above about 1,500 with a curated corpus, building wins on cost and keeps your support data out of a third party. The same discipline applies to the model bill itself; our per-token economics study covers why AI features that look cheap per call become line items per MAU, and support bots are no exception.

Finding 4: Escalation quality predicts deflection

We built EQS to measure handoffs, expecting it to be independent of deflection. It is not. The four bots with the best escalation design deflect at a median of 52%; the four worst at 29.5%. The mechanism became obvious in the conversation logs. When users trust that a human is one click away, they give the bot a real chance, ask full questions, and accept its answers. When the bot behaves like a wall in front of the support team, users learn to type "agent" as their first message, and measured deflection craters. Bad escalation also unwinds deflections retroactively: a third of the bottom quartile's handoffs came back within a week, each one converting an earlier "resolution" into two contacts instead of one.

The external data agrees. The statistics roundups report 92% customer satisfaction when the human handoff is seamless, a figure most support bots never see because the handoff is where the product investment stopped. Our top quartile's 4.4 out of 5 post-handoff CSAT is that effect, measured on our own sample.

Designing the escalation path

Every top-quartile bot in the sample shares the same escalation architecture, and none of it is exotic. The rules we now treat as defaults:

• Escalation is a feature, not a failure state. It gets designed, built and evaluated in the first sprint, not bolted on when the complaints arrive.
• Thresholds are tuned per query class. Retrieval confidence that justifies answering a how-to question does not justify answering a refund request. On refunds and cancellations, the best bots barely try: they collect context and route.
• The exit is always visible.Counterintuitively, hiding the escape hatch lowers deflection. Two bots in our sample removed the "talk to a human" button to protect their numbers; both sat in the bottom EQS quartile with below-median MDR.
• Context travels with the customer. Full transcript, what the bot retrieved, what it already tried. The customer should never repeat themselves, and the agent should never repeat the bot.
• Escalated conversations jump the queue. The customer has already spent minutes with the bot. Making them wait again from the back of the line is how a 3.1 CSAT happens.
• Post-handoff outcomes are measured. First-touch resolution and 7-day repeat contact feed back into EQS, per query class, so threshold tuning is evidence rather than vibes.

One rule deserves its own paragraph: channel continuity. The handoff must continue in the same thread the customer is already in. The moment the bot says "we will email you back", the conversation resets: the customer restates the problem from scratch, the context evaporates, and a healthy share of them simply open a fresh ticket instead, which is exactly the channel-switching leak that finding 1 quantified at 4.3 points of the gap. Every top-quartile bot in our sample keeps the human reply inside the widget conversation, with email as a notification layer rather than a replacement channel.

Context transfer is the piece most deployments skip because it is integration work rather than prompt work. This is the payload our custom builds ship with every escalation:

type EscalationPayload = {
  conversationId: string;
  transcript: Turn[];          // full bot conversation, never a summary
  retrievedChunks: ChunkRef[]; // what the bot read before answering
  botConfidence: number;       // final-turn retrieval confidence, 0 to 1
  queryClass: QueryClass;      // password | billing | howto | config | bug | refund | account
  attemptedAnswers: string[];  // so the agent never repeats the bot
  userContext: {
    plan: string;
    accountAgeDays: number;
    openTicketCount: number;
  };
};

// routing rule: thresholds are per query class, not global.
// refunds escalate at any confidence; how-to holds until confidence < 0.62

On the vendor platforms you do not control the payload shape, but Fin and the Zendesk agents both expose enough hooks to pass transcript and attempted answers into the ticket. The two Fin deployments in our sample that wired this up properly sit in the top half of the EQS table; the ones that left the default behaviour do not.

The bot is only as good as the corpus

Bot N deserves its own section. Technically it was one of the better builds in the sample: hybrid retrieval, reranking, grounded answers with citations. It measured 22%, worst in the study, while its dashboard claimed 58%. The cause was not in the pipeline. It was retrieving, confidently and with citations, from a knowledge base that had not been meaningfully updated in two years. The product had shipped two major versions since. The bot was fluently, verifiably wrong, which is worse than being unavailable, and users learned that faster than the dashboard did.

The pattern generalises: in our sample, corpus age and coverage predicted MDR better than any retrieval configuration choice. Which is why the most valuable support-automation work we have done was not a bot at all. For a $200M-ARR cybersecurity company we shipped six platforms around the same knowledge estate: a knowledge base, a customer forum, a partner portal, release notes, a support console, and public docs. The unglamorous one, the internal support console, exists to close the ticket-to-KB loop: every closed support ticket is a KB-article candidate, and the hook between the ticket system (Zendesk, Intercom or Help Scout) and the editorial workflow is where most knowledge bases quietly stagnate. That loop is what keeps a corpus worth retrieving from, and it is the core of our knowledge base and community platform practice.

Scale has a cliff too: default WordPress search dies above 200 articles, and tag-based discovery breaks at the same point. If humans cannot find the right article in your KB, do not expect embeddings over the same neglected corpus to conjure relevance. Retrieval inherits your information architecture: version-aware routing (the article that applies to v3 but not v4), role-gated content, and an editorial owner are prerequisites, not nice-to-haves.

Once the corpus is healthy, retrieval spend follows a curve we have measured across engagements, and it bends hard in one place:

The jump from 48 to 72 costs $300 a month. The jump from 81 to 84 costs $900. Hybrid retrieval plus reranking at around $500 a month is where the quality per dollar peaks, and it is the configuration behind the $2,233 TCO in finding 3. The stage-by-stage build, chunking through embeddings, retrieval, reranking and caching, is documented in our production RAG pipeline guide. But spend the first budget on the corpus. A $1,800 retrieval stack over bot N's two-year-stale KB still loses to a $500 stack over a maintained one.

What this means for build vs rent

The honest summary of the data: bots work, dashboards inflate, and the decision points are knowable in advance.

• Under about 1,000 measured resolutions a month: rent. Fin or the Zendesk agents will be cheaper than any custom TCO. But instrument MDR from day one, because the renewal conversation should happen on your number, not theirs.
• Between 1,000 and 1,500: negotiate. You are inside the break-even band. Per-resolution pricing renegotiated against a measured definition is worth more than any feature on the roadmap.
• Above about 1,500 with a curated corpus: build. Custom RAG wins on cost, keeps conversation data in your stack, and lets you tune thresholds per query class, which is where the deflection actually lives. This is the work of our AI chatbot and RAG development practice: chunking, hybrid retrieval, reranking, grounding, eval, and the escalation integration, costed before the architecture is locked.
• Whichever you choose, fix the definition in writing. A deflection claim without a measurement window and a channel-matching rule is a marketing number.

And if the support bot is becoming part of the product itself, an in-product copilot rather than a widget in front of the queue, the unit economics and architecture questions change shape again; that is the territory of our AI SaaS product development engagements, where the bot's cost per user has to survive contact with a pricing page.

Limitations and how to read this study critically

Four caveats that should temper any reading of these numbers.

First, sample bias runs in our favour on the gap. Clients tend to bring us bots they already suspect are underperforming; happily-renewing vendor deployments are under-represented. True vendor medians across all deployments are plausibly higher than our 41.2%, though the definitional inflation mechanisms apply to every deployment regardless.

Second, n is 14, and the cohort splits are small: five Fin, four Zendesk, five custom. Read the medians as magnitudes, not precise estimates. The custom cohort's stronger showing partly reflects that those corpora were curated as part of the same engagements, which is an argument about where the work is, not about whose model is better.

Third, the 7-day repeat-contact window is a judgment call. We re-ran a subset at 14 days and median MDR dropped roughly 3 points, so a stricter window makes the headline worse, not better. Cross-channel matching by fuzzy subject line will also miss some repeat contacts, which inflates every MDR figure slightly, vendors' and ours alike.

Fourth, we built five of the bots in this sample, and we sell the service this post links to. Mitigations: one reviewer applied the same rubric to all 14 bots, every number comes from ticket exports rather than our dashboards, and the worst bot in the study is one of ours. The vendors are also moving targets; Fin and the Zendesk agents both shipped meaningful updates during the measurement window, and the Gartner trajectory suggests the 2029 picture will look very different. Read this as a snapshot, July 2026.

The sample at a glance

What to measure before you renew any bot contract

You do not need our rubric to run this audit on your own bot; you need an afternoon and two exports. Pull 90 days of bot conversations and 90 days of tickets. Match on account id. Count a deflection only when an answer was delivered and no human contact followed within 7 days. Split by query class. Divide your true monthly bot cost by that number. Then put the result next to the dashboard and next to whatever number is in the renewal deck.

If the gap looks like the ones in Chart 1 and you want a second pair of eyes on where the points are leaking, corpus, thresholds, or escalation, send us the exports. We will run the same measurement and send back the actual numbers.

More measured-numbers research from the same shelf:

The engagements that map to the failure modes in this study: