Reference architecture 02 Hybrid retrieval Tunable handoff Full conversation audit

Support deflection that knows when to stop.

The hard part of a support agent is not answering questions. It is recognising the questions it should not answer, and handing those over before it has already done damage.

THE PROBLEM

Deflection rate is the wrong headline metric.

Every support-AI vendor quotes a deflection rate. It is close to meaningless on its own, because you can drive it arbitrarily high by having the agent answer everything confidently — including the things it has no basis for. The tickets stop arriving; the damage arrives later, as churn, as a refund, or as a compliance incident.

The pair of numbers that matters is deflection rate at a fixed, measured accuracy floor. An agent that resolves 40% of tickets and is right ~always is worth more than one that resolves 75% and is confidently wrong one time in eight — because the second one destroys trust in the channel, and trust does not come back.

That reframing changes the architecture. The interesting component is not the generator. It is the gate in front of the handoff.

THE DESIGN

TICKET PATH

Intake

email · chat · widget
Classify

intent + risk tier
Retrieve

docs + account state
Draft

answer + citations
Gate

evidence + risk check
Human

with full context
Send

logged + scored

Lower box = escalation path Marker = one ticket through the system

Risk tier is assigned at classification, before retrieval — a billing or security question is escalated on its category alone, however confident the model later becomes.

Two retrieval sources, not one. Product documentation answers "how does this work". Account state answers "why is my thing broken". Most deployments wire up only the first and then wonder why the agent deflects nothing useful — the majority of real tickets are account-specific.

Risk tiering happens first. Certain categories never get an autonomous answer regardless of confidence: billing disputes, account deletion, security reports, anything with legal exposure. This is a static policy, not a model judgement, because a model's confidence is exactly the thing you cannot trust in the tail.

The gate checks evidence, not certainty. A language model's self-reported confidence is poorly calibrated. The gate instead asks a mechanical question: does every claim in this draft trace to a retrieved passage? An answer containing an unsupported assertion is escalated even if the model is sure.

Handoffs carry the work. When the agent escalates, the human receives the retrieved context, the draft, and the reason for escalation. A handoff that dumps a raw ticket back on a person has saved nothing.

agent/gate.py PYTHON

RISK_ALWAYS_ESCALATE = {class="text-emerald-data">"billing_dispute", class="text-emerald-data">"account_deletion",
                                        class="text-emerald-data">"security_report", class="text-emerald-data">"legal"}

                def decide(ticket, draft, retrieved):
                    class="text-dim"># Policy beats confidence. Category alone can escalate.
                    if ticket.category in RISK_ALWAYS_ESCALATE:
                        return Escalate(reason=class="text-emerald-data">"risk_category")

                    class="text-dim"># Every claim must trace to something we retrieved.
                    unsupported = [c for c in draft.claims
                                   if not c.supported_by(retrieved)]
                    if unsupported:
                        return Escalate(reason=class="text-emerald-data">"unsupported_claim",
                                        detail=unsupported)

                    if draft.needs_account_data and not ticket.account_linked:
                        return Escalate(reason=class="text-emerald-data">"missing_account_context")

                    return Send(draft)

The gate is deliberately boring and inspectable. Every escalation reason is a named constant, so the escalation mix is analysable.

WHERE IT BREAKS

Documentation that is quietly out of date

Retrieval is faithful to whatever it retrieves. If the docs describe last year's pricing, the agent will cite last year's pricing accurately. Mitigation: freshness metadata on every source, and a hard exclusion for anything past a staleness threshold.

Multi-turn drift

A conversation that starts in-scope wanders out of it by turn five, and the original risk classification no longer applies. Mitigation: re-classify on every turn, not just the first.

Users who learn to jailbreak the gate

Once customers realise a phrasing gets a human, they use it — which is rational, and which collapses your deflection rate. Mitigation: monitor escalation-reason distribution over time; a spike in one reason is a signal about user behaviour, not model quality.

Deflection measured at the wrong boundary

A ticket the agent "resolved" that reopens two days later was not deflected. Mitigation: measure deflection at a 7-day reopen window, which is usually 10–20 points below the number a vendor quotes on day one.

WHAT TO MEASURE

THE EVALUATION THAT MATTERS

Target by task
Task	n	Baseline	Target
Answer accuracy on deflected tickets	weekly sample, 200+	human agent rate	≥ human
Deflection rate at 7-day reopen window	all tickets	0%	measured, not promised
Escalations with usable context attached	all escalations	n/a	100%
Unsupported claims reaching a customer	all sent	n/a	trend to zero

Method: Deflected tickets are sampled weekly and graded by a support lead blind to whether the answer came from the agent or a human. Accuracy is scored against the same rubric used for human QA, so the comparison is like-for-like. Reopen window is 7 days — a resolution that bounces is counted as a failure, not a deflection. These are the targets we design against and instrument for; they are not results from a specific deployment.

Keep reading

Enterprise RAG Systems

→

Technology

Evaluating AI Systems

→

30-minute technical call · no deck