Technology Cascaded and speech-to-speech Telephony and web

Voice AI systems.
Under a second.

A voice system is a text system with a stopwatch attached. Everything that is merely slow in a chat interface becomes an audible pause on a phone call, and the pause is what people judge the system by.

Last reviewed 19 September 2026 · Engineering guidance, not legal advice.

SEPTEMBER 2026 UPDATE

Keep talking while the tools work.

OpenAI’s 10 September changelog lists GPT-Live 1 as generally available: full-duplex voice with reasoning and tools delegated to a backend model or agent. The listed voice-session rate is $0.05 per minute, billed per second; backend model and tool costs are additional. Official release and pricing note.

A useful pilot is a support call that continues naturally while an approved backend checks an order. Test interruptions, corrections and a delayed or failed lookup. The voice layer must distinguish an acknowledgement from a confirmed action: it should never tell a customer that a refund succeeded before the business system confirms it.

This separation gives teams another architecture to evaluate alongside a speech-to-text pipeline. It does not remove telephony latency or the need for human transfer. See GPT-Live documentation and our September technology briefing.

THE LATENCY BUDGET IS THE ARCHITECTURE

In human conversation the gap between one person finishing and the next beginning is short — a few hundred milliseconds in most studied languages, and long enough to be noticed when it stretches. A voice agent has to land inside roughly that window, or the caller starts talking over it, repeating themselves, or assuming the line has dropped.

That single constraint decides almost every design question on this page. The perceived pause is not the model's response time; it is the sum of every stage in the loop, plus the network under all of them. A transcription stage that is 200 ms slower than it needs to be cannot be recovered by a faster model, and a reasoning step that makes two sequential tool calls has spent the whole budget before it has said a word.

So the first thing to write down on a voice project is not the prompt. It is the budget: how many milliseconds each stage may take, measured end to end from the caller stopping speaking to the first audible syllable coming back. Everything else is negotiated against it.

THE CONVERSATIONAL LOOP

Capture

mic or SIP trunk
Endpoint

has the caller stopped?
Transcribe

streaming, partial results
Reason

model + tools
Barge-in

cancel and re-listen
Synthesise

streamed by sentence
Playback

jitter buffer

Lower box = the caller interrupts Marker = one conversational turn

Every stage streams into the next. The loop is only worth drawing because the failure people complain about — the pause — is not owned by any single box in it.

WHERE THE MILLISECONDS GO

The figures below are a budget, not a measurement. They are the allocation we would start a scoping call from for a telephone agent answering account questions, and the point of writing them down is that each line can be argued with, moved, or bought back somewhere else. Your numbers will differ by provider, region and workload.

	Illustrative allocation	What moves it
Network in (caller to your edge)	80–150 ms	Region choice, carrier route, WebRTC versus SIP
Endpointing decision	200–500 ms	How long a silence has to last before you believe the turn ended
Final transcription after endpoint	50–150 ms	Streaming with partials versus batch on the whole utterance
Reasoning to first token	300–600 ms	Model size, prompt length, cache hits, whether a tool call is needed
Each sequential tool call	+150–800 ms	Your own API latency, and whether calls can run in parallel
Speech synthesis to first audio	100–300 ms	Streaming voice versus waiting for the full sentence
Network out and jitter buffer	80–200 ms	Buffer depth traded against robustness to packet loss

An illustrative budget of the shape we design to, not measured client outcomes. Note what dominates: endpointing and the tool calls, both of which are yours to control, rather than the model, which is usually the part teams try to optimise first.

Two things follow from the arithmetic. The first is that everything streams. Transcription emits partial hypotheses while the caller is still speaking, so the reasoning stage can begin drafting; synthesis starts on the first complete clause rather than the finished answer; audio plays as it arrives. A batch stage anywhere in the chain adds its full duration to the pause, and there is no budget for one.

The second is that a slow tool is a slow agent. If answering requires a lookup in a system that takes 900 ms, no amount of model tuning saves the turn. The fixes are architectural: parallelise independent calls, pre-fetch what the caller's identity already implies, cache what is stable, and — where the wait is genuinely unavoidable — say something. A short, honest acknowledgement while the lookup runs is not a trick; it is what a person does, and it converts dead air into a turn.

TURN-TAKING, WHICH IS THE HARD PART

Endpointing

Deciding that the caller has finished, rather than paused. Fixed silence thresholds fail in both directions: short ones cut people off mid-sentence, long ones make the agent feel sluggish. Someone reading out a card number, thinking aloud, or speaking a language with different pause structure breaks a threshold tuned on fluent short answers. Semantic endpointing — using the partial transcript to judge whether the utterance is syntactically complete — helps, and it is another model in the loop with its own latency and its own errors.

Barge-in

The caller starts talking while the agent is speaking. The system must stop playback immediately, discard the rest of the generated response, and treat what it now hears as the live turn. That means cancellable synthesis, a cancellable model call, and acoustic echo cancellation good enough that the agent does not interrupt itself by hearing its own voice. Systems that cannot be interrupted are the single most common complaint about voice agents, and the fix is plumbing rather than intelligence.

Backchannels and false starts

“Mm-hm”, “right”, “sorry, go on”. These are not turns and should not trigger a full response cycle, but they look identical to speech at the audio layer. Treating every vocalisation as an interruption produces an agent that flinches. Classifying short overlapping utterances before acting on them is a small model doing a small job, and it noticeably changes how the conversation feels.

Latency masking

When a turn will be slow, the response can begin before the answer is known: acknowledge, restate the request, then continue into the substance once the lookup returns. Used sparingly this reads as natural. Used on every turn it reads as stalling, and callers notice quickly. It is a budget-of-last-resort, not a substitute for fixing the slow stage.

CASCADED OR SPEECH-TO-SPEECH

A cascaded pipeline chains three separate systems: speech recognition, then a text model, then speech synthesis. A speech-to-speech model takes audio in and emits audio out, with no text bottleneck between them. Both are in production use, and the choice is rarely about which is more advanced.

	Cascaded pipeline	Speech-to-speech
Stages you can swap independently	Three	One
Latency floor	Sum of three stages	Lower — no text round trip
Prosody, tone and emotion preserved	Discarded at the transcript	Carried through
Inspectable transcript of what was understood	Yes, at every turn	Not inherently — must be produced alongside
Domain vocabulary and biasing	Word lists and custom lexicons	Prompting, with less direct control
Tool calling and structured output	Mature	Improving, more variable by provider
Self-hosting options	Open-weight models at every stage	Fewer, and heavier
Debugging a wrong answer	Attributable to a stage	Harder to localise

Structural properties of the two shapes as of September 2026. Capabilities move quickly; the auditability row is the one that has held longest, and it is usually what decides regulated deployments.

Our default for anything with a compliance or quality-assurance requirement is the cascaded pipeline, for one unglamorous reason: it produces a transcript of what the system believed it heard, at every turn, which is the artefact an investigation asks for. Speech-to-speech earns its place where the interaction is short, conversational and low-stakes, and where tone genuinely matters — reception, triage, qualification. A hybrid is common and sensible: speech-to-speech for the conversational surface, with a parallel transcription stream kept purely for logging and evaluation.

THE TELEPHONE IS A CONSTRAINT, NOT A DETAIL

Bandwidth. A traditional telephone call is narrowband: 8 kHz sampling, roughly 300 Hz to 3.4 kHz of usable audio, compressed by a codec designed in the 1970s. Recognition accuracy measured on studio-quality audio does not survive the trip. If the system will answer phone calls, evaluate it on phone-quality audio from the first week, not on clean recordings from a laptop microphone.

Packet loss and jitter. Real calls drop packets. A jitter buffer smooths that out and costs latency to do it, which is a direct trade against the budget above. Deep buffers sound robust and feel slow; shallow ones feel responsive and stutter on a bad connection. There is no setting that is right for every route.

The rest of the telephony stack. DTMF tones for the cases where speech is the wrong input — card numbers, menu selections. Warm transfer that carries context to the human rather than making the caller start again. Call recording with its own consent and retention rules. Hold, conference and disconnection semantics. None of this is AI work, and it is usually where the integration time goes.

Accents, noise and vocabulary. Recognition quality is not uniform across speakers, and published research has repeatedly found accuracy gaps between accent and dialect groups. Treat that as a design input rather than an unfortunate footnote: build the evaluation set from your actual callers, including the accents and the background noise your service really encounters, and check performance per group rather than in aggregate. Domain vocabulary — product names, drug names, reference formats — is worth handling explicitly with biasing lists or a post-recognition correction step, because these are precisely the tokens a general model has never seen and the caller most needs to be got right.

EVALUATING A VOICE SYSTEM

Word error rate measures transcription. It does not measure whether the caller got what they rang for, and a system can hold an excellent word error rate while failing the job — mishearing the one token that mattered, interrupting badly, or answering the question it heard rather than the one that was asked. The metric that decides whether a voice deployment survives contact with users is task completion, and everything else is diagnostic.

Target by task
Task	n	Baseline	Target
Task completed without transfer	300 recorded calls	existing IVR containment	≥ 0.75
Entity accuracy on reference numbers and postcodes	200 utterances	unmeasured	≥ 0.98
Playback stops within 300 ms of a barge-in	150 interruptions	unmeasured	≥ 0.95
Turn latency inside the agreed budget	2,000 turns	unmeasured	≥ 0.95 at p95
Escalates rather than guessing when confidence is low	80 adversarial	unmeasured	≥ 0.95
Completion parity across accent groups in the set	300, grouped	unmeasured	no group below 0.90 of the mean

Method: An illustrative gate of the shape we build, not results from a specific deployment. Sets are built from recorded calls with consent, replayed through the telephony path at call-quality audio rather than from clean source files, so codec and packet loss are inside the measurement. Latency is measured from the caller's final speech energy to the first returned audio frame, reported at p50 and p95 rather than as a mean, because a mean hides exactly the turns people complain about. Completion is graded by a human against a written rubric; transcription accuracy is scored automatically against verified references. Thresholds are set per engagement against the existing process, never against 100%.

WHEN THE SYSTEM TALKS TO THE PUBLIC

Article 50 of the EU AI Act sets transparency duties for systems that interact directly with people: a person must be informed that they are dealing with an AI system unless that is obvious to a reasonably observant person in the circumstances. On a phone line it is not obvious — synthetic speech is convincing enough that assuming otherwise is not defensible — so the disclosure belongs in the opening turn, in plain language, before anything is collected.

That is cheap to implement and worth doing beyond the compliance argument: callers who know they are talking to a machine phrase things more clearly and ask to be transferred earlier, which improves both completion and satisfaction. The related duties in the same article concern marking synthetic audio content, which matters if the system generates recordings rather than only holding live conversations.

Separately from the AI Act, a voice system usually processes personal data and frequently records it. Recording consent, lawful basis, retention periods and the rights of the caller are ordinary data protection questions with ordinary answers, and they are easier to answer when the recording, the transcript and the model's context are all inside infrastructure you control. Whether an emotion-recognition component is permissible at all is a separate question with a stricter answer in some contexts, and a good reason not to add one because a provider offers it.

QUESTIONS

QUESTIONS — 5

Sub-second from the caller finishing to the first audible syllable is the target that makes a conversation feel like a conversation, and roughly 1.5 seconds is where most people begin to notice a lag. Both are design targets rather than universal thresholds: a caller reading out a long reference number tolerates more than a caller answering yes or no. Agree the budget per turn type at the start, instrument it at p95, and treat a regression in it the same way you would treat a drop in accuracy.

The tools and the retrieval layer usually transfer; the prompt and the interaction design rarely do. Text answers are structured for the eye — lists, headings, links — and all of that is unusable spoken aloud. Voice answers need to be shorter, front-load the answer, offer one thing at a time, and be recoverable when the caller was not listening. Expect to keep the backend and rewrite the conversational layer.

It is a real option: capable open-weight recognition and synthesis models exist, and for high call volumes or audio that may not leave your perimeter, self-hosting is often the right answer. The trade is operational — you own the GPUs, the scaling and the availability of something that is on a phone call when it fails. A common middle position is self-hosted recognition, where the audio is most sensitive, with a hosted model for reasoning against redacted text.

Immediately, on the first request, without negotiation. The transfer should carry context — who the caller is, what they asked, what the agent already did — so they do not start again from the beginning. Systems that make the escalation path hard to find measure well on containment and badly on everything a business actually cares about, and callers learn the trick within one call anyway.

It depends entirely on the provider and the tier you are on, and it is the question to settle before any audio is sent. At the time of writing, enterprise arrangements from the major providers generally offer zero-retention or short-retention processing and do not train on inputs, but those are their terms and they change — confirm against the provider's current terms and get it in the contract rather than relying on a documentation page. Where the answer has to be certain, the deployment moves inside your own infrastructure.

Keep reading

Evaluating AI Systems

→

30-minute technical call · no deck