A voice system is a text system with a stopwatch attached. Everything that is merely slow in a chat interface becomes an audible pause on a phone call, and the pause is what people judge the system by.
OpenAI’s 10 September changelog lists GPT-Live 1 as generally available: full-duplex voice with reasoning and tools delegated to a backend model or agent. The listed voice-session rate is $0.05 per minute, billed per second; backend model and tool costs are additional. Official release and pricing note.
A useful pilot is a support call that continues naturally while an approved backend checks an order. Test interruptions, corrections and a delayed or failed lookup. The voice layer must distinguish an acknowledgement from a confirmed action: it should never tell a customer that a refund succeeded before the business system confirms it.
This separation gives teams another architecture to evaluate alongside a speech-to-text pipeline. It does not remove telephony latency or the need for human transfer. See GPT-Live documentation and our September technology briefing.
In human conversation the gap between one person finishing and the next beginning is short — a few hundred milliseconds in most studied languages, and long enough to be noticed when it stretches. A voice agent has to land inside roughly that window, or the caller starts talking over it, repeating themselves, or assuming the line has dropped.
That single constraint decides almost every design question on this page. The perceived pause is not the model's response time; it is the sum of every stage in the loop, plus the network under all of them. A transcription stage that is 200 ms slower than it needs to be cannot be recovered by a faster model, and a reasoning step that makes two sequential tool calls has spent the whole budget before it has said a word.
So the first thing to write down on a voice project is not the prompt. It is the budget: how many milliseconds each stage may take, measured end to end from the caller stopping speaking to the first audible syllable coming back. Everything else is negotiated against it.
mic or SIP trunk
has the caller stopped?
streaming, partial results
model + tools
cancel and re-listen
streamed by sentence
jitter buffer
The figures below are a budget, not a measurement. They are the allocation we would start a scoping call from for a telephone agent answering account questions, and the point of writing them down is that each line can be argued with, moved, or bought back somewhere else. Your numbers will differ by provider, region and workload.
| Criterion | Illustrative allocation | What moves it |
|---|---|---|
| Network in (caller to your edge) | 80–150 ms | Region choice, carrier route, WebRTC versus SIP |
| Endpointing decision | 200–500 ms | How long a silence has to last before you believe the turn ended |
| Final transcription after endpoint | 50–150 ms | Streaming with partials versus batch on the whole utterance |
| Reasoning to first token | 300–600 ms | Model size, prompt length, cache hits, whether a tool call is needed |
| Each sequential tool call | +150–800 ms | Your own API latency, and whether calls can run in parallel |
| Speech synthesis to first audio | 100–300 ms | Streaming voice versus waiting for the full sentence |
| Network out and jitter buffer | 80–200 ms | Buffer depth traded against robustness to packet loss |
Two things follow from the arithmetic. The first is that everything streams. Transcription emits partial hypotheses while the caller is still speaking, so the reasoning stage can begin drafting; synthesis starts on the first complete clause rather than the finished answer; audio plays as it arrives. A batch stage anywhere in the chain adds its full duration to the pause, and there is no budget for one.
The second is that a slow tool is a slow agent. If answering requires a lookup in a system that takes 900 ms, no amount of model tuning saves the turn. The fixes are architectural: parallelise independent calls, pre-fetch what the caller's identity already implies, cache what is stable, and — where the wait is genuinely unavoidable — say something. A short, honest acknowledgement while the lookup runs is not a trick; it is what a person does, and it converts dead air into a turn.
A cascaded pipeline chains three separate systems: speech recognition, then a text model, then speech synthesis. A speech-to-speech model takes audio in and emits audio out, with no text bottleneck between them. Both are in production use, and the choice is rarely about which is more advanced.
| Criterion | Cascaded pipeline | Speech-to-speech |
|---|---|---|
| Stages you can swap independently | Three | One |
| Latency floor | Sum of three stages | Lower — no text round trip |
| Prosody, tone and emotion preserved | Discarded at the transcript | Carried through |
| Inspectable transcript of what was understood | Yes, at every turn | Not inherently — must be produced alongside |
| Domain vocabulary and biasing | Word lists and custom lexicons | Prompting, with less direct control |
| Tool calling and structured output | Mature | Improving, more variable by provider |
| Self-hosting options | Open-weight models at every stage | Fewer, and heavier |
| Debugging a wrong answer | Attributable to a stage | Harder to localise |
Our default for anything with a compliance or quality-assurance requirement is the cascaded pipeline, for one unglamorous reason: it produces a transcript of what the system believed it heard, at every turn, which is the artefact an investigation asks for. Speech-to-speech earns its place where the interaction is short, conversational and low-stakes, and where tone genuinely matters — reception, triage, qualification. A hybrid is common and sensible: speech-to-speech for the conversational surface, with a parallel transcription stream kept purely for logging and evaluation.
Word error rate measures transcription. It does not measure whether the caller got what they rang for, and a system can hold an excellent word error rate while failing the job — mishearing the one token that mattered, interrupting badly, or answering the question it heard rather than the one that was asked. The metric that decides whether a voice deployment survives contact with users is task completion, and everything else is diagnostic.
| Task | n | Baseline | Target |
|---|---|---|---|
| Task completed without transfer | 300 recorded calls | existing IVR containment | ≥ 0.75 |
| Entity accuracy on reference numbers and postcodes | 200 utterances | unmeasured | ≥ 0.98 |
| Playback stops within 300 ms of a barge-in | 150 interruptions | unmeasured | ≥ 0.95 |
| Turn latency inside the agreed budget | 2,000 turns | unmeasured | ≥ 0.95 at p95 |
| Escalates rather than guessing when confidence is low | 80 adversarial | unmeasured | ≥ 0.95 |
| Completion parity across accent groups in the set | 300, grouped | unmeasured | no group below 0.90 of the mean |
Method: An illustrative gate of the shape we build, not results from a specific deployment. Sets are built from recorded calls with consent, replayed through the telephony path at call-quality audio rather than from clean source files, so codec and packet loss are inside the measurement. Latency is measured from the caller's final speech energy to the first returned audio frame, reported at p50 and p95 rather than as a mean, because a mean hides exactly the turns people complain about. Completion is graded by a human against a written rubric; transcription accuracy is scored automatically against verified references. Thresholds are set per engagement against the existing process, never against 100%.
Article 50 of the EU AI Act sets transparency duties for systems that interact directly with people: a person must be informed that they are dealing with an AI system unless that is obvious to a reasonably observant person in the circumstances. On a phone line it is not obvious — synthetic speech is convincing enough that assuming otherwise is not defensible — so the disclosure belongs in the opening turn, in plain language, before anything is collected.
That is cheap to implement and worth doing beyond the compliance argument: callers who know they are talking to a machine phrase things more clearly and ask to be transferred earlier, which improves both completion and satisfaction. The related duties in the same article concern marking synthetic audio content, which matters if the system generates recordings rather than only holding live conversations.
Separately from the AI Act, a voice system usually processes personal data and frequently records it. Recording consent, lawful basis, retention periods and the rights of the caller are ordinary data protection questions with ordinary answers, and they are easier to answer when the recording, the transcript and the model's context are all inside infrastructure you control. Whether an emotion-recognition component is permissible at all is a separate question with a stricter answer in some contexts, and a good reason not to add one because a provider offers it.
Sub-second from the caller finishing to the first audible syllable is the target that makes a conversation feel like a conversation, and roughly 1.5 seconds is where most people begin to notice a lag. Both are design targets rather than universal thresholds: a caller reading out a long reference number tolerates more than a caller answering yes or no. Agree the budget per turn type at the start, instrument it at p95, and treat a regression in it the same way you would treat a drop in accuracy.
The tools and the retrieval layer usually transfer; the prompt and the interaction design rarely do. Text answers are structured for the eye — lists, headings, links — and all of that is unusable spoken aloud. Voice answers need to be shorter, front-load the answer, offer one thing at a time, and be recoverable when the caller was not listening. Expect to keep the backend and rewrite the conversational layer.
It is a real option: capable open-weight recognition and synthesis models exist, and for high call volumes or audio that may not leave your perimeter, self-hosting is often the right answer. The trade is operational — you own the GPUs, the scaling and the availability of something that is on a phone call when it fails. A common middle position is self-hosted recognition, where the audio is most sensitive, with a hosted model for reasoning against redacted text.
Immediately, on the first request, without negotiation. The transfer should carry context — who the caller is, what they asked, what the agent already did — so they do not start again from the beginning. Systems that make the escalation path hard to find measure well on containment and badly on everything a business actually cares about, and callers learn the trick within one call anyway.
It depends entirely on the provider and the tier you are on, and it is the question to settle before any audio is sent. At the time of writing, enterprise arrangements from the major providers generally offer zero-retention or short-retention processing and do not train on inputs, but those are their terms and they change — confirm against the provider's current terms and get it in the contract rather than relying on a documentation page. Where the answer has to be certain, the deployment moves inside your own infrastructure.
Bring a recording of ten real calls. The latency budget and the escalation path tell you most of what a voice project will cost.
30-minute technical call · no deck
What we built, what stage it is at, and a link to the live thing wherever there is one.
yourOS is an operating system with an AI at its heart. You talk to it like a person, out loud or typed. It makes your calls, plans your days and works on your projects with a crew of agents, on your own private server. Our own product, built from the first commit; the waitlist is open.
Website, backend and app assets for Sonora, a free AI sound-therapy app on iOS and Android published by Cognitive AI.
Brand, website, prototype, apps on five platforms, the AI backend and the algorithm-intelligence pipeline for Podswap, a social growth engine that runs without a human in the loop.
In closed beta: Roxy, a private AI assistant with permanent memory running on infrastructure the user owns. Brand, site, product, backend and conversational design.
Live at coherence.com: brand identity generated with AI agents we directed, the motion system and website for the group behind Sonora, AboutAI, WakeUp and MudShadows. Agent work remains in build.
Live at domainbooth.com for DomainBooth, a premium domain brokerage: a new animated mark, every page redesigned and rebuilt with designed mobile layouts, and a custom CMS for its sales record, featured names, press and enquiries.
Apps, platforms and campaigns for these teams, long before the AI work. Every one is written up on the work page — what we did, and where our part ended.
