Almost every voice-AI team has sat through a demo that felt a beat too slow. The model benchmarks looked fine and every component tested fast, yet the caller hesitated.
A realtime voice agent is a conversational, stateful, tool-using LLM agent with live voice I/O, part of the multimodal interface revolution and far from a deterministic IVR or chatbot. Slowness is the adoption blocker, and most teams optimise model speed when the number that matters is end-to-end time-to-first-audio (TTFA). Around 200 milliseconds feels natural, 500 to 800 is noticeable, and past one second is disruptive.
By the end, you can explain where latency accrues, choose between cascaded and native architectures, and evaluate STT, TTS and realtime vendors against a P50/P95 TTFA budget. The article walks the cascaded pipeline, streaming, the sentence buffer, the native-versus-cascaded trade-off, and a latency, quality and cost vendor lens.
What is an AI voice agent, and how does it differ from a traditional IVR or chatbot?
A traditional IVR is deterministic, a menu tree routing calls on touch-tones or keywords. A chatbot is text-only and asynchronous, with no audio pipeline or turn-taking. A voice agent differs on all three: the voice is only one layer, and the agent also reasons, calls functions, plans, applies guardrails, keeps state and invokes tools mid-turn.
So a voice agent replaces rigid touch-tone menus with natural-language conversations and has to respond like a human, where voice callers have almost no margin before the experience degrades. The category is attracting funding now, and a voice agent demands streaming, endpointing and sub-second TTFA the other two do not.
How do realtime voice agents work under the hood?
Production voice agents run a cascaded STT-LLM-TTS pipeline: capture, voice activity detection and turn-taking, streaming speech-to-text, sentence aggregation, a streaming LLM, and text-to-speech. Audio travels both ways over WebSocket or WebRTC, and a lightweight VAD such as Silero drives a turn-taking state machine, with an interrupt path so a caller can cut the agent off mid-sentence.
No stage waits for the previous one. The STT emits partial transcripts, the LLM streams tokens, and TTS returns audio chunks as they are synthesised. Between the LLM and TTS, a sentence buffer collects tokens until a full stop or question mark, then flushes to TTS while the LLM keeps generating. Because stages overlap, the metric that matters is time-to-first-audio: the gap from the caller finishing to the first audio sample they hear.
Why does “realtime” hinge on streaming and pipelining rather than any single fast model?
Realtime comes from overlapping stages and partial outputs working at once. Run STT, LLM and TTS in sequence and the delays add up: 500 milliseconds of transcription, 700 of reasoning, 300 of synthesis. Stream every stage and let them overlap, and the effective latency approaches the slowest stage, rather than the sum of all of them.
Speculative execution helps: the LLM starts on a stable STT partial before the final transcription lands, and streams tokens. A serving stack such as vLLM streams tokens one by one, so the buffer flushes sentence one to TTS while sentence two is generated. Every layer must stream; batch transcription alone adds 600 to 1,200 milliseconds before the LLM call. The buffer has a price: more context improves accuracy but delays first audio, so teams tune it against TTFA and word error rate.
What is time-to-first-audio (TTFA), and why does it matter more than model inference latency?
TTFA is the time from when the user stops speaking to the first audio sample the agent returns. It is the delay callers experience directly, and it is always larger than any single model’s inference latency because it adds endpointing and VAD, STT finalisation, the LLM’s time-to-first-token (TTFT) and TTS’s time-to-first-byte (TTFB). A 200-millisecond model can still produce a one-second-plus TTFA.
Endpointing is the largest controllable item and the single biggest stage to tune. An acoustic silence timer waits, say, 700 milliseconds of quiet before ending the turn. A 400-millisecond pause mid-sentence, like “a flight to… uh… Lisbon”, is acoustically identical to a pause at the end of a sentence. Semantic VAD reads the meaning of the partial transcript to decide whether the speaker has finished, making turn-taking more natural. Measure at P50 and P95, not the mean, because a few slow calls are what callers remember. TTFA is the number; the architecture you pick is the next lever on it.
Native speech-to-speech versus cascaded STT-LLM-TTS: which architecture wins, and why aren’t native models viable yet?
The cascade is the production default: stages are swappable and observable, a transcript exists for compliance, and function calling is mature. Native speech-to-speech models take audio in and return audio, preserving tone and prosody the ASR stage throws away. They win the latency and emotional fidelity ceiling, but are not yet production-viable. The open models show the gap: Qwen2.5-Omni benchmarks around 13 seconds of sentence-level TTFA, lacks function calling, and has no incremental streaming audio output. The same paper’s cascaded reference build hit roughly 755 milliseconds with full function calling, a 17-fold improvement.
OpenAI’s Realtime API is a managed near-native middle path with mature tool calling, but it costs more and gives you less per-stage control. What flips the choice is native models that stream audio, call tools reliably, and run cheaper on open weights. Until then, the cascade wins for production agents, and native S2S wins for narrow, low-tooling use cases like empathy-driven intake.
How do you evaluate STT/TTS and realtime voice vendors on latency, quality and cost?
Evaluate vendors on a P50/P95 TTFA budget, then weigh quality and cost. Latency means TTFA plus component diagnostics: STT partial and final latency, LLM TTFT and TTS TTFB, at P50 and P95. Quality means word error rate for STT and naturalness and prosody for TTS, tested on your team’s own domain audio. Cost means per-minute platform pricing against per-token LLM and per-character TTS.
Trust independent benchmarks over vendor pages. Independent measurements put Deepgram streaming STT at 337 to 509 milliseconds P50 and ElevenLabs TTS TTFB around 220 milliseconds. Third-party platform tests consistently land above vendor claims, so confirm the percentile, audio profile and cohort. On quality, clean-English WER has plateaued around 2 to 3 per cent, so streaming latency, end-of-turn detection and entity preservation differentiate vendors. Stability levers include warm connections, model routing, caching and regression testing. Self-hosting only helps when components sit close together and stream partials, inside the cloud-versus-self-hosted trade-off. For build-versus-buy, see evaluating STT/TTS and realtime voice vendors.
Conclusion
The vendor you choose is the latency budget your business accepts. A realtime voice agent is a streaming cascade whose effective latency approaches the slowest stage, and the only number the caller feels is end-to-end TTFA at P50 and P95. Model TTFT and TTS TTFB are diagnostic components of that TTFA. The cascade wins today; native speech-to-speech holds the latency and prosody ceiling but is not yet production-viable. Set a decomposed TTFA budget, gate it with WER and naturalness, and test against independent benchmarks. That is the broader voice AI picture, and the next layer up is function calling and the MCP/A2A protocols. The number to optimise is the end-to-end TTFA the caller feels.
Frequently Asked Questions
What is barge-in, and why does it matter for a realtime voice agent?
Barge-in is the caller’s ability to interrupt the agent while it is speaking, and the agent’s ability to detect that speech and stop its own audio. It matters because a voice agent that keeps talking over an interruption feels like a recorded IVR, not a human. Production systems use VAD, often Silero or semantic VAD, to monitor for speech during playback and cut the response immediately.
What is the difference between endpointing and barge-in?
Endpointing decides when the caller has finished speaking so the agent can begin its reply; barge-in decides when the caller starts speaking over the agent so it can stop. Endpointing happens at the end of a user turn, usually through an acoustic silence timer or semantic VAD. Barge-in happens during the agent’s turn and monitors for incoming speech while TTS audio is still playing. Confusing the two produces agents that either cut callers off or wait too long.
What is semantic VAD, and how does it differ from acoustic VAD?
Acoustic VAD detects voice by sound energy and silence timers, which means it can miss the difference between a pause and the end of a thought. Semantic VAD reads the meaning of the partial transcript to decide whether the speaker has actually finished. Semantic VAD produces more natural turn-taking but adds a little latency and compute, so teams often trade the two off against their TTFA budget.
How do sentence buffers trade off accuracy against latency?
A sentence buffer holds streaming STT and LLM output until a complete sentence is ready, then flushes it to TTS. A longer buffer gives the ASR more context and usually improves transcription accuracy, but it adds delay before the first audio. A shorter buffer feels faster but risks emitting a sentence before the model has settled. Teams tune the buffer against their TTFA and WER targets.
What is speculative execution on STT partials?
Speculative execution means the LLM starts generating from a stable STT partial before the final transcription arrives, rather than waiting for the full sentence. It hides ASR latency by overlapping the two stages. The risk is that a later STT correction invalidates the draft, so systems discard or revise output. When tuned well, it is one of the cheapest ways to pull TTFA down without changing models.
What does P95 mean, and why should I track it instead of the average?
P95 is the latency value that 95 per cent of calls beat, so it describes the slow end of the distribution rather than the typical case. Averages hide tail latency: a few one-second responses can drag the mean while most callers still experience a fast agent. Callers remember the slow calls, not the average one, so P50 and P95 together give a truer picture of what users actually feel.
Should I use WebRTC or WebSocket for my voice agent?
WebSocket is simpler and works well for server-to-server audio and telephony integrations, with low overhead once a connection is warm. WebRTC is built for browser and peer-to-peer audio and adds jitter buffering, echo cancellation and NAT traversal, which helps on poor networks but adds complexity. Most managed voice platforms handle the transport for you; choose WebRTC for browser-facing calls and WebSocket for backend pipelines.
Can I run a realtime voice agent fully self-hosted, and does that cut latency?
Yes, you can self-host the STT, LLM and TTS stack, typically with vLLM serving the model, and it can cut latency by removing public-cloud hops and keeping components co-located. The trade-off is operational: you own scaling, GPU capacity and failover. Self-hosting also improves data control and privacy. Latency only improves if the components sit close together and the serving stack streams partials properly.
Do I still need a transcript with a native speech-to-speech model?
Yes, for most regulated or audited use cases you still need a transcript. Native speech-to-speech models go audio in to audio out, so they do not inherently produce the text record that compliance, logging and analytics require. You can run a separate ASR pass to generate one, but that adds latency and cost and partly erodes the architecture’s simplicity. The cascade’s built-in transcript is one reason it remains the production default.
Are realtime voice agents more expensive to run than a traditional IVR?
They are usually more expensive per minute, because you pay for STT, LLM tokens and TTS characters on top of telephony, whereas an IVR only routes calls. The cost is often justified by higher containment and automation value, since the agent resolves more calls without a human. Teams control spend through shorter prompts, cheaper models on simple turns and traffic splitting across primary and fallback providers.
What TTFA budget should I aim for in production?
Aim to keep P50 time-to-first-audio under roughly 500 milliseconds and P95 under one second. Around 200 milliseconds feels natural, 500 to 800 is noticeable and past one second feels disruptive, so the exact budget depends on how conversational the use case needs to be. Decompose the budget per stage and test against it continuously rather than chasing a single headline number.
How do I benchmark my voice agent’s latency in staging before going live?
Run a fixed, recorded test set through the full pipeline and measure end-to-end TTFA at P50 and P95, not averages. Break the result into STT finalisation, LLM TTFT and TTS TTFB so you can see which stage owns the delay. Repeat the test continuously as models and vendors change, and compare against independent benchmarks rather than vendor-published figures.
