Image generated with Google Gemini · BestStartBiz.com
What Is Real-Time Latency in Voice AI Pipelines?
In a voice conversation, half a second feels like forever. The moment a user finishes speaking, every millisecond before your AI responds is being judged — not consciously, but reflexively. If the pause stretches past 800 milliseconds, your user starts to assume the AI didn’t hear them. Past 1.5 seconds, they’ll repeat themselves. Past 3 seconds, they hang up.
Real-time latency is the single technical factor that determines whether a voice AI feels like a conversation or feels like filling out a form by voice. This post breaks down what latency actually means in a voice AI pipeline, how to measure it accurately, and the specific optimizations that take a sluggish prototype down to the sub-500ms response times that modern voice agents require.
The three stages of voice AI latency (STT, LLM, TTS), end-to-end benchmark targets, how to measure each component accurately, and the five highest-impact optimizations for reducing latency without sacrificing quality.
The Three Stages That Define Voice AI Latency
Every modern voice AI pipeline — whether built on OpenAI’s Realtime API, ElevenLabs Conversational AI, or a custom stack — moves user audio through three sequential stages. Each stage adds latency, and the total round-trip time is what the user actually experiences.
The STT stage converts the user’s speech into text. Latency here is dominated by two things: how quickly the model processes audio, and whether it can stream results as the user speaks (true streaming STT) or must wait for the user to finish before beginning transcription (batched STT).
Modern streaming models like Deepgram Nova-3, Whisper Large V3 Turbo, and AssemblyAI’s Universal-Streaming can emit transcriptions with as little as 150ms of lag behind live audio. Batched models typically add 400–800ms of additional latency on top of the time the user spent speaking.
The LLM processes the transcribed text and generates a response. Latency is measured as time to first token (TTFT) — how long before the model begins producing output. The full response takes longer, but TTFT is what matters for voice because downstream TTS can begin speaking as tokens arrive.
GPT-4o typically delivers first token in 300–500ms. Claude Sonnet 4.5 averages 200–400ms. Smaller/faster models like Groq’s Llama 3.3 70B on LPU hardware regularly hit 100–200ms TTFT. The trade-off is response quality — faster models often produce less nuanced answers.
The TTS stage converts the LLM’s text response back into speech. Streaming TTS can begin emitting audio within 100ms of receiving the first words of the LLM response. ElevenLabs Flash v2.5 clocks in at ~75ms, OpenAI’s tts-1 at ~200ms, and Cartesia Sonic averages ~90ms for first audio chunk delivery.
Non-streaming TTS waits for the full LLM response before generating any audio — adding 500–1500ms of additional latency depending on response length. For real-time voice, non-streaming TTS is a non-starter.
End-to-End Latency Targets
The three stages don’t run sequentially end-to-end — modern pipelines overlap them aggressively. STT emits partial transcripts that the LLM can begin processing, and TTS begins speaking while the LLM is still generating. This overlap is what makes sub-500ms end-to-end latency achievable.
| End-to-end latency | User experience | Realistic for |
|---|---|---|
| <300ms | Indistinguishable from human conversation | Specialized hardware (Groq LPU + Cartesia + Deepgram) |
| 300–500ms | Feels natural, no perceived lag | Modern well-optimized pipelines (ElevenLabs Conversational, OpenAI Realtime) |
| 500–800ms | Slight pause but conversational | Most production voice agents in 2026 |
| 800ms–1.5s | Noticeable lag; users begin to hesitate | Early prototypes, unoptimized pipelines |
| >1.5s | Users assume the AI didn’t hear them | Non-streaming architectures (unusable for real-time) |
How to Measure Latency Accurately
The number one mistake teams make when optimizing voice AI latency is measuring the wrong thing. “Response time” from a server log is not the same as what the user perceives. Here’s the correct measurement framework:
The 4 critical timestamps
t0 = user_speech_end_detected // VAD fires end-of-speech
t1 = final_stt_transcript_received // STT finalized text
t2 = llm_first_token_generated // LLM TTFT
t3 = tts_first_audio_byte_played // User hears first sound
// The 3 latencies that matter:
stt_latency = t1 - t0
llm_latency = t2 - t1
tts_latency = t3 - t2
total_latency = t3 - t0 // What the user actually experiencesThe total_latency metric is the only one that matters to users — but you need the component latencies to know where to optimize. Instrument your pipeline to log all four timestamps on every turn, then plot the distribution. Averages lie — look at p50, p95, and p99. A p50 of 400ms with a p99 of 3000ms means 1% of your conversations are catastrophically broken even though your “average” looks great.
Network round-trip time between your server and your STT/LLM/TTS providers adds 20–150ms per hop depending on geography. A pipeline that benchmarks at 400ms locally can balloon to 900ms when your user is in Singapore and your providers are in US-East. Always deploy your orchestration server in the same region as the majority of your API endpoints.
The 5 Highest-Impact Latency Optimizations
-
Switch to streaming at every stage.
Any component that waits for the previous stage to complete adds its full processing time to the critical path. Streaming STT emits partials; streaming LLM emits tokens; streaming TTS emits audio chunks. All three must stream for sub-500ms total latency to be possible. -
Use early LLM invocation.
Don’t wait for the final STT transcript before invoking the LLM. Modern pipelines invoke the LLM as soon as the user pauses for 200ms, using the current partial transcript. If the user resumes speaking, cancel the request and restart. This technique alone shaves 300–500ms off typical pipelines. -
Pre-warm your LLM context.
Large system prompts (think 2000+ tokens) can add 100–300ms to first-token latency. Cache the system prompt on the inference endpoint. OpenAI’s prompt caching and Anthropic’s prompt caching both deliver 50–80% TTFT reduction on cached prefixes — essentially free. -
Choose a faster model where quality permits.
GPT-4o is excellent but slower than Claude Haiku. Haiku is excellent but slower than Groq’s Llama 3.3 70B. Measure actual response quality on your task — not benchmark scores — and use the fastest model that meets your quality bar. In many conversational use cases, Haiku or Groq matches GPT-4o quality at a third of the latency. -
Co-locate infrastructure.
Run your STT, LLM, TTS, and orchestration layer in the same cloud region. Each cross-region hop adds 40–150ms. If your providers are only available in us-east-1, deploy your orchestrator there regardless of where your users are — provider latency dwarfs user geography latency in most cases.
Real-World Latency Benchmarks (October 2026)
Here are measured end-to-end latencies from production voice AI stacks tested on identical conversational tasks:
| Stack | p50 latency | p95 latency | Notes |
|---|---|---|---|
| OpenAI Realtime API (gpt-4o) | 320ms | 580ms | Native multimodal, no STT/TTS orchestration needed |
| ElevenLabs Conversational AI | 380ms | 650ms | Managed pipeline with best-in-class TTS |
| Deepgram + Groq + ElevenLabs | 290ms | 480ms | Fastest custom stack; requires orchestration code |
| AssemblyAI + Claude Sonnet 4.5 + Cartesia | 420ms | 720ms | Best response quality; moderate latency |
| Whisper (self-hosted) + GPT-4o + OpenAI TTS | 680ms | 1100ms | Common entry-level stack; room to optimize |
For most production voice AI applications in 2026, the target is sub-500ms p50 and sub-800ms p95 end-to-end latency. Hitting these numbers requires streaming at every stage, early LLM invocation, prompt caching, and co-located infrastructure. Teams that nail this see dramatic improvements in conversation completion rates and user satisfaction — often doubling conversion compared to 1-second+ pipelines.
→ Related: Voice Activity Detection: Streaming Buffer vs. Turn-Taking · How to Deploy Voice AI Agents That Actually Convert
🎯 Need Help Deploying a Low-Latency Voice AI?
Our team at BestStartBiz builds and deploys production voice AI agents for appointment booking, lead qualification, and customer support — with sub-500ms response times out of the box. Book a free consultation to see if your use case is a fit.
