Voice Activity Detection: Streaming Buffer vs. Turn-Taking

Voice Activity Detection — VAD — is the invisible decision engine that determines when your voice AI thinks the user is speaking, when they’ve stopped, and when it’s safe to respond. Get it right and conversations feel seamless. Get it wrong and your AI either cuts users off mid-sentence or sits in awkward silence after they’ve finished.

Most voice AI tutorials present VAD as a single concept — “detect when someone is speaking.” In practice, there are two fundamentally different approaches that solve very different problems: streaming buffer VAD (frame-level speech detection) and turn-taking VAD (conversation-level endpointing). Confusing them is why so many prototype voice agents feel broken even when the STT, LLM, and TTS stages are working perfectly.

What VAD Actually Does

At the lowest level, VAD inspects an audio stream frame by frame (typically every 10–30ms) and classifies each frame as either speech or non-speech. Non-speech includes silence, background noise, music, and non-human sounds.

Modern VAD models like Silero VAD, WebRTC VAD, and Krisp’s neural VAD run in real time with latencies under 10ms per frame and accuracy above 95% on clean audio. The raw frame-level output looks like this:

frame_0010 → non-speech (confidence: 0.98)
frame_0011 → non-speech (confidence: 0.97)
frame_0012 → speech     (confidence: 0.89)  ← user starts
frame_0013 → speech     (confidence: 0.94)
frame_0014 → speech     (confidence: 0.92)
...
frame_0087 → speech     (confidence: 0.76)
frame_0088 → non-speech (confidence: 0.82)  ← user pauses
frame_0089 → non-speech (confidence: 0.95)

This raw frame-level output is the input to both VAD strategies — but what each strategy does with it diverges completely.

Streaming Buffer VAD vs. Turn-Taking VAD

The Core Difference, Illustrated

Imagine a user says: “I’d like to book an appointment for… [500ms pause] …next Tuesday if possible.”

A streaming buffer VAD sees the 500ms pause and immediately marks it as non-speech. If your pipeline treats “non-speech for 500ms” as “end of turn,” your AI will respond mid-sentence while the user is still thinking. The user has to start over.

A turn-taking VAD is smarter. It analyzes the semantic context — “for…” is clearly an incomplete thought — and waits longer before declaring end-of-turn. Modern turn-detector models like LiveKit’s and Pipecat’s incorporate not just silence duration but linguistic cues from partial STT transcripts to make this decision.

The Three Key Thresholds You Must Tune

Every voice AI pipeline that uses VAD has at least three configurable thresholds. Getting them right for your use case is the difference between natural conversation and constant frustration.

1. Minimum speech duration (start-of-speech)

How many consecutive speech frames before you consider “the user has started talking”? Too low and background noise triggers false positives. Too high and short confirmations (“yeah,” “okay”) get missed. Typical value: 100–200ms (3–6 frames at 30ms each).

2. Minimum silence duration (end-of-speech)

How many consecutive non-speech frames before you consider “the user has stopped talking”? Too short and natural pauses get treated as end-of-turn. Too long and your AI feels sluggish. Typical value: 500–800ms for most voice agents. Phone-based agents often need 800–1200ms because telephony audio has more variable silence detection.

3. Minimum utterance duration (noise filtering)

Shortest “speech” event you’ll treat as real. A single frame of detected speech surrounded by silence is almost always a cough, a door slam, or a VAD false positive. Typical value: 200–300ms minimum utterance length.

vad_config = {
    "min_speech_ms": 150,         # 5 frames @ 30ms
    "min_silence_ms": 650,        # Natural end-of-turn
    "min_utterance_ms": 250,      # Filter coughs/door slams
    "speech_pad_ms": 100,         # Add padding to avoid clipping
    "model": "silero_v4",
    "sample_rate": 16000
}

Handling the Two Hardest Edge Cases

Edge case 1 — User interruption during AI response

When your AI is speaking and the user starts talking, you have about 300ms to detect the interruption, stop TTS playback, and begin listening. Any slower and the user ends up talking over the AI with both audio streams colliding — users hate this.

This requires streaming buffer VAD running continuously on the user’s microphone input, even while the AI is speaking. The moment the VAD detects 200ms+ of speech, cancel the TTS playback, cancel the current LLM request, and begin a new conversation turn with the user’s input. This is called “barge-in handling” and it’s the single most important feature separating consumer-grade voice AI from prototype-grade.

Edge case 2 — Natural thinking pauses

Humans don’t speak in clean sentences. We say “um,” pause mid-thought, and trail off while searching for words. A streaming buffer VAD treats all of these as end-of-turn, resulting in constant interruptions.

The solution is context-aware turn detection that combines three signals:

• Silence duration — how long the user has been quiet
• Partial transcript — does the latest STT output end on a grammatically complete clause?
• Semantic model — a small LLM or specialized model that predicts “end of turn” probability from the partial transcript

LiveKit’s turn detector and Pipecat’s smart turn detection do all three. For a hand-rolled stack, the simplest high-impact rule is: if the partial transcript ends with “and,” “but,” “so,” “um,” “for,” “the,” or similar connective/incomplete tokens, extend the silence threshold by 500ms. This single heuristic eliminates ~80% of mid-thought interruptions.

Which VAD Strategy for Which Use Case

→ Related: What Is Real-Time Latency in Voice AI Pipelines?  ·  How to Deploy Voice AI Agents That Actually Convert