Voice AI feels off when the agent talks over the user, takes too long to start, or starts mid-thought. Three failure modes, all of them solvable. The fix is rarely "a faster model." The fix is turn-taking discipline.
What latency really means in voice
Users do not perceive end-to-end latency. They perceive turn-taking latency: the gap between when they finish speaking and when the agent starts responding. A 1.2-second gap with a tasteful filler word feels conversational. A 600ms gap with no filler can feel jarring, because the user's brain hasn't yet expected a response. Counterintuitive but consistent across our user research: faster is not always better. Predictable is better.
The four numbers to optimize
- VAD endpoint latency. How quickly your voice activity detector decides the user is done speaking. Aim for 200–400ms — more, and you feel slow; less, and you cut people off mid-thought.
- STT finalization. The lag between the last audio packet and the final transcription. Streaming STT can hide this, but the agent shouldn't act on partial transcripts for high-stakes turns.
- First-token latency. The lag from prompt sent to first generated token. This is the only number that benchmarks usually report, and it is one of four.
- TTS first-audio latency. Streaming TTS gets you to first-audio in 100–200ms once the LLM starts emitting. Without streaming, you wait for the full response and feel an extra 800ms.
Barge-in: harder than it looks
Barge-in — the user interrupting the agent — is the hardest UX problem in voice AI. The naive implementation cuts the agent off the moment the VAD detects user speech. The correct implementation distinguishes between an actual interruption ("wait, no") and a backchannel ("uh-huh"). Cut the agent off too aggressively and conversations become stilted. Don't cut it off enough and users feel ignored.
Our heuristic: a backchannel is short (under 600ms) and high-pitch; an interruption is longer or contains lexical content. We classify in real time using a lightweight model and tune the threshold per deployment.
Filler words are a feature, not a bug
"Let me check that for you." "One moment." These are not delays — they are turn-taking signals. They tell the user the agent has heard them and is working. We inject filler audio as soon as the user's turn ends, in parallel with the LLM call. The result feels faster than a system that produces a perfect, silent response 400ms sooner.
What we'd build different in 2026
Two things. First, we now run a tiny on-device model in parallel for the "is this an interruption?" classifier — round-tripping it to the cloud added 80ms of latency that mattered. Second, we treat the dialog state machine as a first-class artifact: explicit states, explicit transitions, explicit eligible barge-in windows. The chattier voice agents that emerged in 2024 — "the LLM is the dialog manager" — sound great but cannot meet enterprise reliability standards. The dialog manager is a state machine. The LLM is a content generator.
Voice agents are conversational, not generative. The conversation is a state machine. The model fills in the words.