Optimizing Interruption Handling in Realtime AI Avatars with VAD
VAD-based interruption handling for realtime AI avatars: tune thresholds, hysteresis, and turn-state policy for synced audio/video.
Introduction
Interruption handling is one of the hardest parts of building a usable realtime avatar. If your agent keeps talking after the user starts speaking, the experience feels robotic. If you cut the agent off too aggressively, you get false interruptions, clipped responses, and a lot of churn in turn-taking logic.
The core problem is simple: in a streaming voice system, you need to decide quickly whether incoming audio is user speech or background noise, and then coordinate that decision with the agent’s own generation pipeline. Voice activity detection, or VAD, is the standard tool for this. Used well, it lets you stop avatar speech at the right moment, start listening promptly, and keep the conversation feeling natural.
In this post, we’ll look at how interruption handling actually works in a realtime avatar stack, what VAD does and does not solve, and how to tune the surrounding policy so your agent behaves predictably. I’ll also show where a Protoface avatar fits into that pipeline when you’re wiring a voice agent to a synchronized video face.
What interruption handling really means
In a realtime agent, “interruption” is not just “the user talked while the assistant was talking.” It’s a set of related decisions:
Speech onset detection: detect when the user has started speaking, with low latency.
Turn preemption: decide whether that onset should stop or pause the agent.
Buffer management: discard, rewind, or retain partial assistant output depending on the turn state.
Recovery: resume generation or ask a follow-up without losing context.
VAD only answers the first question: is there likely human speech in this audio frame? Everything else is policy.
That distinction matters because people often overfit to VAD thresholds and expect them to solve turn-taking end to end. They won’t. You still need to choose frame sizes, hysteresis, minimum speech duration, and how much assistant audio to truncate when an interruption occurs.
How VAD fits into a streaming voice pipeline
A typical voice-agent loop looks like this:
The microphone sends short audio frames, often 10–30 ms each.
A VAD model or algorithm scores each frame, or small window of frames, for speech probability.
The agent aggregates those scores into events like
speech_startedandspeech_stopped.If speech starts while the assistant is speaking, the system interrupts the assistant’s output.
The ASR, LLM, and TTS pipeline then switches to a new turn.
The important engineering detail is that VAD runs on the audio input path, not on the synthesized output path. It’s watching the user side of the conversation so the agent can react before the user finishes the entire sentence. That is what makes interruption feel responsive.
For avatars, the output path matters too. If your assistant is speaking, the video face has to keep lip sync with the audio stream. When you interrupt, the audio pipeline and the avatar renderer need to agree on the cut point, or you’ll see a mouth keep moving after the audio has already stopped. That mismatch is more noticeable in video than in plain audio.
Tuning VAD for real conversations
There is no universal threshold. You tune VAD based on environment, mic quality, and the cost of a mistake.
1. Balance false positives against false negatives
A false positive means you interpret noise as speech and interrupt the assistant unnecessarily. A false negative means the user starts talking but the assistant keeps going. In most conversational interfaces, false negatives are more annoying because they force the user to wait. But if your environment is noisy, too many false positives can make the system feel jumpy.
Practical approach: start with a conservative threshold, then add hysteresis and minimum durations rather than cranking sensitivity high.
2. Use hysteresis and dwell time
Do not fire an interruption the instant a single frame crosses threshold. Require a short run of speech-positive frames before treating it as a real user turn. Similarly, require a short silence window before declaring the user done. This filters out transient noise, aspirated consonants, keyboard taps, and packet jitter.
A useful rule of thumb is to think in terms of:
speech onset window: how long speech must be sustained before it counts
speech offset window: how long silence must persist before the turn is considered over
In practice, those windows are often tens to hundreds of milliseconds, not seconds. You want responsiveness, not perfect certainty.
3. Decide what to do with partial assistant output
When the user interrupts, the agent may already have streamed several words of text and several hundred milliseconds of audio. You have to choose a policy:
Hard stop: immediately stop TTS and avatar playback.
Soft stop: stop after the current audio chunk finishes.
Truncate and replan: drop the remaining assistant text and generate a fresh response once the user finishes.
Hard stop feels most responsive, but it can clip phonemes and create awkward visual transitions. Soft stop is smoother but slightly less interruptible. The right choice depends on whether your product values conversational snappiness or polished audiovisual continuity.
4. Separate “user started talking” from “assistant should yield”
These are related but not identical. Some systems use VAD to trigger a full interruption only if the user speaks above a minimum duration or volume. Others use VAD merely to pause the assistant locally, then resume if the input turns out to be noise.
This separation is especially important in shared spaces, where VAD can pick up background speech or TV audio. If your policy is too aggressive, the assistant will constantly yield to irrelevant audio. If it is too strict, the user will have to shout over the agent.
Implementation pattern: event-driven turn control
A clean way to structure interruption handling is to treat VAD as an event source and keep turn policy in a small state machine. Conceptually:
That state machine does not need to be complicated. What matters is that the transitions are explicit and logged. When something feels off in production, the first thing you want to know is whether the system detected speech late, interrupted too early, or failed to stop TTS at the right boundary.
For example, if you are building around a voice agent framework, you generally want these hooks:
audio frame callback for VAD scoring
speech-start event to preempt the assistant
speech-stop event to resume generation or commit the user turn
assistant-audio-chunk callback for clean truncation
That structure keeps the realtime concerns localized. Your LLM logic does not need to know about frame-level VAD, and your VAD layer does not need to know about prompt engineering.
Short Python example with a realtime avatar session
If you manage avatars programmatically, the same policy applies whether the avatar is surfaced through an API or embedded in a broader agent stack. The exact request fields depend on the docs, but the shape is usually: create a session, connect your agent, then attach interruption callbacks.
The important part is not the exact method names here; it is the separation of concerns. Your VAD decides when the user is speaking. Your session control decides how to stop or resume the avatar.
Common gotchas in production
Streaming latency hides bad thresholds. In a local test, a 150 ms delay may feel fine. Over a real network with ASR, TTS, and avatar rendering in the loop, that delay compounds. Measure end-to-end turn latency, not just VAD accuracy.
Echo cancellation and speaker leakage matter. If the assistant’s audio leaks back into the microphone, VAD can mistake the agent for the user. Proper echo cancellation and input gating are as important as the VAD model itself.
Frame size changes behavior. Smaller frames reduce detection latency but increase sensitivity to jitter and transient noise. Larger windows are more stable but slower to react. Pick one deliberately and verify it under realistic network conditions.
Interruptions should be visible in logs. At minimum, log the detected onset time, stop time, assistant audio cut point, and the reason for the interruption. Without that, it is very hard to debug “the avatar talks over me” reports.
Do not ignore the visual layer. If the audio stops but the avatar keeps lip moving for another 200 ms, users notice. The video face should follow the same interruption boundary as the audio engine.
Where Protoface fits
This is where a realtime avatar layer becomes useful. If you are already running a voice agent in LiveKit, the quickstart examples and the LiveKit integration path let you drop a synchronized talking face into the same turn-taking loop. In practice, that means your VAD-driven interruption event can stop both assistant speech and avatar animation together, rather than treating the face as a separate UI that lags behind the audio state.
For developers using the LiveKit Agents stack, the Protoface plugin is the most direct way to do this. The plugin lives in the LiveKit ecosystem and is designed to attach a realtime avatar to the agent so the visual output stays aligned with the assistant’s speech state. That matters because interruption handling is only as good as the slowest synchronized component.
If you’re implementing the control plane yourself, the REST API and Python SDK are the cleanest surfaces for managing avatars and sessions, while the public docs at docs.protoface.com are the right place to confirm the exact session fields, auth flow, and supported controls.
Conclusion
Good interruption handling is mostly about policy, not just detection. VAD gives you a low-latency signal that the user may have started speaking. The rest of the system decides whether to preempt the assistant, how to truncate output, and how to keep the avatar’s audio and video aligned through the transition.
If you keep the state machine explicit, use hysteresis instead of raw thresholding, and log the turn boundaries, you will get a much more natural realtime experience. If you are adding a face to that experience, make sure the avatar layer participates in the same interruption model rather than reacting independently.
For implementation details, API shapes, and integration examples, start with the docs and the relevant repo for your stack, then test with real microphone input and realistic background noise. That’s where the edge cases show up.