Building Interruption Handling for VideoSDK Voice and Video Agents
How to implement interruption handling in voice/video agents: barge-in, turn cancellation, audio stop, and avatar sync.
Introduction
Interruption handling is one of those details that separates a demo from a voice agent people can actually use. When a user starts talking over the agent, the system should stop speaking quickly, stop rendering stale audio, and keep the conversation state consistent. If you also have a synchronized avatar, the visual layer has to react too: the mouth should stop moving, the next turn should not clip, and the agent should resume cleanly after the barge-in.
This post is about building that behavior in a real-time voice or video agent. By the end, you should understand where interruption detection belongs in the pipeline, how to propagate it through your agent stack, and how to avoid the common failure modes: delayed cutoff, duplicate turns, desynced lips, and state that drifts after a user interrupts.
What interruption handling actually means
In a realtime agent, “interruption” usually means the user starts speaking while the assistant is still generating or playing audio. In practice, you need to handle three related events:
Input barge-in: the user begins speaking while the assistant is mid-turn.
Output cancellation: any queued TTS audio and playback must be stopped immediately.
Conversation replan: the agent should treat the user’s new utterance as the next turn, not as noise.
Those sound simple, but the implementation details matter. If you only stop playback locally and don’t notify the agent state machine, the model may keep generating a response that later gets emitted. If you cancel generation but leave the audio buffer intact, the user hears the tail end of the old answer. If you forget the avatar, the face can continue lip-syncing after speech has stopped, which is uncanny and confusing.
Where interruption detection belongs in the pipeline
A clean implementation splits the problem into three layers:
Voice activity detection / endpointing determines that the user has started speaking.
Agent orchestration decides to cancel the current assistant turn and advance the conversation state.
Media delivery stops audio playback and any synchronized video/lip animation immediately.
The first layer can be model-based or energy-based, but either way it should produce a high-confidence “user is talking now” signal quickly. The second layer needs an explicit cancellation path; “just ignore the old output” is not enough if you stream tokens, stream TTS, or render an avatar from the same turn. The third layer should be idempotent: calling stop twice should be harmless.
A useful mental model is that interruption is not a UI concern. It is a turn-management concern that happens to have UI effects. That distinction matters when you’re debugging race conditions.
Designing the cancellation path
The most common bug is a partial cancel. The user interrupts, you stop the speaker, but the agent keeps producing more tokens or audio chunks in the background. A robust cancellation path should terminate all of these as a unit:
active model generation
TTS synthesis or streaming playback
queued audio frames waiting to be sent
avatar lip-sync associated with the current turn
In code, that usually means a cancellation token, an atomic “current turn” identifier, or both. The core requirement is that every downstream consumer knows whether the turn it is handling is still current. If it is not current, it should drop work silently.
Here is a simple shape for the control flow in Python:
This is intentionally schematic. The important part is not the exact API surface; it is the separation between turn identity, cancellation, and media stop. In a production system, you also want to clear the barge-in flag only when a new assistant turn begins, not immediately on detection.
Latency, endpointing, and false positives
Good interruption handling is mostly about time. Users notice when the system takes too long to stop. If your agent continues talking for 400–700 ms after the user starts speaking, the experience feels broken even if it is technically correct.
There is a trade-off between responsiveness and false positives. Aggressive endpointing makes the system stop more quickly, but it can also interrupt the agent when the user coughs, says “uh-huh,” or makes a short backchannel. Overly conservative endpointing makes the system feel sluggish and disrespectful. The practical approach is:
detect speech onset quickly;
apply a short debounce or confidence threshold if your input is noisy;
cancel output immediately once you commit to barge-in;
let the next user utterance become the new turn without manual recovery.
For full-duplex systems, another subtle issue is echo. If the microphone picks up the agent’s own audio, your interruption logic may think the user started speaking when they did not. Acoustic echo cancellation helps, but you should still treat the interruption signal as a product of the whole audio stack, not just a single detector.
Keeping the avatar in sync with the audio
If your agent has a visual face, interruption must stop animation state, not just sound. Lip sync is typically driven by audio timestamps or phoneme events, so once output is cancelled, the avatar should immediately transition out of “speaking” mode. Otherwise the video keeps moving against silence, which is one of the fastest ways to make a realtime assistant feel broken.
The implementation detail that trips teams up is that the avatar often lags the audio by a frame or two or by a small buffering window. That is fine during normal playback, but on interruption you want the opposite behavior: fail closed. Stop the avatar as soon as the current turn is no longer valid, even if a tiny amount of queued motion would otherwise continue.
That also means your video layer should be driven by the same turn state as your audio layer. Do not let the avatar subscribe to raw TTS chunks independently unless it also knows which turn those chunks belong to. Otherwise a late chunk from the cancelled turn can revive motion briefly.
How Protoface fits into this
If you are already running a LiveKit voice agent, the cleanest place to handle this is in the agent integration layer. The Protoface LiveKit plugin drops a synchronized talking avatar into the agent so the face follows the same turn lifecycle as the voice stream. That means interruption handling is not an extra, separate visual system; it is part of the same turn you are already managing. For implementation details and examples, see the docs and the plugin repository.
A typical integration looks like this at a high level:
The key point is not the constructor shape; it is that the avatar is attached to the same agent turn lifecycle. If your interruption logic already cancels the assistant response in LiveKit, the avatar should receive the same stop signal instead of trying to infer state from media timing alone. That keeps audio, lip sync, and conversation state aligned.
Testing interruption behavior like a real system
You should test interruption at the system boundary, not just with unit tests on the detector. A useful manual test is:
Start an assistant answer that lasts several seconds.
Speak over it after the first sentence.
Verify that audio stops within a small, consistent delay.
Verify the avatar stops speaking immediately and does not “finish the sentence” visually.
Verify the next assistant turn answers the new user utterance, not the original prompt.
Also test partial failures: network jitter, delayed TTS chunks, and rapid back-to-back interruptions. These are the cases that usually reveal whether your cancellation is truly turn-scoped or just best-effort. If you have metrics, measure time-to-cutoff and the number of cancelled-but-still-rendered audio frames. Those numbers tell you more than subjective impressions.
If you need to create or inspect sessions while debugging, the REST API is useful for automation and inspection from trusted backend code. The API is authenticated with bearer keys; keep those server-side. For example:
Use the exact endpoints and fields from the API docs; the important operational point is that session creation and debugging should happen off the client, where you can safely manage credentials and inspect state transitions.
Conclusion
Interruption handling is really about making a realtime agent obey turn boundaries under stress. Detect speech onset quickly, cancel the active assistant turn as a unit, stop audio and avatar motion together, and make the next user utterance the new source of truth. If you get those mechanics right, the rest of the experience becomes much easier to reason about.
For more implementation details, quickstarts, and integration references, start with the docs and the relevant GitHub examples. If you are building on LiveKit, the plugin path is a practical place to wire interruption into the same lifecycle that drives your voice agent. The main thing is to treat interruption as core turn-state logic, not an afterthought.