Introduction

Interruptible voice and video agents are harder than they look. The basic “text in, audio out” loop is straightforward, but once you add a synchronized talking face, barge-in, and low-latency turn-taking, you now have a realtime systems problem: multiple streams, partial outputs, cancellation, and UI state that must stay consistent when the user interrupts mid-utterance.

This post is about building that system cleanly. By the end, you should be able to reason about where interruption should be detected, how to stop generation without leaving stale audio or video on screen, and how to structure your agent so it feels responsive instead of robotic. I’ll also show where a realtime avatar layer like Protoface fits into the stack without turning your application into a pile of ad hoc state transitions.

What “barge-in” actually means in a voice/video agent

In telephony and voice UX, barge-in means the user starts speaking before the agent has finished. In a modern agent, that usually implies three simultaneous cancellations:

• ASR interruption: stop treating the current agent turn as active once user speech is detected.

• LLM interruption: stop generation or discard tokens from the current response.

• TTS/media interruption: stop audio playback and any associated avatar mouth animation or lip-sync output.

The important part is that these are not the same event. Voice activity detection may notice the user first, but your model might already be midway through generating a response and your player may already have queued 300 ms of audio. If you only cancel one layer, the UI will still feel broken: the avatar keeps talking while the app has logically switched turns, or the assistant’s audio cuts off but the face continues moving for another second.

A good implementation treats interruption as a state transition, not just a stop button. The current assistant turn should move from active to aborted, and every stream attached to that turn should respect the same cancellation token or turn identifier.

Design the turn model first

The simplest robust design is to assign every conversational turn a monotonically increasing turn_id. When the assistant starts responding, you create a new turn context containing:

• the generation task for the LLM,

• the TTS synthesis stream,

• the audio playout buffer,

• the avatar/video renderer state.

All downstream components check the same turn_id before emitting output. When a barge-in occurs, you increment the active turn, mark the previous one canceled, and discard anything that arrives from the old context.

That approach solves a subtle class of bugs: late packets. In realtime systems, cancellation is rarely instantaneous. A token may arrive after you told the model to stop; a TTS chunk may already be in flight; a video frame may still be buffered client-side. If each payload is tagged with the current turn, the consumer can ignore stale media deterministically.

# Pseudocode: keep a single source of truth for turn state

# Pseudocode: keep a single source of truth for turn state

# Pseudocode: keep a single source of truth for turn state

Detect interruption early, but confirm it before canceling everything

In practice, barge-in detection usually starts with VAD or energy-based speech detection on the inbound audio stream. That is fast, but it is also noisy. A door slam or cross-talk can look like user speech. For that reason, I prefer a two-stage model:

1. Soft interrupt: pause assistant playout and mark the current turn as suspect when VAD fires.

2. Hard interrupt: once you confirm sustained user speech, cancel the turn and flush buffered output.

This keeps the agent responsive without overreacting to transient noise. The exact threshold depends on your audio pipeline, but the principle is consistent: don’t wait for a perfect signal, and don’t immediately destroy state on the first spike.

Also remember that interruption should happen before the UI feels “stuck.” If the user begins speaking, the system should stop the avatar’s speaking state immediately, even if the backend is still unwinding tasks. The assistant can resume later, but the live interaction should prioritize the user.

Synchronizing audio and avatar video

Once you have a talking face, the hard part is keeping the avatar aligned with the audio stream under cancellation and partial generation. In a clean architecture, the video layer should not independently infer conversational state. It should reflect the same assistant turn that drives audio playback.

That means two rules:

• Drive lip-sync from the same turn context as audio, not from raw text alone.

• Stop both audio and animation together when the assistant is interrupted.

If your avatar renderer takes text chunks, you still want a turn-aware gate in front of it. Text can be revised, truncated, or superseded. Audio is more expensive to unwind. When barge-in occurs, do not let the avatar finish an old sentence just because the last token was already converted to mouth movement.

For WebRTC-based delivery, this usually means your server or agent runtime publishes media state changes separately from text generation events. Clients subscribe to a single “assistant speaking” state, plus the actual audio/video tracks. When the turn is canceled, they tear down or mute the tracks and reset the avatar to an idle pose immediately.

Latency trade-offs that matter in practice

There are three latency budgets you need to watch:

• Input detection latency: how quickly you notice the user started speaking.

• Cancellation latency: how quickly generation and playback stop after detection.

• Recovery latency: how quickly the agent can produce the next useful response.

Most teams optimize the first one and ignore the third. That is a mistake. If barge-in is too aggressive and your agent needs a full cold start after every interruption, the conversation becomes choppy. A better pattern is to preserve conversation context and only cancel the active turn, not the whole session.

A few practical tips:

• Use short audio chunks so playback can be interrupted cleanly.

• Stream tokens instead of waiting for a full LLM completion.

• Prefer idempotent “stop this turn” control messages over shared mutable flags scattered across services.

• Log turn transitions and cancellation causes; debugging realtime UX without this is painful.

One more gotcha: browsers and clients may buffer media. Even if your server stops immediately, the frontend might still have a small queue of audio samples or animation frames. Your client code should treat interruption as authoritative and clear local playout buffers when the turn changes.

Implementation pattern: stream events, not just text

The cleanest implementation uses an event stream for the agent lifecycle, not a single “send me the final response” API. Your events might look conceptually like:

• user_speaking_started

• assistant_turn_started

• assistant_audio_chunk

• assistant_video_frame

• barge_in_detected

• assistant_turn_canceled

That event model gives you one place to coordinate audio, video, and UI state. It also makes it easier to recover from transport hiccups because each client can replay the state machine from known events rather than guessing what the assistant is doing.

For developers integrating with existing voice stacks, the pattern is the same whether you’re on LiveKit, Pipecat, or another realtime runtime: keep the conversation turn as the unit of cancellation, and treat media as a projection of that turn.

Where Protoface fits: adding the avatar layer without breaking turn control

If you already have a voice agent and want a synchronized face, the least disruptive path is to add the avatar as a realtime media surface that follows the agent turn state rather than becoming the authority for it. That is exactly the kind of integration the LiveKit agent plugin is meant for: you keep your agent logic where it already lives, and the avatar tracks the speaking state and lip-sync output on top of it. The plugin examples in the OpenAI Realtime quickstart and the docs are good references for the wiring patterns.

If you need to manage avatars or sessions directly, the REST API and Python SDK give you a backend-controlled path. A typical flow is: create an avatar, start a session, then bind that session to the agent runtime. The exact request fields depend on the endpoint, but the shape is straightforward:

curl -X POST https://api.protoface.com/<endpoint> \
}'

curl -X POST https://api.protoface.com/<endpoint> \
}'

curl -X POST https://api.protoface.com/<endpoint> \
}'

And in Python, the SDK is used in the same spirit: create the avatar/session objects, then attach them to your agent lifecycle. See the SDK repo and the public docs for the exact method names and payloads: Python SDK, docs.

The key design point is that the avatar should inherit interruption behavior from your voice system. When the user barges in, you cancel the active turn once, and the audio/video layer follows that state change. That keeps the implementation sane and prevents “talking head” artifacts where the face keeps moving after the response has logically ended.

Conclusion

Interruptible voice and video agents are mostly a coordination problem. Use turn IDs, propagate cancellation through the whole media pipeline, and make the avatar a consumer of agent state rather than a separate source of truth. Detect user speech quickly, confirm interruption before hard canceling, and clear both audio and video state together.

If you’re building this stack now, start with a turn-aware event model, then integrate the avatar layer on top. The implementation details vary by runtime, but the systems principle does not. For integration specifics, quickstarts, and API details, go to docs.protoface.com.