Introduction

When people say “barge-in,” they usually mean one thing: the user starts speaking while the agent is still talking, and the system needs to stop cleanly enough to make the conversation feel natural. With a streaming avatar, this gets a little more subtle than in a pure voice bot. You are not only interrupting audio playback; you are also dealing with lip sync, video frame generation, transport latency, and state that may already be in flight.

Protoface is built for this class of problem. By the end of this post, you should have a practical mental model for barge-in control in a realtime avatar stack, know where to detect interruption, how to propagate cancel semantics through the pipeline, and what to do with the avatar when speech is cut off mid-utterance. I’ll use VideoSDK as the transport and conference layer, but the same control flow applies to other WebRTC-style systems.

What barge-in actually means in a realtime avatar pipeline

In a voice-only agent, barge-in is mostly about recognizing user speech early enough to stop TTS playback and suppress the rest of the current answer. In a video avatar system, there are three streams to manage:

• Input audio: the user’s microphone stream, usually arriving over WebRTC.

• Agent output audio: synthesized speech or generated speech chunks.

• Avatar video: lip-synced frames driven by the same utterance state as the audio.

The important detail is that barge-in should be treated as a control-plane event, not just a media mute. If you only stop audio, the avatar may continue animating for a few hundred milliseconds. If you only stop video generation, the audio tail still plays. The user experiences that as lag or, worse, talking over them.

A clean implementation has a single interruption signal that fans out to:

1. Cancel the current LLM/TTS turn generation.

2. Stop or drain any queued audio packets.

3. Reset the avatar mouth/pose state.

4. Start a new turn only after the user utterance is stable enough to commit.

That sounds obvious, but in practice each layer has its own buffering and timing.

Detecting interruption without false positives

The first engineering problem is deciding when a user is truly barging in. If you trigger on every transient microphone spike, the agent will constantly cut itself off. If you wait for full end-of-speech detection, the user will be talking over the model for too long.

In live systems, barge-in detection is usually a combination of:

• Voice activity detection on the inbound stream.

• Energy thresholding with a short lookahead window.

• Conversation state, so you only interrupt while the agent is actively speaking.

• Optional ASR confirmation if you want to reduce accidental cutoffs.

The simplest useful rule is: if the agent is speaking and the user stream shows sustained speech above threshold for N milliseconds, issue a barge-in cancel. N is commonly in the 150–400 ms range depending on latency tolerance and microphone quality.

Do not tie barge-in directly to raw packet arrival. WebRTC jitter and client-side audio processing can create brief gaps and bursts. You want a state machine that distinguishes:

• idle — agent not speaking

• speaking — current turn in progress

• interrupted — current turn canceled, waiting for user turn to stabilize

Once you model it that way, the rest of the stack gets easier to reason about.

Cancel semantics: make one signal stop the whole turn

The key implementation detail is propagating a single cancellation token or interrupt message through every stage that can produce output. In a typical agent pipeline that means:

• the LLM stream should stop yielding tokens,

• the TTS synthesizer should stop emitting audio chunks,

• the avatar renderer should abandon the current utterance animation, and

• any client-side playback queue should be flushed.

If your architecture is event-driven, use a monotonically increasing turn_id or utterance_id and drop late packets on receipt. That is often more reliable than trying to “stop” work already in transit. The rule is simple: every audio/video chunk belongs to exactly one turn, and anything from an older turn is ignored once a new user utterance wins arbitration.

A minimal sketch in Python looks like this:

from dataclasses import dataclass

send_audio_to_client(pcm_bytes)

from dataclasses import dataclass

send_audio_to_client(pcm_bytes)

from dataclasses import dataclass

send_audio_to_client(pcm_bytes)

That example is intentionally generic. In a real implementation, cancel_turn() should also notify your avatar layer so it can stop emitting mouth shapes and snap back to a neutral state. If you leave the renderer unaware, the video and audio will diverge for a visible fraction of a second.

Keeping audio and avatar state aligned

Streaming avatars add one extra constraint: the face must remain synchronized to whatever the agent is saying, even when the speech is cut short. The practical implication is that the avatar should not be driven by the raw audio stream alone. It should be driven by turn state plus audio timing metadata.

That gives you a few useful behaviors:

• Graceful stop: when interrupted, the avatar can close the mouth and return to idle rather than freeze mid-phoneme.

• Late-frame suppression: if the renderer already generated frames for the canceled turn, the client can ignore them using the turn id.

• Turn-bound lip sync: all frames for one answer stay visually coherent, even if network delivery is slightly out of order.

There is also a product-level choice here. Some teams prefer hard interruption: the agent fully stops speaking the instant the user begins. Others prefer a short “politeness tail,” where the agent finishes the current syllable before yielding. Technically both are the same system; they differ only in how aggressively you treat the interrupt and how much of the buffered media you allow to flush.

For most conversational experiences, a small tail is fine as long as the user can reliably take the floor. If the agent is doing support or transactional workflows, responsiveness usually matters more than perfectly finishing a sentence.

Example: wiring barge-in around a VideoSDK session

Here is the control flow I would use with VideoSDK as the realtime transport layer: keep the call session open, monitor the inbound mic for user speech, and on interrupt send a cancel event to the agent session that owns the current turn. The exact VideoSDK event names depend on your integration, but the structure is stable.

async def on_user_speech_start(session, current_turn_id: int):

return

async def on_user_speech_start(session, current_turn_id: int):

return

async def on_user_speech_start(session, current_turn_id: int):

return

If you are already using a voice-agent framework, the cancel hook often belongs in the agent runtime rather than your application layer. The same logic applies: detect speech, cancel the current generation, and make sure stale output is dropped by turn id.

One practical gotcha: if the microphone stream is duplexed back into the same room, make sure you are not triggering barge-in on your own synthesized audio. Echo cancellation and server-side VAD thresholds matter more than they usually do in pure text chat.

Where Protoface fits

For teams using LiveKit Agents, the most direct integration point is the VideoSDK quickstart together with the LiveKit plugin surface. The plugin adds a synchronized avatar to the voice agent so your turn state, audio stream, and face animation move together instead of drifting apart. If you prefer the LiveKit path, the docs cover the integration details and the expected control flow; the exact event names and request fields are documented there.

The important operational point is that barge-in should be handled at the agent/session layer, not by trying to “pause video” in isolation. Protoface’s session model is a good fit for that because the avatar is treated as part of the conversation turn, not as a separate UI widget. That makes it straightforward to cancel the active utterance, drop stale frames, and start the next response without visible desynchronization.

If you need to drive avatars programmatically outside a voice agent, the Python SDK is useful for creating sessions and managing state from your backend: protoface-sdk-python.

Testing and failure modes

Barge-in logic is easy to get mostly right and still feel wrong. A few failure modes are worth testing explicitly:

• Mic echo: the agent hears itself and interrupts its own turn.

• Premature cutoffs: the user inhales or says a filler sound and the bot stops too early.

• Late cancel: the user has already spoken for 300 ms before the agent yields.

• Stale media: audio stops but a buffered video segment still renders.

Build a small test matrix with real browser clients, mobile devices, and low-bandwidth conditions. The goal is not just correctness; it is to make the interruption feel predictable. Users tolerate a little latency. They do not tolerate a system that sometimes talks over them and sometimes panics at every sound.

Conclusion

Barge-in control is really about turn ownership. The user owns the floor the moment they start speaking, and your system should cancel the entire active turn quickly enough that audio, video, and state all agree. In a streaming avatar stack, that means using a unified interrupt signal, dropping stale chunks by turn id, and resetting the avatar to a neutral state when the current utterance is abandoned.

If you are implementing this on top of a realtime agent, start by wiring the cancel path end-to-end before you tune VAD thresholds. Then test with real microphones and real network conditions. For integration details, examples, and the available SDK surfaces, check docs.protoface.com and the relevant quickstarts on GitHub.