Introduction

Barge-in is the moment a user starts speaking while your avatar is still talking, and the system has to decide what happens next. In a realtime AI avatar pipeline, that sounds simple until you have to coordinate microphone input, ASR, LLM token generation, TTS playback, lip-sync video, and transport timing across WebRTC and WebSocket connections. If you get it wrong, users hear overlapping audio, see the mouth keep moving after they interrupted, or experience a long “dead air” pause before the agent acknowledges them.

By the end of this post, you should have a practical mental model for designing barge-in handling that feels responsive, avoids race conditions, and works across typical voice-agent architectures. We’ll cover where barge-in should be detected, how to propagate cancellation cleanly, how to keep audio and avatar video in sync, and what to watch for when the avatar is delivered over WebRTC versus a WebSocket-driven web app.

What barge-in really means in a realtime avatar stack

In a voice-only agent, barge-in usually means “stop speaking when the user starts talking.” With an avatar, there are two more layers to cancel:

• Audio output: stop TTS playback immediately and prevent queued audio from continuing.

• Visual output: stop or transition the avatar’s talking animation so lip-sync does not continue after the interruption.

• Agent turn state: cancel the current response generation, or at least mark it stale so late tokens don’t get spoken.

The important distinction is that barge-in is not just a UI gesture. It is a turn-management event that crosses process boundaries. If your avatar renderer, audio player, and conversational agent each make independent decisions, they will drift. Design for one authoritative turn state, then fan out cancellation to all downstream components.

Detecting interruption without over-triggering

The simplest implementation is “if VAD says user is speaking, barge in.” That works poorly in practice. Short breaths, keyboard noise, and echo from the avatar’s own audio can all look like interruption. You want a policy that combines activity detection with a notion of intent.

A good starting point is:

1. Use endpointed voice activity detection on the inbound mic stream to detect a real speaking onset, not just a transient spike.

2. Gate on output state: only allow barge-in when the agent is actively producing audio or animated speech.

3. Add a minimum interruption window: require some sustained activity before canceling the current turn.

4. Optionally apply echo suppression: if the user is on speakers, your system may hear the avatar’s own voice coming back through the microphone.

In practice, the first few hundred milliseconds matter most. You want to react quickly enough that the user feels heard, but not so aggressively that a half-formed utterance blows away a useful answer. For many products, a threshold in the 150–300 ms range after speech onset is a better trade-off than immediate cancellation on the first non-silent frame.

Turn-state design: one source of truth, many side effects

Make barge-in a state transition on the current assistant turn. A minimal state machine looks like this:

• Idle: no assistant speech is playing.

• Speaking: TTS/audio/video are active for the current turn.

• Interrupted: user speech was detected; current turn is canceled.

• Generating next response: the agent is now listening and will answer the user’s interruption.

The crucial behavior is that all downstream tasks are tied to a turn ID. When barge-in occurs, increment or invalidate the turn ID and propagate cancellation to every consumer. That prevents stale work from leaking through after the user has already taken over.

In a WebSocket pipeline, this often means sending a cancel message to the server and ignoring any late-arriving chunks that belong to the old turn. In a WebRTC pipeline, you may also need to stop a local media track or stop feeding frames into the avatar renderer. The exact transport differs; the rule does not: old turn data must become non-authoritative immediately.

Keep audio and lip-sync aligned during cancellation

Most user-facing bugs show up as desynchronization. The avatar keeps moving for half a second after speech stops, or the audio cuts out but the lips keep flapping because the renderer is still consuming buffered phonemes. The fix is to treat audio and video as siblings under the same cancelable turn.

Three practical rules help:

1. Cancel generation first, then playback: stop future tokens/phonemes before draining buffers. Otherwise you may create a race where new output is produced while old output is still being flushed.

2. Bound the tail: keep the audio/video buffer small enough that interruption feels immediate. Large buffers make cancellation sluggish.

3. Snap the avatar out of speech mode: once barge-in is accepted, transition the face to an idle or listening state rather than letting it naturally finish the old sentence.

Be careful with “graceful fade-out” behavior. It can look polished in demos, but it often makes interruption feel less responsive. For barge-in, responsiveness usually beats aesthetics. A short, deterministic stop is easier to reason about and easier for users to understand.

WebRTC vs. WebSocket: where the control plane should live

WebRTC and WebSocket solve different parts of the problem. WebRTC is typically your media path: low-latency audio/video transport, jitter handling, and playback. WebSocket is often your control path: session events, token streams, agent messages, and cancellation signals.

That split matters because barge-in is both a media event and a control event. If you only stop the media track, the agent may still continue generating response text or TTS. If you only cancel the LLM, audio already buffered into the player may still leak out. The best design is:

• Media plane: stop the current audio/video output immediately.

• Control plane: emit a cancellation event tied to the current turn/session.

• Application plane: record that the current assistant turn was interrupted and can no longer produce user-visible output.

For server-side agents, this often means the ASR/VAD service, the LLM orchestration layer, and the TTS/avatar layer all need to understand the same turn ID. For browser-based clients, you also need to avoid stale UI state: disable “speaking” indicators as soon as the interruption is accepted, not when the backend eventually confirms it.

Implementation pattern: cancel by turn ID, not by vague flags

A common anti-pattern is a shared boolean like isSpeaking = false. That works until two turns overlap, a reconnect happens, or a late network packet arrives. Prefer explicit turn IDs and idempotent cancel messages.

class

class

class

That pattern is intentionally boring. Boring is good here. If every chunk of TTS, every avatar frame, and every agent token carries the turn ID it belongs to, cancellation becomes a simple equality check instead of a distributed guessing game.

When using a streaming LLM or TTS service, cancel early and make cancellation idempotent. You should be able to send the same cancel event twice without breaking the session. In real systems, retries happen, websocket reconnects happen, and message ordering is not guaranteed unless you enforce it.

How Protoface fits in

If you are adding a talking avatar to an existing LiveKit voice agent, the cleanest path is to keep your turn management in the agent and let the avatar layer consume the resulting speech stream. The LiveKit/Pipecat integration and the Pipecat guide are useful references for this pattern: your agent owns interruption policy, and the avatar renderer follows the agent’s current turn state.

For a Python-based integration, the general shape is straightforward: create the avatar/session, attach it to your agent pipeline, and propagate interrupts by turn ID. Exact fields and method names vary by SDK version, so treat the example below as illustrative and verify against the docs.

from protoface_sdk import Client<p></p>

from protoface_sdk import Client<p></p>

from protoface_sdk import Client<p></p>

If you are wiring this manually over the REST API, use the dashboard or API keys for session management and keep secrets out of the browser. The public docs at docs.protoface.com cover the supported session and avatar flows. For local experimentation, the quickstart repos linked from the project README are a practical way to see turn cancellation in context.

Testing barge-in like a distributed systems problem

Do not rely on happy-path demos. Test interruption under realistic failure modes:

• Network jitter: does the avatar keep speaking after the cancel event is delayed?

• Late frames: are old audio/video chunks dropped after interruption?

• Echo: does the agent interrupt itself when the avatar’s voice leaks into the mic?

• Reconnects: does a websocket reconnect resurrect a canceled turn?

• Partial utterances: can the user say “wait, actually...” without losing the interruption?

A useful test harness records the current turn ID, every cancel event, and the exact timestamps of mic onset, audio stop, and avatar state transition. If those three timestamps are far apart, users will feel it even if your code looks correct on paper.

Conclusion

Barge-in handling is mostly a coordination problem: detect user speech reliably, invalidate the current turn immediately, and make sure audio, lip-sync, and agent generation all stop using the same cancellation signal. The design works best when you treat turn state as the source of truth and make every downstream component idempotent and turn-aware.

If you are implementing this in a realtime avatar pipeline, start with explicit turn IDs, small buffers, and a strict cancel path. Then validate it under jitter and echo, not just in a clean local demo. For integration details and supported session flows, check the docs and the relevant quickstarts before you wire it into production.