Introduction

When you add a talking avatar to a realtime voice agent, “user barge-in” becomes a core control problem, not a UX detail. The user starts speaking before the agent finishes. If you do nothing, the assistant keeps talking, the audio stream collides with the user’s microphone input, and the avatar keeps lip-syncing through a turn that should have been interrupted.

In practice, barge-in means detecting that the user has started a new turn and immediately stopping any avatar speech already in flight. The implementation details vary by stack, but the goals are consistent:

• stop TTS or agent output fast enough that the user feels heard,

• cancel or ignore any queued avatar audio/video frames,

• reset turn state so the next response starts cleanly, and

• avoid race conditions when the interruption arrives mid-stream.

This post walks through the mechanics of reliable barge-in handling for realtime avatars, including where to hook interruption detection, how to wire cancellation through your agent pipeline, and what to watch for when an avatar is synchronized over WebRTC. I’ll also show how Protoface fits into that loop without changing the underlying turn-taking model.

What “stop avatar speech” actually means

For a realtime avatar, “speech” is not one thing. It usually spans at least three streams:

1. generated text from the agent,

2. audio from a TTS or voice provider, and

3. video frames or lip-sync state driving the avatar face.

If you only stop one of these, the system still looks broken. For example, muting audio but letting the video continue causes obvious mouth motion after the user has interrupted. Likewise, clearing the video without canceling TTS can leave a hidden audio backlog that resumes too late.

A clean interruption path should therefore cancel the whole response pipeline, not just the render layer. The avatar should be treated as a downstream consumer of the agent’s current turn. Once the user barges in, that turn is invalid.

Detecting barge-in reliably

The best barge-in detection is usually upstream of the avatar, at the point where you already have realtime audio or turn-state signals from the voice stack. You want to detect user intent to speak, not just arbitrary noise.

Common triggers are:

• voice activity detection crossing a threshold for a minimum duration,

• speech-to-text partials arriving while the assistant is speaking,

• an explicit “interruption” event from your voice agent framework, or

• a turn state change from the transport/session layer.

The important part is to debounce it. A single spike of audio energy should not cancel the assistant. In a noisy room, you need a short confirmation window, and you should generally require that the user is actually producing speech-like input while the assistant is mid-turn.

Also decide what you consider “speaking.” In a voice agent, the assistant may be producing text, synthesizing audio, or already streaming audio/video. Only the last two are visible to the user, but the cancellation should often start as soon as the system knows the turn is being superseded.

Propagate cancellation through the turn pipeline

Once you detect barge-in, the implementation should be a straight cancellation path, ideally using a shared cancel token or equivalent turn-scoped state. Do not rely on “just stop sending frames” unless every component honors that immediately.

A practical pipeline looks like this:

1. mark the assistant turn as canceled,

2. stop TTS generation or close the TTS stream,

3. drop any queued audio/video chunks for that turn,

4. flush or reset the avatar renderer, and

5. accept the user’s utterance as the new active turn.

That last step matters. If you cancel the assistant but keep the turn state ambiguous, the next few hundred milliseconds can get misclassified as residual assistant activity, and you end up with stuck half-turns or duplicated responses.

Here’s a compact Python sketch using the kind of turn-scoped cancel flow you’d want in a realtime agent integration:

import asyncio

await stop_tts_provider()

import asyncio

await stop_tts_provider()

import asyncio

await stop_tts_provider()

The exact APIs depend on your voice stack, but the pattern is the same: the renderers should be consumers of a turn that can be invalidated synchronously.

Handling the avatar side cleanly

From the avatar’s point of view, barge-in is a reset condition. If your avatar is lip-syncing from audio, the safest move is to stop emitting avatar audio immediately and clear any buffered mouth-motion state. If your avatar is driven by a streaming media pipeline, you want to drop pending frames from the canceled turn before the user’s speech becomes the new active source.

This is where timing bugs show up. Network and media pipelines buffer by design. A few frames may already be in flight when the interruption arrives, so “stop now” usually means “stop now plus discard anything queued after this turn boundary.”

There are two useful invariants:

• no frame generated after cancellation should be rendered for the old turn,

• the avatar should re-enter an idle state quickly enough that the user sees the interruption take effect.

If your system keeps a local playback buffer, you should flush it. If the avatar is on a remote renderer, you need a control message or session update that invalidates the turn on the server side. Either way, the key is that turn ownership changes atomically.

What to do about race conditions

Interruption handling is a concurrency problem. The user may start speaking at the same time the assistant is finishing a sentence, the TTS provider may already have produced the next chunk, and the avatar renderer may be halfway through a frame queue. Without explicit turn ownership, those events race.

Three practical rules help:

1. Make cancellation idempotent. Call it more than once if needed; it should be safe.

2. Check the cancellation flag before every downstream send. Don’t assume one check at the top of the coroutine is enough.

3. Attach a turn ID to every chunk. If a late chunk arrives, you can discard it because it no longer matches the active turn.

That turn ID pattern is especially useful when you have separate processes for agent logic, TTS, and avatar rendering. The message bus can be slow; the turn boundary must still be authoritative.

In systems with WebRTC transport, remember that media packets and control events do not arrive with the same timing guarantees. The control plane should be the source of truth. Media packets are best-effort and can lag slightly behind the state transition.

Using Protoface in a LiveKit voice agent

If you’re already running a LiveKit-based voice agent, the simplest way to add synchronized avatar speech is the LiveKit plugin. The relevant package is the Protoface quickstart for VideoSDK for a similar integration pattern, and the LiveKit plugin follows the same underlying idea: the avatar is attached to the agent’s turn lifecycle, so when the user barges in, you cancel the active turn and the avatar stops with it.

In a LiveKit Agents setup, the practical integration point is the same one you’d use for audio output: when your agent framework signals interruption or the user starts talking over the bot, cancel the current speech task and let the plugin observe the turn end. The plugin handles the synchronized face; you still own the interruption policy.

A representative pattern looks like this:

from livekit.agents import JobContext

await avatar.stop_current_speech()

from livekit.agents import JobContext

await avatar.stop_current_speech()

from livekit.agents import JobContext

await avatar.stop_current_speech()

The exact class and method names depend on the SDK version, so treat this as a shape, not a drop-in snippet. The important part is that the avatar does not get special treatment. It follows the same cancellation semantics as the rest of the assistant turn.

If you prefer to inspect the lower-level API surface or connect your own pipeline, the docs at docs.protoface.com and the plugin examples in the relevant GitHub repos are the right starting point.

Testing the interruption path

Barge-in bugs are easy to miss if you only test happy-path conversations. You need scenarios that intentionally interrupt the assistant at different points in the turn:

• before TTS starts,

• mid-utterance,

• while video is already playing but audio is buffered,

• during network jitter, and

• with noisy background audio that should not trigger cancellation.

What to observe:

• Does the assistant audio stop within one perceptible beat?

• Does the avatar mouth stop moving immediately after the cancel boundary?

• Does the next user utterance become the active turn cleanly?

• Are there any stale frames or late audio chunks after cancellation?

If you have logs, include turn IDs and the reason for cancellation. That makes it much easier to distinguish a legitimate barge-in from an accidental teardown or session reset.

Conclusion

To stop avatar speech on user barge-in, treat interruption as a turn-level cancellation problem, not a rendering problem. Detect user intent to speak, invalidate the current assistant turn, cancel TTS and queued media, flush the avatar output path, and make the whole flow idempotent so late packets do not leak through.

That approach works whether your avatar is embedded in a browser, driven through a voice agent, or attached to a streaming transport like WebRTC. If you want a concrete implementation path, start with the relevant quickstart or plugin example, then map your own interruption policy onto the same turn boundary. For the exact SDK and session details, see the docs.