Introduction

In realtime voice systems, “barge-in” is the difference between a conversational agent and a monologue machine. If the user starts speaking while the agent is still talking, you need to detect that interruption quickly, stop any in-flight synthesis or playback, and hand control back to the user without creating overlap, echo, or state confusion.

This post walks through the mechanics of interruption handling for voice agents that stream audio and drive a synchronized avatar. By the end, you should be able to reason about where voice activity detection (VAD) fits in the pipeline, how to wire barge-in to your turn-taking logic, and what to watch for when the agent also has to animate a talking face.

What barge-in actually means

At a protocol level, barge-in is just an interruption policy:

• The system is currently in an agent-speaking state.

• The user begins speaking before the agent finishes.

• The system detects the user’s speech quickly enough to treat it as a new turn.

• Any ongoing agent output is canceled or faded out, and the new user utterance becomes the active input.

The hard part is not the cancellation itself. It is deciding when to cancel and what signals to trust. In a typical WebRTC or media-streaming setup, you may have:

• mic audio arriving as a real-time stream,

• ASR running on partial audio frames,

• TTS streaming synthesized speech back to the client, and

• an avatar or lip-sync renderer consuming the same outbound audio timing.

If you wait for final ASR results, your interruption handling will feel sluggish. If you trigger on raw energy alone, you’ll cancel on coughs, keyboard noise, or the agent’s own audio leaking back through the mic. Good barge-in typically uses VAD as a fast front-end signal, then layers in a small amount of debouncing and state awareness.

Use VAD as a turn-taking signal, not as the whole decision

Voice activity detection answers a narrow question: “Does this frame look like speech?” It does not know whether the speech is the user, the agent audio echoing into the microphone, or some transient burst of noise. That distinction matters.

A practical interruption policy usually combines three checks:

1. Speech probability: the VAD score crosses a threshold for enough consecutive frames.

2. Context: the system is currently in an agent-speaking or TTS-streaming state.

3. Stability: the signal stays above threshold long enough to avoid canceling on a single frame spike.

This is why most production systems do not switch turns on the first positive frame. A common pattern is to require a short “speech onset” window, often tens of milliseconds, before declaring barge-in. That delay is small enough to feel immediate, but large enough to filter a lot of garbage.

Design the interruption state machine first

If you implement barge-in ad hoc, you’ll end up with race conditions between audio playback, ASR, and avatar animation. A simple state machine makes the behavior explicit:

• Idle: waiting for user input.

• Listening: user audio is active, ASR may be producing partials.

• Thinking: model is generating a response.

• Speaking: TTS/audio is streaming to the client; avatar mouth movement is following the same timeline.

• Interrupted: agent output has been canceled and the system is returning to Listening.

In practice, the important transition is Speaking → Interrupted. When VAD fires during Speaking, you want to:

1. stop sending additional TTS chunks,

2. cancel any pending synthesis or playback buffers,

3. freeze or reset avatar motion tied to the outgoing utterance, and

4. accept the new user audio as the start of the next turn.

The cancellation path should be idempotent. You do not want repeated VAD frames to trigger repeated cancels, nor do you want a late TTS chunk to restart playback after the user has already taken the floor.

Practical VAD tuning for barge-in

There are a few tuning choices that matter more than the specific model behind the VAD:

• Window size: shorter windows reduce latency but are noisier. You usually want a low-latency frame size for onset detection.

• Threshold: lower thresholds catch quiet speech sooner, but increase false positives.

• Hangover/debounce: keep speech “active” for a short tail after the last positive frame so you do not flap between states.

• Echo handling: if the agent audio can leak into the microphone, either use acoustic echo cancellation or enforce a stricter barge-in threshold while speaking.

One useful rule: tune for interruption latency before tuning for recall. Users generally forgive the agent missing a very soft utterance once in a while, but they do not forgive the agent continuing to talk over them.

Implementation sketch: cancel speech on user onset

Below is a simplified Python sketch of the control flow. The exact SDK fields and event names depend on your stack, but the shape is what matters.

from asyncio import Event

from asyncio import Event

from asyncio import Event

In a real agent, this signal usually fans out to multiple subsystems:

• the TTS stream gets aborted,

• the player drops queued audio frames,

• the dialog manager marks the response as interrupted, and

• the avatar renderer is told to stop animating the outgoing sentence.

The most important constraint is that the interruption decision should be made upstream of the audio sink. If you wait until the last stage to suppress output, you can still end up with audible overlap or a visible “mouth keeps moving after the user interrupts” artifact.

Integrating with a realtime avatar pipeline

With a talking avatar, interruption handling is not just about audio. The avatar’s motion needs to track the same turn state as the audio pipeline. If the user barges in and you stop playback, the avatar should stop lip-syncing immediately rather than finishing the last phoneme of the canceled response.

That means the avatar layer should consume the same turn events as the media layer:

• start speaking when the first outbound audio frame is committed,

• continue animating while audio is buffered and playing,

• stop or blend out motion on cancel, and

• restart cleanly on the next agent response.

If your video face is generated from the same response timeline as your TTS, interruption handling is straightforward: cancel the response timeline and the avatar naturally follows. If the avatar is driven by a separate animation clock, you need explicit synchronization so it does not drift from the audio state.

How Protoface fits in

Protoface is useful here because it gives you a synced talking avatar surface that can be attached to an existing voice-agent stack, including LiveKit-based agents. For a LiveKit voice agent, the plugin path is the cleanest place to integrate interruption handling: when your agent state transitions from speaking to interrupted, you cancel the voice output as usual and the avatar stays in lockstep because it is tied to the same session stream.

If you are wiring this into a Python agent, the flow is the same regardless of vendor: your code owns the state machine, and the avatar layer consumes the resulting turn events. The Protoface docs have the exact integration details and SDK/API shapes.

# Illustrative only: exact class names and session fields are in the docs

# Illustrative only: exact class names and session fields are in the docs

# Illustrative only: exact class names and session fields are in the docs

For developers using LiveKit directly, the relevant plugin examples are in the GitHub repo linked from the quickstart materials, and the broader API behavior is documented at docs.protoface.com.

Common failure modes

Three issues show up repeatedly in production:

• False barge-ins from echo: the agent hears itself. Fix with echo cancellation, stricter thresholds during speech, or a short barge-in holdoff when playback begins.

• Late cancel: the user has already started speaking, but the agent keeps talking for another second because cancellation is waiting on final ASR. Fix by triggering on VAD onset, not final transcript arrival.

• State desync: the audio stops but the avatar keeps moving, or vice versa. Fix by making one turn-state source of truth and fanning out events from it.

Also remember that not every user noise should interrupt. Background speech in a shared office, keyboard taps, or a half-spoken utterance that never continues should usually be ignored. That is why debounce and state awareness are not optional details; they are the difference between a usable conversational UI and a flaky one.

Conclusion

Barge-in is mostly an engineering problem of state synchronization. Use VAD to detect speech onset quickly, feed that signal into a simple turn-state machine, and cancel both audio output and avatar animation from the same interruption event. Keep the detection threshold and debounce tuned for low latency, and treat echo as a first-class failure mode.

If you are integrating a realtime avatar into an existing voice agent, the main work is making sure the avatar follows your conversation state exactly. The docs at docs.protoface.com cover the platform surfaces and integration details, and the quickstarts are a good reference point if you want to compare this flow against a working LiveKit-based setup.