Introduction

Natural barge-in is what makes a realtime avatar feel like a participant in the conversation instead of a scripted animation. The user starts talking, the system detects it quickly, speech synthesis stops cleanly, the avatar stops “holding the floor,” and the face transitions without a visible glitch. That sounds simple until you wire together three separate clocks: streaming STT, streaming TTS, and video animation.

This post covers the system design patterns that make barge-in work in practice. By the end, you should be able to reason about turn-taking state, coordinate partial transcripts with audio playback, and avoid the usual failure modes: double-speaking, late interruptions, lip-sync drift, and avatars that keep mouthing words after the user already cut in.

Start with a single turn-state machine

The biggest mistake is treating STT, TTS, and animation as independent subsystems. They are not. They are three views of the same conversation turn, and they need a shared state machine.

At minimum, model four states:

• Idle — listening, no agent speech in progress.

• Speaking — TTS is streaming and the avatar is rendering speech animation.

• Interrupted — user speech has been detected during agent speech, and you are stopping output.

• Listening-After-Interrupt — the system has yielded the floor and should prefer the user until intent is clear.

Do not let the avatar animation infer state directly from audio amplitude. Audio playback, animation, and speech generation all lag slightly and differently. Instead, drive them from explicit events:

• user_speech_started from VAD or STT endpointing.

• assistant_tts_started when the first audio chunk is committed.

• assistant_tts_stopped when the stream is canceled or naturally ends.

• avatar_speaking_started and avatar_speaking_stopped for UI and diagnostics.

Internally, keep a monotonically increasing turn_id or generation_id. Every streaming response, audio chunk, and animation command should carry that ID. When a barge-in happens, you cancel only the current generation. Late packets from the canceled generation should be ignored, not reconciled.

Detect user intent to barge in early, but confirm it conservatively

For natural interruption, “early” matters more than “perfect.” If you wait for final STT, the user will already have been talking over the avatar for hundreds of milliseconds. If you react to every acoustic blip, you’ll interrupt on coughs and keyboard noise.

The practical approach is two-stage:

1. Fast trigger from VAD or low-latency partial STT when speech energy crosses a threshold.

2. Commit trigger when the partial transcript or speech segment is stable enough to treat as intentional input.

In other words, stop the agent quickly, but only hand the turn to the user once the signal is credible. A common policy is:

• If the agent is speaking and VAD sees sustained user speech for a short window, pause/cancel TTS immediately.

• If the user’s audio ends before any meaningful transcript emerges, treat it as a false start and remain in listening mode.

• If the partial transcript looks like a clear interruption (“wait”, “actually”, “no”), commit aggressively.

The threshold depends on your domain. A customer-support bot should probably yield faster than a game NPC. The key is to make the policy explicit and testable rather than burying it inside whichever library happens to expose the audio stream.

Cancel TTS, don’t just “mute” it

When a barge-in occurs, the correct action is usually to cancel the TTS generation and drop any buffered audio that has not yet reached the client. Muting the output at the UI layer prevents the user from hearing more speech, but the rest of the pipeline still believes the assistant is talking. That leads to mismatched lip motion, stale transcripts, and awkward recovery.

Cancellation should propagate through three layers:

• TTS request — stop synthesizing new audio chunks.

• Audio transport — flush or discard queued frames for the canceled turn.

• Avatar animation — transition the mouth and facial expression to an idle/listening pose.

One subtlety: even after cancellation, the client may have already received a few audio frames. That is normal. Design the renderer to fade or snap to idle over a few frames rather than trying to “finish” the old utterance. A short transition looks cleaner than pretending the canceled speech never happened.

from livekit import rtc

from livekit import rtc

from livekit import rtc

Keep lip-sync tied to audio, not text

For avatars, “barge-in” is not just about stopping words; it is about stopping mouth motion in a way that matches the audio timeline. If animation is driven from text tokens alone, the avatar will continue to shape syllables even after audio is canceled. If it is driven from coarse speaker state alone, mouth motion can become detached from actual phonemes.

Better systems derive animation from the same audio stream used for playback. The avatar renderer needs a notion of:

• Speaking onset — first audible frame.

• Active speech — ongoing frames and visemes.

• Speech stop — either natural end or forced interruption.

For barge-in, the important edge case is forced stop. If you terminate audio mid-phoneme, the avatar should not hold a half-open mouth for a full second. A good renderer will blend from the current viseme into a neutral or listening pose within a small, bounded number of frames.

Also, do not make the animation pipeline wait for final STT. You want the avatar to begin speaking as soon as the audio starts, and you want it to stop as soon as the assistant output is canceled. The transcript is for cognition; the waveform is for motion.

Handle race conditions explicitly

Realtime systems fail at the boundaries: the user starts speaking exactly as the assistant ends, STT partials arrive after cancellation, or the avatar client receives stale audio after a new turn has begun. These are not edge cases; they are the normal operating environment.

There are three races you should expect:

1. Speech-start race — user speech is detected while the assistant is just about to finish.

2. Late-partial race — a transcript fragment for the canceled turn arrives after the cancel signal.

3. Overlapping-turn race — a new assistant response begins before all old frames have drained.

The fix is versioning. Every event and every stream chunk should be associated with the current turn generation, and consumers should reject anything older than the active generation. If you are using WebRTC or a streaming transport, assume out-of-order delivery is possible and code defensively.

A useful rule: the system may transition from Speaking to Interrupted only once, and only the latest generation can transition back to Speaking. This avoids “zombie speech” where an old TTS request reasserts itself after cancellation.

Practical integration pattern with a LiveKit voice agent

If you are already running a LiveKit-based voice agent, the cleanest place to implement barge-in is in the agent layer, before audio reaches the avatar. The LiveKit plugin for Protoface is designed for that wiring: the agent owns conversational state, and the avatar follows along as a synchronized speaking face. See the repository for examples and integration notes: https://github.com/protoface-ai/protoface-plugin-pipecat.

The implementation pattern is straightforward:

1. The agent starts a TTS stream for the current turn.

2. The avatar begins animating from the first audio frames.

3. VAD or partial STT detects user speech.

4. The agent cancels the active TTS turn and notifies the avatar layer to stop speaking.

5. The agent enters a brief listening-after-interrupt state and waits for a stable user utterance.

In practice, the avatar should not own interruption policy. It should be a consumer of turn-state events. That separation keeps the logic testable and lets you swap STT or TTS providers without rewriting the avatar behavior.

import asyncio

import asyncio

import asyncio

Where Protoface fits

This is the exact kind of coordination that Protoface is meant to absorb for developers: the avatar side of a realtime voice agent without forcing you to build the lip-sync and session plumbing yourself. Depending on your stack, you can integrate through the LiveKit plugin, the REST API for session management, or the Python SDK for programmatic control. The details live in the docs, but the important architectural point is that your agent still owns turn-taking; the avatar consumes synchronized events. For API shapes and current fields, use the official docs: https://docs.protoface.com.

import os

import os

import os

Testing and operational guardrails

Natural barge-in is hard to reason about by inspection, so measure it. Three metrics matter most:

• Interruption latency — time from user speech onset to assistant cancellation.

• Stale-audio leakage — how much canceled speech still reaches the client.

• Recovery time — how quickly the system returns to a stable listening state after interruption.

Log the active turn ID, VAD onset, STT partial arrival, TTS cancel, and avatar stop timestamps. When users complain that the avatar “talks over me,” those five timestamps usually tell you which layer is late.

Also test with real-world noise: keyboard taps, TV audio, overlapping speakers, and users who begin with fillers like “um” or “wait.” A barge-in policy that looks perfect on clean recordings often falls apart in a busy office.

Conclusion

Natural barge-in is mostly a coordination problem. The core design is simple: one shared turn state, early user-speech detection, hard cancellation of the active TTS generation, and animation that follows the audio timeline instead of guessing from text. Once you make generation IDs and explicit state transitions first-class, the common race conditions become manageable rather than mysterious.

If you are building a realtime voice agent with a face, start by implementing the state machine and cancellation semantics in your agent layer, then plug the avatar renderer into those events. For integration details, examples, and current API shapes, check the docs and the relevant quickstart or plugin repository. That will save you a lot of time compared with trying to retrofit barge-in after the system is already speaking in production.