Introduction

Mid-sentence interruptions are one of the first places a realtime avatar system feels “off.” The user starts speaking, the model is answering, and then the user cuts in. If your stack keeps streaming TTS and lip-sync all the way through the interruption, the avatar will talk over the user, react late, and generally violate the conversational turn-taking that makes voice agents feel usable.

This post is about handling that case correctly in a WebRTC-based avatar app: detecting when the user barges in, stopping audio and animation quickly, and handing control back to the agent without leaving the media pipeline in a broken state. By the end, you should be able to reason about interruption handling in a realtime avatar app, implement the control flow in your agent, and know where the WebRTC, STT, and avatar layers each need to react.

What actually happens during an interruption

In a typical voice-agent pipeline, microphone audio flows to an STT service, transcript events feed an LLM or dialogue manager, and the resulting response goes to TTS. For an avatar, that TTS output also drives lip sync and facial animation over a WebRTC media stream.

An interruption is not just “the user spoke while the bot was speaking.” It is usually detected from a combination of signals:

• Voice activity: the user’s audio energy rises above threshold.

• Streaming STT partials: the transcript starts showing a new utterance before the agent has finished.

• Turn-taking policy: your application decides that a new user utterance should preempt the current assistant turn.

Once you decide the assistant has been interrupted, you need to stop three things in sync:

1. The remaining TTS audio generation or playback.

2. The avatar’s speaking animation and mouth cues tied to that audio.

3. The current assistant turn state in your agent so it doesn’t continue reasoning as if it still has the floor.

Those are separate layers. WebRTC can stop sending or playing media, but it does not magically cancel your LLM or STT pipeline. Conversely, canceling your agent task does not necessarily stop already-buffered audio from being rendered unless you explicitly tear it down.

Design the interruption path as a first-class state transition

The cleanest pattern is to model the agent as a small state machine with explicit turn ownership:

• Listening: no assistant audio is active, user has the floor.

• Speaking: assistant TTS and avatar animation are active.

• Interrupted: assistant was preempted and must stop immediately.

• Recovering: system is flushing buffers and preparing for the next user turn.

That sounds basic, but it prevents a lot of race conditions. In particular, interruption handling becomes reliable when the same event that marks user barge-in also cancels the assistant’s active task, clears any pending audio frames, and tells the avatar layer to stop speaking.

Detecting barge-in with STT and audio timing

For realtime systems, you usually want interruption detection before a final transcript is available. Waiting for the end of an utterance is too late. Streaming STT partials are the usual trigger because they arrive early enough to interrupt TTS while the user is still speaking.

A practical rule is:

• If the assistant is speaking and the user audio crosses a voice-activity threshold for more than a short debounce window, treat it as a barge-in candidate.

• If the streaming STT emits a partial that is not just noise or a false start, confirm the interruption.

• Cancel the assistant turn immediately; do not wait for the current sentence to finish.

The debounce window matters. Without it, keyboard clicks, room noise, or echo from the assistant’s own TTS can falsely trigger interruption. With WebRTC you also need to be aware of audio echo cancellation and server-side buffering; if you only look at raw energy, you may accidentally interpret your own output as user speech in some environments.

Stopping assistant output without breaking the media session

When an interruption is confirmed, the most common failure mode is to stop the text generation but leave the last audio chunk or animation cue playing. Users notice that immediately. The correct stop sequence is usually:

1. Mark the current assistant turn as canceled.

2. Stop or cancel TTS generation.

3. Flush any queued audio frames destined for the WebRTC sender or player.

4. Send an avatar state update to stop speaking / lip sync.

5. Resume listening for the user’s transcript.

If your architecture uses a worker or async task for generation, cancellation should be cooperative and idempotent. In practice that means the “stop speaking” action might be invoked more than once from slightly different signals: VAD, STT partials, UI controls, or a timeout. Make sure repeated stop calls are harmless.

Minimal agent-side control flow

Here is a simplified Python-style sketch of the interruption logic. Exact APIs differ by stack, but the control flow is the important part:

async def on_user_partial(text: str, is_speaking: bool):

await update_avatar_mouth(chunk)

async def on_user_partial(text: str, is_speaking: bool):

await update_avatar_mouth(chunk)

async def on_user_partial(text: str, is_speaking: bool):

await update_avatar_mouth(chunk)

The key detail is that the assistant turn checks state continuously. Once the state flips away from speaking, the task stops producing media. That is what keeps a late interruption from leaking through as a few extra frames of audio or lip movement.

WebRTC gotchas: media buffering and latency

With WebRTC, “stop” is not instantaneous from the user’s perspective because there is always some buffer in flight. That is normal. What matters is minimizing the tail.

Three practical points:

• Audio buffering: if your TTS output is packetized into small chunks, you can usually stop within one or two packets. Large chunks make interruption feel sluggish.

• Transport versus application state: pausing the media sender does not necessarily stop the application from generating more chunks. Cancel upstream first, then stop transport.

• Avatar sync: lip sync should follow the same cancellation boundary as audio, or the face will keep articulating after audio has stopped.

It is also worth handling the “resume” case explicitly. After an interruption, the user may continue speaking, or they may stop and expect the system to answer. Your agent should treat the interrupted turn as abandoned and start a new turn from the user’s latest utterance, rather than trying to splice the new input into the old assistant response.

Where Protoface fits

This is exactly the sort of turn-state problem a developer-facing avatar layer should stay out of while still exposing the right control hooks. In a LiveKit-based voice agent, the Pipecat Protoface integration and the LiveKit plugin can keep the avatar synchronized with the agent’s speaking state, while your application owns interruption policy and cancellation. The important part is that your agent can stop the current response cleanly and the avatar will follow that state transition instead of trying to guess what happened.

If you are building against the LiveKit plugin directly, the mental model is simple: treat the avatar as another realtime consumer of the assistant’s turn state. When the user barges in, cancel the assistant generation and stop the avatar speaking path in the same code path. If you are using the broader API surfaces, the same principle applies: the session should reflect the current conversational turn, and your app should be the source of truth for when a turn ends.

For implementation details, setup steps, and exact fields, see the documentation at docs.protoface.com and the relevant examples in the GitHub organization.

A practical interruption policy

There is no universal “correct” threshold for interruption. Good defaults depend on your use case:

• Support bots: interrupt aggressively. Users want the bot to yield quickly.

• Sales or demo assistants: use a slightly higher debounce to avoid false positives during planned monologues.

• Game NPCs: favor responsiveness, but keep the avatar motion stable so frequent stops do not look jittery.

Two implementation details are easy to miss:

• Do not let the assistant continue “thinking” in the background after an interruption unless you intend to reuse the partial reasoning. Otherwise you risk answering a question the user already moved past.

• Log interruption events separately from regular turn completions. That gives you a real signal for tuning VAD, STT partial thresholds, and TTS chunking.

Conclusion

Handling mid-sentence interruptions well means treating turn-taking as a coordinated state change across STT, agent logic, audio transport, and avatar animation. Detect barge-in early, cancel the active assistant turn immediately, flush any queued media, and keep the avatar synchronized with the same state transition. If you get that right, the system feels responsive instead of stubborn.

For concrete setup details and integration examples, start with the docs at docs.protoface.com and the relevant quickstarts in the Protoface GitHub organization.