Introduction

Interruption handling is one of the hardest parts of building a conversational video agent. Audio-only agents can often get away with a simple turn-taking model: user speaks, ASR finalizes, LLM responds, TTS plays. Once you add realtime video lip-sync, interruption becomes a coordination problem across three streams at once: microphone audio, synthesized speech, and animated face output.

That coordination matters because users interrupt for normal reasons. They correct the agent mid-sentence, change topics, or speak over a long response when the answer is already clear. If your stack doesn’t handle barging-in cleanly, you get the classic failures: stale audio keeps playing after the user starts talking, the avatar keeps mouthing words that no longer match the conversation, or the model generates a response to something the user never intended to finish saying.

This post breaks down how to think about interruption handling in realtime conversational agents with lip-synced video. By the end, you should be able to design a turn model that cleanly stops output, ignores stale generations, and resumes with correct state across voice and video.

What “interruption” actually means in a realtime agent

In practice, “interruption” is not a single event. It usually means one of these cases:

• User barge-in: the user starts speaking while the agent is still talking.

• User correction: the user says “wait” or “no” and wants the current response abandoned.

• Agent-side preemption: a higher-priority event arrives, such as a tool result or system event that should replace an in-flight answer.

For audio/video agents, the key is that interruption has to propagate through the entire output pipeline. Stopping TTS alone is not enough. You also need to stop or invalidate any queued video frames, clear local playback buffers, and prevent the next LLM/TTS turn from being generated from stale context.

A good mental model is: turns are cancellable jobs, not irreversible outputs. Once the user barges in, the current assistant turn should be treated as aborted, even if a few packets are already in flight.

The state machine you actually need

Most robust implementations end up with a small state machine per conversation participant, even if it’s implicit in the code:

1. Idle — no one is speaking.

2. UserSpeaking — microphone activity or VAD indicates the user has taken the floor.

3. AgentSpeaking — TTS is streaming and the avatar is animating.

4. Interrupted — agent output was cancelled because user speech arrived.

The important edge is the transition from AgentSpeaking to Interrupted. That transition should do three things immediately:

• cancel the current synthesis/streaming task,

• discard any queued audio/video frames not yet rendered, and

• mark the turn generation as stale so late-arriving chunks are ignored.

If you only cancel the async task but leave already-buffered audio in the player, the user still hears the agent finish the sentence. If you only stop audio but leave the video pipeline running, the avatar keeps lip-syncing to silence or, worse, to old text. The two streams need to be bound to the same turn identifier.

Designing for cancellation, not just stop

The most common implementation mistake is treating interruption as a UI action instead of a distributed systems problem. In a realtime agent, a single assistant response may involve:

• streaming tokens from an LLM,

• incremental TTS synthesis,

• packetized audio delivery over WebRTC,

• avatar frame generation or face animation, and

• browser or client-side playback buffers.

Each stage can lag independently. So interruption should be modeled as a cancellation token or generation ID that every stage checks before emitting more output. The moment a new user utterance is detected, you increment the active turn ID. Anything still holding the previous ID becomes stale and should stop emitting.

This pattern is better than trying to “flush everything” because flush semantics vary by layer. Some buffers can be cleared immediately; others can only be drained; some WebRTC transports will still deliver a few in-flight packets. If your downstream consumers are generation-aware, a few late packets are harmless—they get dropped.

VAD, end-of-turn detection, and barge-in thresholds

Interruption handling starts with detecting that the user is speaking again. In production, that usually combines voice activity detection with an end-of-turn heuristic. You want to avoid false positives from background noise, but you also want to react quickly enough that the agent doesn’t talk over the user for another second.

Two practical rules help:

1. Use a short preemption window. If you wait for perfect certainty, your barge-in feels sluggish. Most systems accept a small number of false interrupts in exchange for responsiveness.

2. Prefer retriggerable detection. Brief silence should not immediately finalize the turn if the user is still clearly engaged. A stable utterance boundary is more useful than raw silence duration.

For lip-synced agents, the interruption signal should not merely stop future output. It should also reset any animation state tied to the current phoneme sequence. Otherwise the avatar can visibly finish “speaking” words after the user has already started talking. The user experience gets uncanny very quickly.

Practical implementation pattern

Here is a compact Python sketch of the core idea: a turn ID protects the agent from stale output. This is illustrative; exact SDK fields and method names depend on your stack.

import asyncio

await speak(agent, response)

import asyncio

await speak(agent, response)

import asyncio

await speak(agent, response)

The details vary, but the structure should be familiar: one increment to invalidate the old turn, one cancellation call to stop the current stream, and generation gating on the active turn ID. If your stack has separate audio and video emitters, both need to consult the same invalidation source.

What to do when the user interrupts mid-word

Mid-word interruption is where implementation details become user-visible. There are three common choices:

• Hard cut: stop audio immediately on barge-in.

• Soft fade: ramp volume down over a few tens of milliseconds.

• Natural completion: let the current phoneme finish, then stop.

Hard cut gives the fastest response but can sound abrupt. Soft fade is usually the best default for speech, especially if your TTS is loud or the user interrupt is likely to be intentional. Natural completion feels smoother but delays the handoff, which can frustrate users in task-oriented flows.

The video side should generally stop as soon as the user takes the floor. Continuing lip motion after the user starts speaking is more jarring than abruptly stopping audio. If you need a graceful visual transition, hold a neutral face rather than trying to “finish” the previous sentence.

One useful integration point

If you are already using LiveKit Agents, the cleanest place to solve this is in the voice-agent layer, because that is where turn ownership and cancellation already live. The quickstart examples are useful reference material for how a realtime voice pipeline is structured, and the live avatar plugin wires the face into that same turn stream. With that setup, interruption handling becomes a single concern: cancel the active assistant turn and let the avatar follow the same lifecycle as the audio.

For developers working directly with the avatar/session control plane, the REST API and Python SDK are also useful when you need to create sessions, manage avatars, or build your own orchestration around turn state. The main operational point is the same either way: treat output as cancellable, and make sure stale generations cannot leak through after a barge-in.

Here is a minimal API example to show the shape of the interaction. Exact payload fields are documented in the API reference.

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","quality_tier":"standard"}'

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","quality_tier":"standard"}'

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","quality_tier":"standard"}'

And a similarly compact Python sketch:

from protoface import Client

session = client.sessions.create(avatar_id="avt_123", quality_tier="standard")

from protoface import Client

session = client.sessions.create(avatar_id="avt_123", quality_tier="standard")

from protoface import Client

session = client.sessions.create(avatar_id="avt_123", quality_tier="standard")

Use these as orchestration primitives, not as a substitute for cancellation logic. The hard part is still making sure your realtime pipeline can preempt itself cleanly.

Debugging the failure modes

When interruption handling is wrong, the bugs are usually obvious to users but subtle in logs. A few symptoms to watch for:

• Late audio after barge-in: usually caused by buffered TTS chunks not being invalidated.

• Avatar keeps talking after silence: video generation is not linked to the current turn ID.

• Agent answers the wrong thing: the interrupted transcript was still promoted to final text.

• Double responses: both the aborted and the replacement turn were allowed to continue.

Instrument your system with turn IDs in logs and traces. When a user interrupts, you should be able to answer three questions quickly: when did the interruption arrive, which turn was invalidated, and which downstream components actually stopped.

Conclusion

Clean interruption handling is mostly about discipline: treat every assistant response as cancellable, propagate a single turn identity through ASR, LLM, TTS, and avatar output, and stop both audio and video as soon as the user takes the floor. If you do that, realtime lip-sync stops being fragile and starts behaving like a normal interactive system.

If you want implementation details, API shapes, and integration examples, start with the docs at docs.protoface.com. If you’re wiring this into a voice agent stack, the key is to test barge-in early, with real audio and real playback buffers, not just mocked async calls. That’s where the edge cases show up.