How to Prevent Talk-Over in Realtime Avatar Agents: Architecture Patterns for Interrupt Detection
Prevent talk-over in realtime avatar agents with VAD, ASR partials, turn IDs, and end-to-end cancellation.
Introduction
Talk-over is the failure mode that makes realtime avatar agents feel sloppy: the model keeps speaking while the user is already speaking, then both sides collide, the avatar lip-syncs through an interruption, and the conversation loses turn-taking. In a plain voice agent, this is mostly an audio problem. In an avatar agent, it becomes an audio + video synchronization problem, because you need to stop not just text generation or TTS, but the avatar render pipeline as well.
By the end of this post, you should be able to reason about where interrupt detection belongs in a realtime agent stack, how to avoid race conditions between ASR, LLM, TTS, and avatar playback, and how to structure your system so user barge-in feels immediate instead of laggy.
Start with the actual conversation loop
At a high level, a realtime agent usually has four moving parts:
Input capture: microphone audio, WebRTC track, or streamed audio chunks.
Speech detection / ASR: detects speech activity and produces partial/final transcripts.
Response generation: LLM or agent logic decides whether to speak and what to say.
Output playback: TTS audio, streamed to the client, often with avatar lip sync tied to the audio timeline.
Interrupt detection sits across these stages, not inside just one of them. The simplest mistake is to treat “user started speaking” as a boolean that only gates LLM generation. That is not enough. You also need to stop or fade out audio playback, cancel pending TTS synthesis, invalidate any queued avatar animation, and mark the current turn as aborted so downstream code does not flush stale audio later.
In practice, the cleanest mental model is a small state machine:
Once the agent is responding, any high-confidence user speech event should transition the turn to Interrupted. From there, your system should aggressively cancel work in flight and return to Listening. The important part is that this state change needs to be atomic from the perspective of your app logic, even if the underlying media pipeline takes a few hundred milliseconds to drain.
Detect interruption from speech activity, not from transcripts alone
For talk-over prevention, waiting for a final transcript is too late. You want a barge-in signal derived from voice activity or streaming ASR partials, ideally with a short debounce window. The exact threshold depends on your domain, but the pattern is consistent:
While the agent is speaking, continue listening to the inbound audio track.
If the user’s voice activity exceeds a minimum duration or energy threshold, mark the current assistant turn as interrupted.
Cancel pending generation and playback immediately.
Only resume after the interruption is acknowledged by the client state and output pipeline.
There are two common sources of interruption signals:
VAD events: low latency, good for barge-in, but can false-trigger on noise.
Streaming ASR partials: more semantically robust, but generally slower and sometimes too late to prevent audible overlap.
The right answer is usually both. Use VAD for fast interruption detection and ASR partials to confirm that the speech is real and to extract the user’s intent. If your app works in noisy environments, add a short hold-off window or a two-stage threshold so a cough does not nuke the agent’s turn.
A subtle but important implementation detail: do not reset the barge-in detector when the avatar starts talking. The detector should stay live for the entire response window, because the user can interrupt at any time. The agent is never “done listening” just because it is producing output.
Make cancellation propagate through every layer
Interrupt detection only works if cancellation is end-to-end. In a typical implementation, you need to cancel four things:
LLM request: stop generating more text tokens.
TTS job: stop synthesizing remaining audio.
Playback buffer: flush queued audio frames so the client stops hearing the old turn.
Avatar animation: stop lip sync and any motion tied to the aborted utterance.
If you only cancel the LLM but let synthesized audio continue, the user still hears talk-over. If you cancel audio but leave the avatar animation running, the face keeps “speaking” after the sound is gone, which is just as distracting.
A good implementation uses a per-turn cancellation token or abort controller. Every async task spawned for that response receives the token and checks it frequently. When the interruption event fires, you flip the token once and treat all later results as stale.
The precise API will vary, but the principle is the same: interruption is a turn-level concern, not a function-level concern. Avoid nested tasks that each manage their own idea of cancellation. That is how stale audio sneaks through after the state has already changed.
Debounce, confidence, and race conditions
Most talk-over bugs are race conditions disguised as product issues. The user starts talking right as the assistant is finishing a sentence, and your system has to decide whether the overlap is a true interruption or merely conversational backchannel. You can make this robust with three tactics:
Debounce speech onset: require a short continuous speech window before interrupting, usually on the order of tens of milliseconds.
Respect turn-finalization: once the assistant has already finished and playback is drained, ignore late user speech as a new turn rather than an interruption.
Use monotonic turn IDs: tag every generated chunk with the turn it belongs to so late arrivals can be dropped safely.
Turn IDs matter more than they first appear. Imagine the assistant has already been interrupted, but a delayed TTS chunk from the previous response arrives after the new turn starts. If you do not check the turn ID before sending that chunk to the client, you will resurrect the old response and create a phantom overlap.
Another practical edge case: if your avatar client renders video and audio separately, you must coordinate the stop signal across both streams. Audio may cut instantly, while the video renderer may still have a small buffered window. That is fine as long as the client treats “interrupt” as a synchronization event and flushes both pipelines together.
One useful architecture pattern: a central turn controller
The most maintainable pattern is to put all turn lifecycle logic behind a single controller object or service. It owns:
the current turn ID
the cancellation token
the speaking/listening state
the barge-in detector wiring
the rules for when a new turn may begin
This keeps interrupt handling consistent whether the source is a browser mic, a telephony bridge, or a LiveKit track. It also makes testing easier, because you can unit-test the state transitions without spinning up a full realtime stack.
A good test matrix includes:
user interrupts before any audio is played
user interrupts during TTS synthesis
user interrupts during buffered playback
late transcript arrives after cancellation
noise triggers VAD but no real speech follows
If your code passes those cases, it is usually ready for real users.
Where Protoface fits
This is exactly the kind of turn coordination that a developer-facing avatar layer should help with. In a LiveKit voice agent, the Pipecat integration and the Protoface plugin are the natural place to wire avatar playback to the agent’s speech lifecycle so lip sync stays aligned with the active turn. If you are building directly against the service, the REST API and Python SDK give you programmatic control over avatars and realtime sessions; the exact request fields and session parameters are documented at docs.protoface.com.
For example, if you are orchestrating sessions yourself, the pattern is straightforward: create a session, attach the avatar, and make sure your application treats interruption as a first-class session event, not a UI afterthought. Exact payloads vary, but the shape looks like this:
The useful part is not the endpoint itself; it is the boundary it creates. Your agent owns turn state and interrupt detection, while the avatar layer consumes only the currently valid turn.
Conclusion
Preventing talk-over is mostly about disciplined turn management. Detect user speech early, cancel every downstream task for the active turn, tag all generated media with turn IDs, and synchronize audio and avatar rendering around a single state transition. If you do that, interruptions become natural instead of glitchy.
For implementation details, integration examples, and current API shapes, start with docs.protoface.com, then use the relevant quickstart or plugin repo for your stack. If you are already shipping a realtime voice agent, the next step is simple: add a turn controller, instrument interruption events, and test the edge cases before users find them for you.