How to Handle User Interruptions, Turn-Taking, and Audio Sync in a Vue 3 SDR Avatar
Vue 3 avatar turn-taking: barge-in handling, state machines, and audio-clock lip sync for realtime SDR voice apps
Introduction
Realtime avatars are easy to demo and surprisingly hard to make feel natural. The hard parts are not the video renderer or the model output; they are the conversational mechanics around it: when the user interrupts, when the agent should stop talking, how to avoid talking over the user, and how to keep lip sync aligned with audio without visible stalls or “ghost speech.”
This post is about the software engineering side of that problem in a Vue 3 application using an SDR-style avatar feed: you want the UI to reflect a live conversational state machine, not just a video element that happens to play audio. By the end, you should have a clear mental model for handling turn-taking, interrupting cleanly, and keeping audio/video synchronized enough that the avatar feels responsive rather than robotic.
Start with a state machine, not with media playback
If you only track “playing” and “not playing,” your UI will drift out of sync with reality. Real-time conversation has at least four useful states:
Idle: no active turn, waiting for user or agent.
User speaking: mic is active and the agent should generally yield.
Agent speaking: avatar is rendering speech and audio.
Interrupted / barge-in: user started speaking while the agent was mid-turn, so the agent must stop quickly and cleanly.
In practice, the state machine is driven by events from your voice pipeline: VAD start/stop, ASR partials/finals, agent TTS start/stop, and transport-level disconnects. The browser should not guess based on audio element state alone. It should react to explicit control events from your realtime backend or agent runtime.
A good rule: the browser owns presentation, but the conversation server owns turn authority. The server decides whether the agent is allowed to continue. The client reflects that decision and only performs local actions such as muting playback, fading out audio, or updating the avatar’s speaking animation.
Handle user interruptions explicitly
“Barge-in” is the standard name for user interruption in voice UX. The implementation usually has two parts:
Detect the interruption using voice activity detection or ASR onset.
Cancel the agent turn by signaling the speech pipeline to stop generating, stopping audio playback, and clearing any queued video frames tied to that utterance.
Do not wait for the current sentence to finish. If the user speaks, latency matters more than perfect truncation. A 150–300 ms cancel path usually feels acceptable; anything longer starts to feel like the agent is talking over the user.
In Vue, keep this logic outside component templates. Put it in a composable or store so the state transition is explicit and testable:
The important detail is that interruption is a transition, not a side effect. Once you treat it that way, you can debounce duplicate VAD events, suppress late-arriving audio chunks, and make your UI deterministic under packet loss or jitter.
Turn-taking is about sequencing, not just detection
Good conversational turn-taking is mostly about ordering events correctly under imperfect network conditions. In a browser-based avatar client, events often arrive in this order:
User stops speaking.
ASR final arrives.
Agent thinks.
TTS starts streaming.
Audio frames and video frames arrive independently.
That sequence is idealized. In reality, partial ASR updates may arrive late, TTS start may be signaled before the first audible packet, and video frame timing may not perfectly track audio packet timing. Your client should tolerate all of that.
Two implementation details matter a lot:
Monotonic turn identifiers: assign a turn ID or sequence number to every agent response. Drop any audio or video belonging to stale turns.
Single writer semantics: only one active turn should be able to drive avatar speech at a time. If a new turn starts, cancel the previous one before rendering the new one.
Without this, it is easy to create a race where an old TTS packet arrives after a newer user interruption and briefly resurrects the avatar mouth movement. That kind of bug looks small in logs and very obvious on screen.
Keep audio sync anchored to the audio clock
For lip sync, the audio clock is the source of truth. Video should follow audio, not the other way around. In a real-time avatar, the mouth animation usually comes from either viseme timing, phoneme timing, or a learned speech-to-face renderer. However the model works, the browser still has to present frames in a way that stays aligned with the actual playout position of the audio.
Three practical rules help:
Buffer a little, but not too much. A small jitter buffer smooths network variance. Too much buffer makes the avatar feel delayed and hurts turn-taking.
Use one playout timeline. Tie video frame selection to the current audio playout timestamp rather than to wall-clock arrival time.
Drop stale frames aggressively. If the browser is behind, skip ahead. It is better to be slightly under-rendered than visibly out of sync.
In practice, you will often receive audio and video as separate realtime streams or as coordinated frame updates. If the render path is in Vue, avoid doing per-frame reactive updates that trigger full component re-renders. Push timing-sensitive work into requestAnimationFrame, a dedicated media controller, or a lightweight store that updates only the avatar surface.
A common mistake is to bind the <video> element directly to state that changes on every frame. That works in a toy demo and falls apart under load. Let the video element handle decoded media playback, and let Vue manage higher-level state such as speaking, listening, muted, interrupted, and reconnecting.
Make interruption and sync bugs observable
These bugs are hard to fix if you cannot see the event timeline. Add logging for:
VAD start/stop timestamps
ASR partial and final timestamps
TTS start, first audio packet, last audio packet
avatar speaking start/stop
interrupt cancel requests and acknowledgements
When debugging, you are looking for gaps and inversions: user speech started before cancel was sent, audio packets continued after cancel was acknowledged, or the video kept animating after audio stopped. Those are usually sequencing bugs, not rendering bugs.
Also watch for browser audio policy issues. If the session relies on autoplay, the first user interaction may be required before playback can begin. Handle that up front with a clear “tap to enable audio” path rather than trying to recover mid-conversation.
How Protoface fits into this
Protoface is useful here because it gives you the avatar side of the pipeline without forcing you to invent the entire media stack yourself. If you are already running a voice agent, the LiveKit integration is the most direct path: the plugin drops a synchronized talking face into the agent so your turn-taking logic can focus on conversation control rather than custom lip-sync plumbing. The same system also exposes a REST API and Python SDK for creating avatars and sessions programmatically; exact request fields and lifecycle details are in the docs.
For example, the REST surface is straightforward to automate from backend code:
And if you are wiring a LiveKit voice agent, the plugin path keeps the avatar coupled to the agent’s speaking state so your barge-in handling stays aligned with the same turn boundaries you already use for audio.
Practical Vue 3 integration pattern
In a Vue 3 app, the cleanest integration is usually:
Subscribe to conversation events from your realtime backend.
Store a small, explicit conversation state in Pinia or a composable.
Drive the avatar UI from that state only.
Keep media transport and timing in a separate controller object.
That separation keeps the UI predictable. The component can render a talking indicator, an interrupt button, and connection status, while the controller handles cancellation, buffering, and stale-frame dropping.
A minimal interface often looks like this:
That is enough structure to prevent most of the visible glitches: overlapped speech, delayed stops, and mouth movement from stale turns.
Conclusion
For realtime avatars, the difficult part is not rendering a face; it is managing conversation timing under jitter, interruption, and asynchronous media delivery. Model the interaction as a state machine, treat barge-in as a first-class transition, anchor lip sync to audio playout, and drop stale frames aggressively. If you do those four things, the avatar will feel much more responsive and much less fragile.
If you want a reference implementation or integration details for the REST API, Python SDK, or LiveKit plugin, start with the docs at docs.protoface.com and the relevant examples in the GitHub repos linked there. The quickest way to validate the ideas in this post is to wire up a small demo, add turn-state logging, and then test interruption timing with real network jitter instead of localhost.