Introduction

When a realtime avatar is speaking, there are really two states you care about: the audio pipeline and the visual speech state. In a VideoSDK-style interrupt flow, those states can drift apart very easily. A user barges in, the agent stops generating audio, but the avatar keeps “finishing” the last utterance for a few hundred milliseconds, or the face snaps to idle before the audio actually drains. Either failure mode feels broken.

This post is about keeping those states aligned after an interruption. By the end, you should be able to reason about where the mismatch comes from, what to reset, what to preserve, and how to design your agent loop so an interrupt produces a clean transition instead of a half-stopped voice and a half-speaking face.

Why interruptions cause desync

Realtime avatars are usually driven by some combination of:

• an audio stream or audio chunks being played to the user,

• a speech state machine that decides when the avatar is “speaking,” “listening,” or “idle,”

• lip-sync metadata derived from the text-to-speech output, phoneme timing, or audio energy,

• networked transport and buffering on the client.

An interruption can hit at any point in that chain. The user speaks, your VAD or agent logic decides to cancel the response, and now you need to unwind three layers consistently:

1. Stop the TTS or audio playback job.

2. Invalidate any queued lip-sync frames or timing derived from the canceled utterance.

3. Tell the avatar renderer that the current speaking turn is over immediately, even if the transport is still draining buffered audio.

The bug usually appears when you do only one of these. For example, canceling the LLM/TTS task does not automatically clear already-produced audio packets or animation timestamps on the client. Likewise, forcing the face to idle without stopping playback can make the avatar stop moving while speech continues for another beat.

Model the problem as a turn, not a stream

The cleanest way to keep things synchronized is to treat each agent response as a turn with an explicit lifecycle. The turn begins when you commit to speaking and ends when you either:

• finish playback normally, or

• receive an interruption and abort the turn.

That sounds obvious, but it matters because stream-oriented code tends to append audio and animation opportunistically. Turn-oriented code gives you one place to attach state: a turn ID, cancellation token, queue of pending audio frames, and the avatar’s speaking flag.

A useful internal contract looks like this:

turn_id
cancel_token: shared cancellation signal

turn_id
cancel_token: shared cancellation signal

turn_id
cancel_token: shared cancellation signal

On normal completion, you drain the buffer and transition to idle. On interruption, you cancel the token, drop any queued frames for that turn, and force the visual state to idle only after the transport acknowledges the stop. That last clause is important: do not let “idle” mean “we stopped generating.” It should mean “the user will not see or hear additional output from this turn.”

What to reset, and what to keep

Not everything should be wiped on interrupt. If you reset too aggressively, you get a jarring visual pop or lose conversational continuity. The practical split is:

Reset immediately:

• current audio generation task,

• queued but unplayed audio frames,

• lip-sync frames tied to the canceled utterance,

• “speaking” animation state for the canceled turn.

Preserve:

• the conversation history,

• the session-level voice or avatar configuration,

• the user’s interruption context, so the next reply can pick up naturally,

• connection/session metadata, unless the entire session is being torn down.

That separation matters because the next response should start from a clean slate visually, but not from a blank conversation state. In practice, the next turn should re-enter “speaking” only once the first new audio frame for that turn is actually ready to play. Don’t pre-arm the avatar based solely on the LLM starting to generate text.

Make the interrupt path idempotent

Interrupts are messy in real systems. You can get duplicate stop events from voice activity detection, a UI button, or transport reconnect logic. You can also receive a late audio chunk after cancellation if there is buffering in a worker or over WebRTC. So your stop logic must be idempotent.

That means your interrupt handler should be safe to call multiple times for the same turn. A good pattern is:

def interrupt_current_turn(state):

state.current_speaking = False

def interrupt_current_turn(state):

state.current_speaking = False

def interrupt_current_turn(state):

state.current_speaking = False

In a real implementation, the “transport confirms the stop” step may be explicit or implicit. With some stacks, you can flush or clear a playback queue. With others, you simply stop enqueueing and let the current packet drain. Either way, the rule is the same: the avatar’s visible speech state must follow the same stop condition as the audible state.

Watch for buffering on both sides of the wire

Audio desync after interruption is often a buffering problem, not a lip-sync algorithm problem. There are two common buffers:

• Server-side buffering: generated audio chunks sitting in a worker, queue, or SDK layer.

• Client-side buffering: WebRTC jitter buffers, media element buffers, or custom playback queues.

If you only clear one, the other can keep playing. For avatars, that means the face may switch to idle based on server state while the client still has audible frames queued. The inverse can also happen: the audio stops, but the renderer still animates mouth shapes from already-sent timestamps.

The practical fix is to define a hard stop boundary per turn. Everything after that boundary belongs to the next turn, not the current one. If a late packet arrives from the old turn, discard it. Do not “helpfully” append it to the next response. Once a turn is interrupted, its audio and animation data are poison.

How this maps to a VideoSDK voice-agent flow

In a VideoSDK-based agent, interruption usually comes from the user speaking over the agent or from explicit UI control. The key implementation detail is that the agent loop, TTS output, and avatar renderer need to share the same cancellation signal. If they each maintain separate notions of “stop,” they will diverge under load or jitter.

A reliable flow is:

1. User interrupts.

2. Voice activity detection or UI event marks the current turn canceled.

3. LLM/TTS generation stops.

4. Any unplayed audio for that turn is discarded.

5. The avatar speaking flag is cleared only for that canceled turn.

6. The next turn starts only when new audio is ready, not when new text is first emitted.

If you are wiring this into a LiveKit-style agent, the same principle applies: the voice agent owns the turn state, and the avatar renderer consumes it. The renderer should not infer state from partial text or from stale audio timing. If the cancel token fires, the entire turn becomes invalid.

For a concrete starting point, the VideoSDK quickstart is useful because it shows the integration shape without hiding the realtime plumbing.

Using Protoface without losing state coherence

Protoface is designed around this exact problem: attach a synchronized talking face to a voice agent, then keep the session state coherent when the agent is interrupted. In practice, the important part is not “render a face,” it is “render the face from the same turn state that drives audio.”

With the LiveKit Agents plugin, for example, you drop the avatar into the voice agent and let the agent control speaking state as part of the same conversation loop. That keeps the avatar aligned with the audio source rather than trying to reconstruct speech from client-side heuristics. If you are implementing this in Python, the SDK surface is what you use to create or manage avatars and sessions programmatically, while the plugin handles the realtime embedding into the agent.

Illustrative Python with the SDK looks like this; exact field names live in the docs:

from protoface import ProtofaceClient

print(session.id)

from protoface import ProtofaceClient

print(session.id)

from protoface import ProtofaceClient

print(session.id)

If you prefer REST for orchestration, the API follows the same pattern: create a session, attach an avatar, and keep a clean cancellation path in your agent code. The important bit is not the transport choice; it is making the interrupt path authoritative. The API, SDK, and plugin should all respect the same session/turn boundary.

For implementation details and the exact session fields, use the documentation. If you want the LiveKit-side integration, the plugin repository is the best reference point for how the agent and avatar state are wired together.

Testing the edge cases

If you only test the happy path, you will miss the bugs that matter. I would explicitly test these cases:

• interrupt before the first audio frame is sent,

• interrupt mid-utterance while audio is already playing,

• duplicate interrupt events for the same turn,

• late audio chunk arriving after cancellation,

• rapid interrupt followed immediately by a new response.

In each case, verify two things: the user hears no extra audio from the canceled turn, and the avatar does not keep lip-syncing to that turn. If you have logs, add the turn ID to audio and animation events so you can trace when a stale packet escaped the cancellation boundary.

Conclusion

Keeping lip-sync and audio in sync after an interruption is mostly about discipline in state management. Use explicit turns, share a single cancellation signal, clear queued audio and animation for the canceled turn, and make sure the avatar’s speaking state follows the same boundary as playback. Do that, and interruptions become a normal conversational transition instead of a source of visual glitches.

If you are implementing this with a realtime avatar stack, start with the docs, trace the turn lifecycle in your agent, and test the hard interrupt cases early. The more your audio, lip-sync, and session state behave like one unit, the fewer weird edge cases you will debug later.