A Practical Guide to Fixing Mouth-Shape Drift in Realtime Video Avatars
Practical debugging guide to mouth-shape drift in realtime video avatars: timing, buffering, sync metrics, and fixes.
Introduction
Mouth-shape drift is one of the most common quality failures in realtime video avatars: the face looks plausible at first, but over time the mouth stops matching the phonemes, lags behind the audio, or seems to “stick” in a previous shape. In production, this usually shows up as a synchronization problem, not just a rendering problem. The audio stream, the animation stream, and the transport all have their own clocks, buffering, and latency characteristics, and if you ignore any of them the avatar will drift.
This post is a practical guide to diagnosing and fixing that drift. By the end, you should be able to reason about where the misalignment comes from, instrument a realtime avatar pipeline, choose the right sync strategy, and avoid the usual traps when you wire a voice agent to a talking face.
What mouth-shape drift actually is
In realtime avatars, the face animation is typically driven by one of three signals:
Audio features derived from the speech stream, such as energy, pitch, or mel-spectrograms.
Text or phoneme timing, where the system predicts viseme timing from a transcript or TTS metadata.
Direct animation tokens produced by a model designed to generate facial motion.
Drift happens when the animation pipeline and the audio pipeline stop agreeing on which moment in the utterance each frame represents. Common symptoms:
Initial sync is fine, but the mouth slowly lags by a few hundred milliseconds.
After a pause or interruption, the mouth jumps to an incorrect pose and stays there.
The avatar over-articulates or under-articulates because the wrong audio chunk is being used to drive the wrong frame window.
In practice, this is almost always a timing and buffering issue, occasionally compounded by model choice or resampling bugs.
Start by locating the clock that is drifting
The first debugging step is to identify whether the problem is caused by generation latency, transport latency, or playback latency. Those are different failure modes and they require different fixes.
Generation latency means the avatar frames are produced too late relative to the corresponding speech. This is often caused by too-large inference windows, batching, or waiting for too much context before starting motion.
Transport latency means audio and video arrive with uneven delay. WebRTC, server-side relays, and browser decoding all introduce jitter buffers. If the audio path is more buffered than the video path, the mouth will appear behind the sound even if the model output was correct.
Playback latency means the client is rendering frames based on local time, while the audio element or WebRTC receiver is playing at a different effective clock. This is common when you mix custom animation timing with standard media playback.
A useful test is to log the following for each utterance:
Audio chunk timestamp at source.
Animation generation start/end time.
Frame timestamp or viseme timestamp emitted by the avatar service.
Arrival time at the client.
Audio playout time at the client.
If the offset between audio and mouth grows monotonically, you likely have accumulated drift. If it is roughly constant, you probably have a fixed sync offset that can be compensated. If it jumps after pauses, you likely have state reset or buffering boundaries that are not aligned.
Make the animation pipeline explicitly stream-oriented
The biggest mistake is treating lip sync like a batch job. Realtime avatars need bounded latency and continuous state. That means the avatar generator should consume short audio windows, maintain internal continuity, and emit animation frames with a stable mapping to time.
For audio-driven systems, keep these principles in mind:
Use consistent chunk sizes. Very large chunks increase latency. Very small chunks can create jitter if the model has too little context.
Preserve sample rate and frame rate assumptions. Resampling bugs are a classic source of mouth drift. If the animation model was tuned for 16 kHz mono PCM and you feed it 48 kHz stereo after a lossy conversion, expect trouble.
Carry timestamps end to end. Do not infer timing from “arrival order” alone. When possible, attach source timestamps to each audio packet and each animation packet.
Reset state at speech boundaries. When an agent pauses, is interrupted, or starts a new turn, the mouth state should be re-anchored. Otherwise the previous utterance can bias the first few frames of the next one.
If your pipeline emits visemes, the useful abstraction is not “frame N” but “frame at audio time T.” That lets you rebuffer, resample, or even reorder packets without destroying sync, as long as you preserve the timebase.
Handle turn-taking, interruptions, and silence carefully
Most visible drift appears during conversational edges: barge-in, brief silence, and agent handoffs. These are where state machines matter more than model quality.
Three rules help a lot:
Define explicit speech turns. A turn should start when speech begins and end when the system decides the utterance is complete. Tie mouth animation to that lifecycle, not to arbitrary media chunks.
Decay to neutral on silence. Don’t freeze the last mouth shape indefinitely. Most avatars should decay to a neutral or lightly open rest pose after a configurable silence threshold.
On interruption, discard stale animation. If a user interrupts an agent mid-utterance, old visemes should not continue to play out. Flush the queue and start from the new audio immediately.
Another subtle issue is “late context”: if the voice model finishes generating an utterance after the audio has already started playing, the mouth can end up trying to catch up from behind. The fix is usually to shorten the front buffer and prioritize initial articulation over perfect long-horizon prediction. In other words, start cleanly and stay approximately synchronized rather than waiting for complete certainty.
Measure drift, don’t guess at it
You can’t reliably fix what you aren’t measuring. In a production system, add a simple per-utterance metric: the delta between the audio playout time and the animation time for the same speech segment. Track it over time and across clients.
What to look for:
Mean offset: a stable bias means you need a calibration offset.
Offset variance: high variance usually means jitter buffer or packet timing issues.
Offset growth over time: this points to clock drift, incorrect frame pacing, or state that is never reset.
Tail latency after pauses: often caused by a stale buffer being drained after the new utterance begins.
In debugging sessions, a dead-simple overlay helps more than fancy tooling: show current audio time, current viseme time, and the difference in milliseconds. If you can record a few problematic sessions and replay them with those traces, the root cause usually becomes obvious.
Protoface in practice: using the LiveKit plugin to keep audio and face aligned
For voice-agent developers, the easiest way to avoid drift is to keep the avatar inside the same realtime session that already carries the agent’s audio. That is exactly where the LiveKit integration is useful: the avatar becomes part of the agent pipeline instead of a separate media subsystem with its own timing model.
The practical benefit is that the plugin can align the mouth animation to the agent’s speech stream rather than to an after-the-fact recording. That reduces the chance of a hidden buffer mismatch. If you are using LiveKit Agents, this is the least operationally awkward place to start.
If you are building from scratch and want to inspect the lower-level API flow, the REST surface at docs.protoface.com shows how to create and manage avatars and realtime sessions. That is useful when you need to script session setup, test timing issues with controlled inputs, or separate avatar lifecycle from your application logic.
The important architectural point is not the exact endpoint shape; it is that the avatar session should have a single authoritative timeline. Once you have that, the remaining work is mostly tuning buffer sizes, reset behavior, and interruption handling.
Common fixes that actually work
When a production avatar drifts, these are the fixes that usually move the needle:
Reduce front buffering. Start animation sooner, then smooth as more audio arrives.
Normalize sample rates early. Convert audio to the model’s expected format before it enters the lip-sync path.
Pin a single timebase. Choose source time, session time, or playout time, and be consistent.
Flush on state changes. New turn, barge-in, reconnect, or session resume should not reuse stale mouth state.
Test under jitter. Drift often appears only when latency varies, not when the network is perfectly clean.
Also, do not overfit to one browser or one network condition. WebRTC and media stacks can behave differently under mobile network jitter, VPNs, and long-running sessions. A fix that hides drift in a local demo can fail after ten minutes in a real call.
Conclusion
Mouth-shape drift is usually a synchronization problem disguised as a rendering problem. The durable fix is to treat the avatar as a realtime stream with an explicit timebase, bounded buffers, and clear turn boundaries. Once you do that, you can isolate whether the issue is generation, transport, or playback, and fix the right layer instead of guessing.
If you are integrating a voice agent, start with the LiveKit path, keep your chunking and reset behavior disciplined, and instrument the offset between audio and animation. For API-driven workflows or custom session management, the docs at docs.protoface.com are the right place to verify the exact request and session fields. If you already have a problematic session, reproduce it with controlled audio, log the time deltas, and fix the first point where the clocks stop agreeing.