How to Reduce Lip-Sync Drift in Realtime AI Avatars
Reduce lip-sync drift in realtime AI avatars with timestamped audio, bounded buffers, jitter handling, and resync logic.
Introduction
In realtime AI avatars, lip-sync drift usually isn’t a “bad model” problem. It’s a systems problem: audio, video, and inference each run on their own clocks, and the user notices the mismatch long before they notice the root cause. A face that speaks 150–300 ms late, slowly accumulates latency over a call, or snaps back after a network hiccup feels broken even if the underlying speech and animation are individually fine.
This post is about reducing that drift in practice. By the end, you should be able to reason about where sync error comes from, what to measure, which knobs actually matter, and how to design a realtime pipeline that stays stable under jitter, variable inference time, and browser playback behavior.
Start with the right mental model
Drift is not just “video is behind audio.” In a realtime avatar pipeline, you typically have:
a text or voice agent generating content,
speech synthesis or audio playback producing an audio stream,
face animation or video generation producing visual frames, and
a transport layer delivering both to the client.
Each stage contributes latency. More importantly, each stage can contribute variance. Constant latency is annoying but manageable; variable latency is what creates perceptible desync because the system can’t hold a stable offset.
The most common failure mode is a pipeline that starts in sync and then drifts because one side is effectively “free-running” instead of being paced by timestamps. The fix is to treat audio as the timing reference, and make video generation, buffering, and playback follow that clock as closely as possible.
Measure the problem before you optimize it
Before changing architecture, instrument the pipeline. If you cannot answer “how far behind is the mouth relative to the audio right now?” you are guessing.
Useful metrics:
End-to-end audio latency: time from synthesis or agent output to audible playback.
Video presentation latency: time from frame generation to display.
Audio-video offset: current difference between the mouth pose being shown and the audio sample being heard.
Jitter: variance in delivery timing for audio packets and video frames.
Queue depth: how much unsent/undisplayed media is buffered at each hop.
If you are building your own client, log timestamps at every boundary: model output, TTS start, packetization, WebRTC send, client receive, decode, and render. The important thing is not perfect nanosecond precision; it is consistent measurement of where latency accumulates.
A practical diagnostic pattern is to compute a moving average offset and a max offset over a sliding window. If the average grows gradually, you likely have a pacing problem. If the max spikes and recovers, you likely have jitter and buffer underflow/overflow. Those are different fixes.
Keep a single master clock and pace everything against it
The core rule: pick one timing source and make the rest conform. In realtime avatar systems, audio is usually the master clock because humans tolerate a little visual inconsistency more than audio glitches. Once playback starts, the avatar renderer should advance based on audio timestamps, not on wall-clock rendering intervals alone.
That means:
Generate or receive audio with timestamps.
Buffer enough audio to cover network jitter, but not so much that latency becomes visible.
Associate animation frames or viseme state with the corresponding audio time.
Render the latest frame that should be visible for the current audio position, not the latest frame that happened to arrive.
One subtle but common mistake is to let the video side “catch up” by dropping or fast-forwarding frames whenever the queue grows. That can reduce lag temporarily, but it also creates visible discontinuities and often makes drift worse over time because the underlying timing mismatch is still present. A better approach is to keep the queues bounded and stable so that catch-up behavior is rare.
Reduce drift by controlling buffering, not just latency
Low latency and low drift are related, but they are not the same thing. A tiny buffer gives you fast startup but high dropout risk. A large buffer hides jitter but adds delay. The goal is a buffer that is just large enough to absorb typical network and processing variance.
For realtime avatars, these rules help:
Bound the audio queue. If the queue grows past a threshold, prefer dropping or resynchronizing old buffered content over letting latency compound.
Use smaller, regular chunks. Large irregular audio or frame chunks increase burstiness and make pacing harder.
Prefer steady frame cadence. Even if the avatar is generated from audio features rather than full frames, the client should receive updates at a predictable rate.
Don’t overbuffer video. Video is usually the first place drift becomes visible; a long video queue makes the face look “late” even when audio is fine.
On the browser side, remember that media playback itself adds buffering and decode delay. If you control the client, use the browser’s actual playback position, not just “time since I received the packet,” as the sync reference. WebRTC already gives you a lot here, but only if you respect its timing model instead of layering a second, conflicting clock on top.
Handle jitter and recovery explicitly
Realtime network conditions are messy. The avatar should not assume packet delivery is smooth. Instead, design for two behaviors: small jitter and large disruption.
For small jitter, concealment is enough. Hold the most recent stable mouth state for a short interval, or interpolate between adjacent animation states rather than jumping immediately. That keeps motion smooth when audio packets or animation updates arrive a little late.
For larger disruption, resync quickly and deliberately. If the offset passes a threshold, it is better to rebase the visual state to the current audio time than to slowly accumulate more error trying to preserve continuity. This is the same trade-off you make in live streaming: graceful degradation until a threshold, then a hard correction.
A useful rule of thumb is to treat the system as two layers:
local smoothing for sub-threshold jitter, and
global resynchronization for real drift.
If you do not separate those cases, you tend to overcorrect small problems and undercorrect large ones.
Watch the generation side, not just playback
Many sync bugs originate upstream of the browser. If the agent generates speech in uneven bursts, or the avatar pipeline waits for full sentences before emitting animation, you get phase lag that no amount of client-side tweaking can fully hide.
Common upstream causes:
LLM output arriving in large chunks rather than streamingly.
TTS beginning late because the system waits for too much text.
Face animation depending on the final audio waveform instead of incremental audio segments.
Per-request cold starts or variable model latency causing unstable initial timing.
The fix is to make the avatar pipeline incremental end to end. Start audio as soon as you have enough text to speak, emit animation updates continuously, and keep session state warm when possible. For conversational agents, a stable 200 ms offset is often preferable to a wildly varying 50–800 ms offset.
A practical implementation pattern
If you are building your own avatar client, a good pattern is:
Attach timestamps to every audio chunk and animation update.
Use audio playback time as the source of truth.
Keep a small jitter buffer for both streams.
Render based on “what should be visible now,” not “what arrived last.”
When drift exceeds a threshold, resync instead of slowly accumulating error.
A simplified Python example that creates a session over the REST API might look like this. Exact request fields depend on the docs, so treat this as a shape, not a copy-paste contract:
And if you are using a LiveKit voice agent, the plugin approach is usually the lowest-friction way to keep the face tied to the agent’s audio timing. The important part is that the avatar is part of the same realtime pipeline as the voice agent, not a separate process trying to infer timing after the fact:
If you want to see the exact integration surface and available quickstarts, the relevant starting points are the docs and the LiveKit plugin examples in the plugin repository when you are working in that ecosystem.
Protoface-specific notes that matter in production
For developers integrating a realtime avatar into a voice agent, the main advantage of using a platform instead of hand-rolling the full stack is consistency of timing behavior across sessions. With a REST API for session creation, a Python SDK for programmatic control, and a LiveKit plugin for voice-agent integration, you can keep the avatar attached to the same realtime path as the rest of your application rather than bolting it on after the fact.
That matters for drift because the closer the avatar is to the actual audio source, the fewer opportunities there are for queue growth and timestamp mismatch. If you are embedding an avatar on a website, the iframe-based approach also helps by isolating playback and transport details behind a controlled client boundary, which reduces the chances that your app code accidentally introduces extra buffering or timing bugs.
The practical takeaway is simple: keep the avatar close to the media clock, keep buffers bounded, and use explicit resync logic when the system falls behind. If the platform provides the synchronization layer for you, lean on it; if not, replicate the same principles in your own implementation.
Conclusion
Lip-sync drift is mostly a timing and buffering problem. Measure it, tie video to audio timestamps, keep queues short and bounded, and distinguish between small jitter and real divergence. That will get you farther than chasing frame-perfect animation in isolation.
If you are implementing this in a live voice agent or web avatar, start with a single pipeline, instrument the offsets, and verify that your system can hold a stable sync target under network variance. Then refine startup latency and resync behavior. For docs and integration examples, see docs.protoface.com.