Introduction

Lip-sync drift is what happens when an avatar’s mouth motion slowly stops matching the audio that drives it. At first it looks fine; then the face starts opening a beat early, lagging behind speech, or finishing a phrase after the audio has already ended. In streaming systems, that usually means your audio and video pipelines are no longer sharing the same timing assumptions.

This post is about diagnosing that mismatch and fixing it in practice. By the end, you should be able to identify whether the drift is coming from capture, buffering, sample-rate conversion, network jitter, or downstream rendering, and then apply the right correction strategy instead of papering over the symptom.

What “drift” actually means in a realtime avatar pipeline

For a talking avatar, the audio stream is the source of truth. Lip motion is typically driven by one of three signals:

• the same PCM audio used for playback,

• phoneme/viseme events derived from the text or TTS audio, or

• an alignment model that estimates mouth shapes from the stream.

Drift appears when the timing of those signals diverges. The common failure modes are different:

• Constant offset: lips are always, say, 150 ms ahead of the audio. This usually means a fixed buffer mismatch.

• Accumulating drift: the gap gets worse over time. This usually points to clock mismatch, resampling error, or queue buildup.

• Jittery sync: sometimes aligned, sometimes off by a frame or two. This is usually packet jitter or render scheduling variance.

It helps to think in timestamps. Each audio chunk and each video frame should be associated with the time it is supposed to be presented, not just when it happened to arrive. If you only process “latest chunk wins,” the system will look okay under ideal conditions and fail as soon as latency changes.

Step 1: Find the source of truth and keep one clock

The first fix is structural: decide which stream owns the timeline. In a voice-agent avatar setup, that should almost always be the audio pipeline. The mouth should follow the audio, not the other way around.

That means:

• Use a single monotonic clock for buffering and scheduling.

• Timestamp audio frames as close to capture or synthesis as possible.

• Preserve those timestamps through any transport layer.

• Render mouth shapes against the intended playout time, not arrival time.

If you are mixing generated speech with visual events, be careful not to introduce a second timeline. For example, if your TTS service emits phonemes and your media stack separately renders audio packets, don’t drive the avatar from the phonemes alone unless you can guarantee they were generated from the same segment boundaries as the audio.

In practice, a constant offset often comes from a hidden buffer on one side of the pipeline. A WebRTC track may be buffered by the browser, while your avatar renderer assumes near-zero latency. Fixing it means aligning the expected playout delay, not just “making it faster.”

Step 2: Measure end-to-end latency before changing anything

Before changing code, instrument the pipeline. You want at least four timestamps:

1. audio produced

2. audio sent

3. audio rendered

4. avatar frame rendered

If the gap between produced and rendered audio is stable but large, you likely have a fixed buffer issue. If the gap grows over time, look for drift in sample rates or unbounded queues. If the gap jumps around, look at network jitter or frame scheduling.

A simple diagnostic is to log the delta between the avatar frame timestamp and the audio playout timestamp every second. If that delta trends upward, the system is not staying synchronized even if it still “looks close enough” in short demos.

# Pseudocode for drift logging
logger.info("lip_sync_drift_ms=%d", drift_ms)

# Pseudocode for drift logging
logger.info("lip_sync_drift_ms=%d", drift_ms)

# Pseudocode for drift logging
logger.info("lip_sync_drift_ms=%d", drift_ms)

At this stage, the goal is not to eliminate latency. It is to separate fixed latency from drift. Fixed latency can be hidden with a larger initial offset. Drift cannot.

Step 3: Check sample rates, frame sizes, and resampling

Accumulative drift often comes from one of the mundane problems everyone skips past: sample-rate mismatch, audio chunking, or resampling that is slightly off. A common example is receiving 48 kHz audio, converting it, and then treating the output as if it still represented the original time base.

Things to verify:

• Sample rate: the audio producer and consumer must agree on the actual rate.

• Frame duration: your chunk size should map cleanly to time, for example 20 ms or 40 ms frames.

• Resampler behavior: some libraries output variable-length chunks; make sure you account for the exact number of samples produced.

• Silence handling: dropping silent frames can speed up the apparent timeline and desynchronize the avatar.

A subtle bug is to treat “chunk count” as time. Ten chunks of audio is not meaningful unless each chunk is the same duration and none were dropped, merged, or padded. Use sample counts and timestamps.

If you need to realign after the fact, compute the presentation time from sample position:

# Each chunk carries samples and a start timestamp.
chunk_end_time_ms = chunk_start_time_ms + chunk_duration_ms

# Each chunk carries samples and a start timestamp.
chunk_end_time_ms = chunk_start_time_ms + chunk_duration_ms

# Each chunk carries samples and a start timestamp.
chunk_end_time_ms = chunk_start_time_ms + chunk_duration_ms

That seems basic, but it prevents a class of drift bugs where the renderer assumes a fixed cadence while the encoder actually emits variable-sized packets.

Step 4: Make the avatar renderer tolerant of jitter, not blind to delay

Good realtime sync is not “render immediately.” It is “render at the right scheduled time, with a bounded buffer.” In a streaming avatar, a small jitter buffer is useful because it absorbs network variance and avoids mouth motion that snaps back and forth between late packets.

The trick is to keep the buffer bounded. If you keep accumulating delayed events, sync will degrade over time. If you flush too aggressively, you convert latency into visible jitter.

A practical rule:

• buffer enough to cover typical jitter,

• drop or coalesce stale frames once they miss their playout window,

• never let the queue grow without limit.

For lip sync, stale mouth shapes are usually worse than slightly simplified motion. If a viseme event arrives after its intended audio window, it is better to skip it than to apply it late and cause the face to “chase” the speech.

Also watch the difference between render time and audio time. If your browser tab is under load or the server is busy, video frames may be delayed even though audio still plays in time. The fix there is to make the mouth animation deterministic for the scheduled audio window, not tied to the exact wall-clock moment the render callback happened to execute.

How Protoface fits when you want a synchronized face without building the media layer yourself

If you are adding a talking face to a voice agent, one clean way to avoid a lot of low-level drift work is to use the LiveKit plugin from the [protoface plugin repo](https://github.com/protoface-ai/protoface-plugin-pipecat) or the published package on PyPI. The plugin’s job is to drop a realtime avatar into the agent pipeline so the face stays synchronized with the voice stream instead of forcing you to hand-roll timestamping and frame scheduling.

For example, in a LiveKit agent setup, you typically wire the avatar as part of the media pipeline rather than as an independent video source. The exact API surface can change, so treat this as illustrative:

from livekit.plugins import protoface

from livekit.plugins import protoface

from livekit.plugins import protoface

If you are orchestrating sessions directly, the REST API and Python SDK are useful for creating and managing avatars and realtime sessions without exposing credentials in the client. The API is authenticated with an API key in the Authorization header, and the Python SDK lets you script session creation and teardown from your backend. See the docs for the exact request and object shapes.

import requests<p></p>

import requests<p></p>

import requests<p></p>

The main operational benefit is that you keep the avatar and media timing inside one supported integration point instead of stitching together separate video, TTS, and browser timing layers yourself.

Common fixes that actually work

Once you know where the drift comes from, these are the fixes that usually matter most:

• Normalize sample rates before buffering or rendering.

• Timestamp everything using a monotonic clock.

• Keep queues bounded and drop stale lip-sync events instead of accumulating them.

• Measure playout delay separately from network delay.

• Use one timing model for audio, visemes, and frame rendering.

If you are debugging in a browser, also verify that the tab is not background-throttled and that your render loop is not coupled to requestAnimationFrame for timing-critical decisions. requestAnimationFrame is fine for presentation; it is not a reliable media clock.

And if your source is a generated voice agent, check the TTS engine’s chunk boundaries. Some engines emit speech in bursts with variable latency. That is fine as long as your avatar follows the actual audio timestamps rather than the arrival cadence of the bursts.

Conclusion

Lip-sync drift is almost always a timing problem, not a “face animation” problem. Start by deciding which stream owns the timeline, measure end-to-end latency, verify sample-rate and frame-duration assumptions, then bound your jitter buffers so they absorb variance without accumulating delay.

If you want a supported path for this in a realtime voice-agent stack, use the Protoface docs and the relevant integration guide for your setup. The quickest next step is to reproduce your issue with timestamps turned on, then compare that trace against the examples in the documentation. Once you can see where the drift starts, the fix is usually straightforward.