Debugging Audio-Video Desync in Realtime AI Avatars: Causes, Metrics, and Fixes
Debug audio-video desync in realtime avatars: measure latency, offset, and drift; fix buffering, timestamps, and scheduling.
Introduction
Audio-video desync in realtime avatars usually shows up in one of three ways: the mouth lags the speech, the face leads the speech, or sync is good at first and then drifts. In a voice-agent pipeline, that almost never comes from a single bug. It is usually the interaction of ASR latency, LLM latency, TTS chunking, transport jitter, buffering, and the avatar renderer’s own frame timing.
If you are building a talking face for a voice agent, customer-support bot, game NPC, or web embed, the core skill is not “make the lips move.” It is being able to measure where the pipeline is losing time, distinguish true A/V drift from normal startup latency, and pick the right fix without masking the underlying issue.
By the end of this post, you should be able to:
identify where desync is introduced in a realtime avatar stack,
measure sync with practical metrics instead of gut feel,
apply fixes at the transport, buffering, and scheduling layers, and
understand where a platform like Protoface fits into the debugging workflow.
Where desync actually comes from
In a typical voice-agent-to-avatar flow, audio is generated or streamed first, then video is synthesized or animated to match it, and both are delivered over a realtime transport such as WebRTC or a similar low-latency media path. Each stage adds latency and can add jitter. The important point is that “sync” is not a single property; it is the result of several clocks and buffers staying aligned.
Common failure modes:
Upstream latency variance. ASR, LLM, or TTS timing changes from turn to turn, so the avatar receives speech chunks late or unevenly.
Audio chunking mismatch. If TTS emits audio in irregular chunks but the avatar pipeline assumes steady cadence, lip motion can stutter or drift.
Render queue buildup. The face renderer may keep animating based on stale audio frames if it cannot keep up with the incoming stream.
Transport jitter and reordering. Even with low latency, packet delay variation causes buffers to expand, which adds delay.
Clock drift between processes. Separate processes or devices can disagree about wall-clock time, especially over longer sessions.
A useful mental model is: audio and video are not “kept in sync” by one magical setting. They are synchronized by timestamps, buffering policy, and disciplined scheduling across the whole pipeline.
Measure the problem before changing anything
The easiest way to waste time is to guess. You want two classes of metrics: startup latency and steady-state offset.
Startup latency is the delay from user speech or agent turn start to the first audible audio and first visible mouth motion. Some delay is unavoidable, but large variance here makes the avatar feel broken even if steady-state sync is fine.
Steady-state offset is the ongoing difference between when audio is presented and when the corresponding mouth shapes appear. This is what users notice as “the lips are behind the voice.”
Track at least these timestamps in your logs:
input turn start
LLM first token or first synthesis request
TTS first audio byte
first audio packet sent
first video frame enqueued
first audio playout
first video playout
From those, you can derive:
generation latency = synthesis start to first audio byte
transport delay = send time to playout time
A/V playout offset = video playout time minus audio playout time
drift rate = change in offset over time
If your stack exposes per-chunk timestamps, log them. If not, add them. Without timestamps, “desync” is just a feeling.
Debug by isolating the layer that is wrong
Once you have metrics, isolate the layer before you tweak the avatar. A good workflow is:
Verify audio alone. Play the generated TTS stream without the avatar. If audio itself is late or uneven, the issue is upstream.
Freeze the video input. Feed the avatar a fixed audio track. If the mouth motion is still off, the problem is in lip-sync inference or render timing.
Compare source timestamps to playout timestamps. If the gap is stable but large, you have latency. If it grows, you have drift.
Test under load. CPU spikes, GC pauses, and event-loop stalls often create bursty desync that disappears in isolated tests.
A practical rule: if sync is bad only when the system is busy, it is usually a scheduling or buffering problem, not an avatar-model problem.
The fixes: buffer less, timestamp more, and control cadence
The right fix depends on the failure mode.
1) Reduce variable buffering
Over-buffering hides jitter at the cost of delay. That is often the wrong trade-off for conversational avatars. Aim for a minimal, fixed playout buffer that absorbs normal network jitter without building a latency tail.
If the browser or client accumulates audio packets before starting playback, tune the buffer size down. If you are on WebRTC, remember that jitter buffers are adaptive; they will increase delay when network conditions worsen. That is good for continuity, bad for perceived sync if your pipeline already has enough latency.
2) Use timestamps all the way through
Every audio chunk and video frame should carry a production timestamp and, ideally, a target playout timestamp. That lets the renderer decide whether a frame is still valid or should be dropped. Dropping stale video is often better than showing a mouth shape that matches speech from 200 ms ago.
For realtime avatars, “drop late video” is usually preferable to “queue everything.” A conversational face can tolerate skipped frames better than lagging articulation.
3) Keep audio cadence stable
Speech lip sync works best when the audio stream arrives in a predictable rhythm. If your TTS service emits irregular chunk sizes, normalize them before feeding the avatar. Even if the overall latency is unchanged, stable cadence helps the mouth motion track phoneme timing more cleanly.
4) Avoid head-of-line blocking in your app
One slow callback should not block media handling. A common anti-pattern is doing model inference, state updates, and UI work on the same event loop that is responsible for media dispatch. If the loop stalls for even 50-100 ms, sync can visibly break.
Separate control-plane logic from media-plane logic. Keep media handling lean, asynchronous, and backpressure-aware.
5) Calibrate for network and device conditions
Mobile devices, low-power laptops, and high-jitter networks often need different defaults from a desktop on wired Ethernet. Use measured latency to select a conservative profile rather than one static buffer size for everyone.
Small code example: instrument the pipeline
Even if your avatar layer is hidden behind a plugin or SDK, add timing around the calls you do control. For example, when requesting synthesis or creating a session, log the request/response timestamps and the first media event you observe.
That kind of coarse instrumentation is often enough to tell whether the delay lives in session setup, media start, or render timing.
Where Protoface fits
If you are using LiveKit Agents, the plugin and the underlying integration pattern are useful when you want to drop a synchronized talking face into an existing voice agent without rewriting the rest of the pipeline. The main debugging advantage is that you can treat the avatar as a distinct media component, which makes it easier to compare voice-agent timing against avatar rendering timing.
For lower-level investigation, the REST API and Python SDK let you create sessions programmatically, reproduce a problematic configuration, and capture timing around session creation and media startup. Keep in mind that exact request and response shapes are documented in the public docs, and you should not rely on guessed field names.
Example of a minimal API call pattern:
If you are integrating through LiveKit, the plugin repository and examples are the quickest way to compare a known-good configuration with your own. Start from the relevant quickstart, then add logging around the points where audio enters the agent and where avatar frames are emitted. That gives you a baseline before you tune buffers or transport settings. The public docs at docs.protoface.com are the right place for the exact SDK and API details.
Practical debugging checklist
When a bug report says “the avatar is out of sync,” work through this checklist:
Measure first audio, first video, and steady-state offset separately.
Check whether the issue is constant latency or growing drift.
Test audio alone before blaming the avatar renderer.
Look for event-loop stalls, GC pauses, or CPU spikes.
Reduce buffering before increasing it.
Drop stale video rather than queueing it indefinitely.
Normalize chunk cadence where possible.
If you can reproduce the issue on a fixed input, keep that test around. Realtime sync bugs are notoriously regress-prone, especially when you later optimize for “just a little less latency.”
Conclusion
Audio-video desync in realtime avatars is usually a systems problem, not a lip-sync model problem. The fastest path to a fix is to measure startup latency, steady-state offset, and drift; isolate the layer that is actually wrong; and then tune buffering, timestamps, and scheduling with intent.
If you are building on Protoface, start by instrumenting your voice-agent pipeline and reproducing the issue in a controlled session. Then use the docs, SDK, or plugin path that matches your integration and verify whether the problem is in generation, transport, or render timing. The public docs at docs.protoface.com are the best place to fill in the exact API and integration details.