Debugging Lip-Sync and Audio Drift in an Android AI Travel Assistant Avatar
Debug Android avatar lip-sync and audio drift by measuring timestamps, buffering, clocks, and WebRTC sync issues.
Introduction
When an avatar’s mouth is slightly ahead of the voice, or the video “snaps” back after a network hiccup, the problem usually is not “bad lip sync” in a vague sense. It is almost always a timing problem: audio and video are being generated, buffered, encoded, transported, and rendered on different clocks. In an Android travel assistant, that can show up as a chatbot that sounds correct but feels uncanny because the face is late, early, or periodically drifts out of phase.
This post walks through how to debug that class of issues in a real mobile app. By the end, you should be able to isolate where drift is introduced, distinguish capture-time delay from playback-time delay, and apply fixes that keep the avatar’s mouth motion anchored to the audio stream. I’ll also show where Protoface fits if you want a synchronized talking face without building the whole avatar pipeline yourself.
First, define the failure mode precisely
“Lip-sync” issues are usually one of four different problems:
Constant offset: video is always, say, 120 ms behind audio. This is often a startup or buffering issue.
Progressive drift: alignment is good at first, then worsens over time. This usually means the audio and video clocks are not locked or the app is resampling/re-timestamping incorrectly.
Jitter: the offset oscillates because frames arrive unevenly or the renderer is stalling.
Segmented desync: every time the assistant pauses or resumes, sync breaks again. This often points to state resets or incorrect handling of stream boundaries.
For an AI travel assistant on Android, the most common root cause is progressive drift caused by mixing three time domains: the avatar/video generator, the audio TTS pipeline, and the device’s playback/render clock. If any one of those gets re-anchored independently, the mouth shapes and the utterance no longer line up.
Understand the timing chain end to end
A useful debugging model is to trace one spoken token through the system:
The LLM produces text.
The TTS service converts text into audio frames.
The avatar service generates video frames that are temporally aligned to that audio.
The app receives audio/video over WebRTC or a similar low-latency transport.
Android decodes, buffers, and renders both streams.
Every stage adds latency, and only some stages should add variable latency. Fixed delay is fine if it is stable. Variable delay is what makes the mouth drift.
On Android, be especially suspicious of:
AudioTrack buffer sizing: too much buffering makes video feel late and hides timing issues until the buffer drains.
Video rendering on the UI thread: frame drops create bursts that look like desync.
Independent start times for audio and video playback: if one stream starts on receipt and the other starts after a decode queue fills, you are already offset.
Clock conversion bugs: mismatching sample timestamps, presentation timestamps, and monotonic time.
One practical rule: if audio and video are generated from the same spoken segment, treat them as a single synchronized session, not as two loosely related streams.
Instrument both streams before changing code
Do not guess. Measure. The fastest way to debug drift is to log timestamps at each boundary and compute deltas.
At minimum, record:
When the assistant text was emitted.
When TTS started producing audio.
When the first audio packet arrived in the Android client.
When audio playback actually started.
When the first avatar frame arrived.
When the first avatar frame was rendered.
If you can, log these in a single monotonic timebase. In Kotlin, that usually means SystemClock.elapsedRealtimeNanos() or a comparable monotonic clock, not wall-clock time.
Then compute a few simple metrics:
Initial offset = video render time - audio playback start time.
Drift rate = offset at end of utterance - offset at start, divided by duration.
Jitter = variance in frame arrival intervals.
If the offset is stable but wrong, you likely need a startup alignment fix. If it grows during longer utterances, look for clock mismatch, buffering asymmetry, or a renderer that is falling behind.
Check the client: Android-specific failure points
Most “the avatar is off” bugs on Android turn out to be client behavior, not the avatar model itself. A few high-probability culprits:
1. Audio starts before video is ready. If the app begins playout as soon as audio arrives but waits for several video frames before rendering, the face will appear late for the entire utterance unless you explicitly compensate.
2. Video frames are decoded on a congested thread. If frame decode or texture upload blocks, the device may render a burst of stale frames. The user experiences this as a frozen mouth followed by a catch-up jump.
3. Buffering strategy is asymmetric. A large audio prebuffer with a tiny video buffer makes the voice stable but the face twitchy; the reverse makes the face smooth but late.
4. The app drops late frames incorrectly. Dropping obsolete frames is good, but if your renderer drops the current key frame or resets its state too aggressively, the mouth animation may jump to a new phoneme out of sequence.
5. The session is being re-created mid-conversation. If a new realtime session gets minted after reconnect or activity recreation, timestamps reset and the avatar can lose continuity.
When debugging, try a controlled experiment: keep the network local, speak a long known sentence, and disable nonessential UI work. If the drift disappears, it was likely scheduler pressure or buffering, not transport.
Make the transport do less work for you to clean up later
Real-time media systems are easier to keep synchronized when they preserve timestamps and avoid unnecessary resampling or re-encoding. For an Android client, that means:
Prefer a single realtime session for the life of the conversation.
Do not convert sample rates unless you need to.
Keep encoder and decoder settings stable once playback begins.
Use monotonic timestamps end to end.
Separate capture/decoding work from UI rendering.
If you are using WebRTC, remember that it is optimized for low latency and playout adaptation, not for preserving a perfect fixed offset between arbitrary audio and video sources. The browser or client may adapt to network conditions by altering playout timing. That is usually good for intelligibility, but it means your app should not invent its own extra timing corrections unless you have measured the actual error.
Also watch for “helpful” Android audio APIs or media wrappers that silently resample or normalize buffers. Those can introduce tiny errors that become obvious after several seconds.
Debug with a known-good utterance and a known-good timeline
To separate model behavior from client behavior, use a deterministic test phrase and replay it repeatedly. The goal is to make the audiovisual timing comparable across runs.
A simple workflow:
Choose a fixed sentence with several phoneme transitions, including plosives and vowels.
Run the same utterance multiple times over the same network path.
Measure the offset at the beginning, middle, and end.
Compare results with Wi-Fi, LTE, and a local dev network.
If the offset changes with network quality, you may be over-buffering or failing to handle jitter correctly. If it changes with device load, your render pipeline needs isolation. If it is stable but consistently wrong, the issue is likely session startup or timebase alignment.
Where Protoface fits
For teams building a travel assistant, the easiest way to remove one large source of sync bugs is to let the avatar layer handle its own synchronized talking face instead of stitching video generation onto a separate voice pipeline. The LiveKit plugin, Pipecat integration, and the Python SDK are the right surfaces when you want to add a realtime face to an existing voice agent while keeping the media timing tied to the session rather than to a homegrown renderer. The public docs at docs.protoface.com cover the exact session and avatar fields; the snippet below is intentionally illustrative.
If you are already on LiveKit, the plugin path is often the least invasive route because the avatar arrives as part of the agent stack instead of as an unrelated video component. That matters for sync debugging: one session, one set of timestamps, fewer places for drift to hide.
A practical checklist for fixing drift
When the avatar still looks off, work through this order:
Confirm whether the problem is constant offset or progressive drift.
Log audio start, video start, and render timestamps on the device.
Reduce buffering on both streams and see whether the offset changes.
Move decode and texture upload off the UI thread.
Keep the same realtime session alive across the utterance.
Verify you are not resampling, re-timestamping, or recreating the pipeline mid-stream.
If a single change improves sync dramatically, keep it. If multiple changes are needed, re-test after each one so you know which layer actually fixed the issue.
Conclusion
Avatar lip-sync problems are usually timing bugs, not magic. The key is to treat audio and video as a single synchronized pipeline, instrument every boundary, and identify whether the failure is a startup offset, drift, or jitter. On Android, most issues come from buffering, thread scheduling, and clock mismatches rather than from the model itself.
If you want to avoid rebuilding the avatar side of that pipeline, start with the docs, inspect the LiveKit or SDK integration that matches your stack, and test against a fixed utterance before shipping to users. For implementation details and integration patterns, see docs.protoface.com and the relevant repositories in the developer quickstarts.