Introduction

Lip sync drift is what happens when the avatar’s mouth motion stops matching the audio in a realtime call: the face starts speaking “ahead” of the voice, lags behind it, or oscillates between the two. In a production iOS app, that drift is usually not a single bug. It’s the result of timestamp misalignment, audio buffering, jitter handling, or a pipeline that mixes independently clocked media sources without a shared notion of time.

This post walks through how to diagnose the problem in a realtime avatar app, how to distinguish true lip-sync drift from normal network latency, and what fixes actually matter. By the end, you should be able to isolate whether the issue is in capture, transport, rendering, or the avatar service itself, and apply the right mitigation instead of just adding more buffering and hoping for the best.

What “drift” actually means in a realtime pipeline

In practice, lip sync drift is a relative timing problem. Audio and video can both be delayed by 200 ms and still look correct if they stay aligned. The issue appears when one stream advances on a different clock, or when one side applies a different amount of buffering, resampling, or frame dropping than the other.

On iOS, the usual failure modes look like this:

• Audio arrives in chunks, video in frames. If you render video as soon as frames arrive but play audio through a separate buffer, they will diverge unless both are driven by compatible timestamps.

• Network jitter causes uneven arrivals. Realtime transport can reorder delivery timing even when end-to-end latency is acceptable.

• Clock drift accumulates over time. If the audio clock, device clock, and remote timestamp source are not reconciled, a small mismatch becomes visible after a minute or two.

• Queue growth hides the problem. A growing audio queue or frame queue can look like “smooth playback” while the avatar slowly falls behind the speech.

The important distinction: latency is not drift. Latency is a constant offset; drift is a changing offset.

Start by measuring the offset, not guessing

Before you change buffering settings, instrument the pipeline. You want at least three timestamps for each turn of speech:

1. when audio was captured or synthesized,

2. when the avatar video frame was produced or received,

3. when the media was actually rendered on device.

If you control the client pipeline, log the arrival time and the presentation timestamp separately. If you only consume an embedded avatar stream, log the timestamps you can observe at the edge of your app and compare them over time. What matters is whether the delta between audio and mouth motion stays stable.

// Example shape of diagnostics you want to collect in an iOS client.

}

// Example shape of diagnostics you want to collect in an iOS client.

}

// Example shape of diagnostics you want to collect in an iOS client.

}

When you inspect logs, look for one of these patterns:

• Stable offset, always late: usually buffering or encode/decode delay, not drift.

• Offset grows over time: clock mismatch or queue accumulation.

• Offset jumps after network events: jitter buffer reset or packet loss recovery.

• Offset varies with device load: main-thread stalls or dropped frames on the render path.

Common causes in iOS apps

Most lip sync issues in iOS apps come from one of four layers.

1. Separate clocks for audio and video

If audio playback is driven by Core Audio while video rendering is driven by a display link or an async callback, you may be using two unrelated timing models. That is fine only if the media layer keeps them anchored with timestamps. If your code treats “audio received” and “video received” as equally real-time, drift is inevitable.

Fix: choose one canonical timeline. In practice, this usually means respecting presentation timestamps from the media pipeline and rendering both modalities relative to those timestamps instead of their arrival time.

2. Overbuffering to hide jitter

A common instinct is to increase the audio buffer until playback stops glitching. That can improve quality, but it also increases latency. If video is not buffered by a similar amount, the avatar’s mouth will appear to lag more and more behind the spoken content.

Fix: keep audio and video buffering policy consistent. If you need a larger jitter buffer, apply it as a system-wide decision, not just on one stream.

3. Main-thread stalls and render drops

Even if media arrives on time, rendering can slip if the UI thread is busy. On iOS, heavy layout, image decoding, or synchronous work on the main thread can delay frame presentation. The effect can resemble lip sync drift because audio continues uninterrupted while mouth frames are late or skipped.

Fix: keep decoding and network work off the main thread, and make the render path predictable. If you must drop frames, drop them intentionally and keep the avatar on the current timestamp, rather than trying to render a backlog.

4. Resampling and sample-rate mismatches

If your audio source is 48 kHz but another stage expects 44.1 kHz, resampling can introduce latency and subtle rate errors if configured poorly. Over time, those errors look like drift. This is especially easy to miss when mixing synthesized speech, microphone input, and remote avatar playback.

Fix: verify the sample rate at every boundary. Avoid implicit conversions in multiple places. One clean resampling step is better than several opaque ones.

How to debug the pipeline systematically

When you are troubleshooting a realtime avatar app, isolate the problem one boundary at a time:

1. Capture. Is the source audio timestamped correctly? Is microphone capture stable under load?

2. Transport. Are packets arriving with jitter, loss, or reordering? Does the offset change only after network events?

3. Decode. Are frames or audio samples queued longer than expected before decode?

4. Render. Are you hitting main-thread stalls or frame drops?

5. Service output. Is the avatar endpoint producing synchronized media, or are you assembling separate streams incorrectly on the client?

A good sanity test is to run the same session on a stable network, on a low-end device, and with a CPU load spike. If the problem only appears under load, the culprit is likely client-side scheduling rather than the avatar service itself.

// Curl example for a session-level diagnostic fetch or control call.
}'

// Curl example for a session-level diagnostic fetch or control call.
}'

// Curl example for a session-level diagnostic fetch or control call.
}'

Practical fixes that usually help

Once you know where the offset is introduced, the fixes are usually straightforward:

• Use timestamps, not arrival order. Arrival time is a transport artifact. Presentation time is what the user perceives.

• Keep buffering bounded. A queue that grows without a cap will eventually desynchronize the modalities.

• Render the newest coherent state. If the system is behind, catching up matters more than showing every intermediate frame.

• Stay off the main thread. Especially for decoding, compositing, and network event handling.

• Log offsets in production. The fastest way to find regressions is to compare audio/video delta before and after a release.

If you are using TTS or a voice agent upstream, make sure the speech timing fed into the avatar is the same timing used to schedule mouth animation. Many bugs are introduced when developers treat the avatar as a separate visual effect instead of part of the media pipeline.

Where Protoface fits

If your app needs a synchronized talking face for a LiveKit voice agent, the cleanest path is to let the avatar service own the avatar timing instead of trying to hand-roll it in the client. The LiveKit integration is designed for exactly that use case, and the relevant examples are in the plugin repository: https://github.com/protoface-ai/protoface-plugin-pipecat for the Pipecat integration, or the LiveKit plugin package if that is your stack. In both cases, the key is the same: keep the avatar aligned to the agent’s speech stream and avoid reconstructing lip motion from loosely related client events.

If you are building against the REST API or Python SDK, keep the session contract explicit: create the avatar/session, pass the voice and instructions you intend to use, and verify that your client does not introduce an extra playback buffer on top of the service’s own streaming cadence. The API and SDK docs are the right place for the exact request and response shapes: https://docs.protoface.com and the Python SDK repository: https://github.com/protoface-ai/protoface-sdk-python.

Conclusion

Lip sync drift is usually a timing bug, not a graphics bug. Treat it that way. Measure the offset between audio and video over time, determine whether the problem is stable or cumulative, and then fix the boundary where timing is being lost: capture, transport, decode, or render.

For most teams, the fastest route to a correct realtime avatar pipeline is to rely on a service that already understands the timing constraints of speech-driven video rather than building one from scratch. Start with the docs, instrument your app, and verify the alignment under real network and device conditions before you ship. If you need implementation details or quickstarts, docs.protoface.com and the linked repos are the right next stops.