Debugging Latency and Lip-Sync Drift in Realtime AI NPCs for iOS
Debug iOS realtime AI NPC latency and lip-sync drift with timestamp tracing, audio-clock sync, and streaming fixes.
Introduction
When a realtime AI NPC feels “off” on iPhone, it is usually not one problem. It is a stack of small timing errors: TTS starts late, video frames are delivered irregularly, the browser buffers too aggressively, or the avatar renderer and audio clock drift apart over a multi-turn conversation. The result is the same to the user: the face talks a beat behind the voice, mouth shapes miss phonemes, or the entire interaction feels sluggish even when the backend is healthy.
This post is about debugging that class of issue systematically. By the end, you should be able to identify where latency is being introduced, distinguish network delay from rendering delay, and make practical changes that reduce lip-sync drift in iOS clients. I’ll also show where a realtime avatar layer like Protoface fits into the pipeline without changing your agent architecture.
Start by measuring the pipeline, not the symptom
“Latency” in a voice-driven avatar is usually the sum of several independent stages:
Speech recognition or user input capture
LLM inference or agent planning
TTS or audio synthesis
Avatar generation / lip-sync video generation
Transport over WebRTC, WebSocket, or HTTP-based streaming
Decoding and rendering on-device
If you only measure “time until the face starts moving,” you cannot tell which stage is responsible. Instead, instrument each boundary with timestamps. At minimum, capture:
t_user_speech_endt_agent_first_tokent_tts_first_audiot_avatar_first_framet_client_rendered
On iOS, use monotonic clocks for client-side measurements. Do not compare wall-clock timestamps across devices and expect meaningful latency numbers; clock skew and NTP adjustment will ruin that data. If you need end-to-end timing, propagate a request ID through the agent, TTS, avatar service, and client logs, then compare relative deltas inside each subsystem.
Understand where lip-sync drift actually comes from
Lip-sync problems are not always “bad sync.” In practice, they come from one of four sources:
Audio-video timeline mismatch. Audio packets are rendered based on one clock, video frames on another, and the client does not reconcile them.
Frame pacing jitter. The server emits video in bursts, or the client receives frames unevenly and displays them as fast as possible instead of preserving cadence.
Queue buildup. The system keeps generating frames after the speech segment has moved on, so the avatar is always lagging behind the current utterance.
Model-level delay. The avatar generator waits for too much context before producing a stable mouth pose, which increases initial latency even if later sync looks fine.
The practical consequence is that “fixes” aimed only at the renderer often miss the real bottleneck. If your first frame arrives 700 ms late, shaving 20 ms off client decode won’t matter. Conversely, if generation is fast but the client buffers 1 second of media before playback, the server can be perfect and the user will still see a delayed face.
Debug latency in the order users perceive it
Users perceive three milestones: acknowledgment, first audio, and first visible mouth motion. Debug them in that order.
1. Time to acknowledgment. This is the pause before the system “reacts” at all. For voice agents, it often corresponds to the time to first assistant token or first partial action. If this is high, profile your agent loop, not the avatar layer.
2. Time to first audio. If your TTS engine starts slowly, the avatar can’t lip-sync anything yet. Some TTS systems stream early phonemes; others buffer more aggressively for quality. Know which mode you’re in. Streamed audio usually reduces perceived latency more than trying to force earlier video.
3. Time to first visible mouth motion. This is where the avatar pipeline matters. Check whether your client waits for a minimum buffer before starting playback. A conservative buffer improves smoothness but increases initial delay. On mobile, that trade-off is especially visible because network conditions vary and the OS scheduler is less forgiving when the app is backgrounded or under load.
In practice, I recommend tracing both the server and the client with the same interaction ID and logging these specific events:
That last event can be manual at first; you do not need computer vision to start. Just knowing whether the render path or the upstream generation path is responsible gets you 80% of the way.
Reduce drift by making audio the clock source
For realtime avatars, audio should generally be the master timeline. Human perception is much more sensitive to audio timing than to tiny visual deviations, and lip movement should follow phonetic timing rather than compete with it.
That means your client should:
Buffer audio and video independently, but begin playback based on a coordinated start point.
Use the audio clock to drive visual presentation whenever possible.
Drop or skip stale video frames rather than trying to display everything in order.
Avoid unbounded queues. If the renderer falls behind, latency grows without bound and the avatar appears “possessed” by the previous sentence.
On iOS, the cleanest implementation is usually to treat the media pipeline as a live stream, not a file. If you are using AVFoundation or a WebRTC stack, verify that frame presentation timestamps are respected and that you are not accidentally re-timestamping incoming frames based on arrival time. Arrival time is network noise; presentation timestamp is the schedule.
Another common issue is audio buffering that is too deep. A large buffer hides jitter, but it also increases end-to-end delay and makes the avatar look disconnected from the current utterance. If your network is stable, reduce the buffer until you start seeing underruns, then back off slightly. The right number is environment-dependent, which is exactly why you should test on a real iPhone over real networks, not just in the simulator.
Make the server stream early and stop on time
On the generation side, the two biggest wins are simple:
Start early. Begin emitting usable media as soon as the system has enough context to do so. For conversational avatars, the initial expression and mouth motion do not need to wait for a fully polished response. Partial output is often better than a perfect output that arrives late.
Stop promptly. Once the turn ends, the server should stop generating stale frames. If the backend continues producing “tail” frames after the speech has effectively completed, the client will keep playing old motion and the next turn will feel delayed.
This is especially important in multi-turn conversational agents where the LLM may revise or extend its response midstream. The avatar layer needs a clear notion of the active utterance boundary. If your agent emits cancellation or interruption events, propagate them all the way down to media generation.
When you are diagnosing the server, inspect whether your media stream has a bursty pattern. A smooth stream should look like a cadence of regular frame and packet arrivals, not large clumps separated by silence. Bursts can be caused by batching, thread starvation, or a queue that only flushes on size thresholds instead of time thresholds.
Where Protoface fits
In a LiveKit-based voice agent, the most practical place to add a talking face is at the agent boundary: the assistant already owns the turn-taking and audio stream, and the avatar layer only needs to stay synchronized with that stream. The LiveKit plugin published as livekit-plugins-protoface is designed for that use case, and the same underlying avatar/session model is also exposed through the REST API and Python SDK.
For example, you can keep your agent logic in Python and attach an avatar surface without changing the rest of the voice stack. The exact session fields depend on the docs, but the shape looks like this:
If you are integrating through LiveKit, the useful part is not the API surface itself; it is that the avatar is driven by the same conversational turn that drives audio, which makes it much easier to reason about sync. For setup details, the docs are the right place to confirm the current fields and event names: docs.protoface.com.
iOS-specific gotchas that look like backend problems
A surprising number of “server latency” bugs are actually client issues:
Backgrounding. When iOS suspends your app, rendering and networking behavior changes. If your tests include app switching, expect timing shifts.
Main-thread contention. If you decode, layout, or update UI on the main thread, you can introduce visible stutter even when media arrives on time.
Thermal throttling. Older devices can drop frames under sustained load. What looks like drift may just be the device failing to keep up.
Over-aggressive buffering. Some client code “fixes” jitter by buffering more. That works until the conversation gets interactive and the avatar is now a full second behind the user.
If you need a quick sanity check, test on three conditions: strong Wi-Fi, moderate LTE, and poor-but-functional cellular. If sync only fails under poor network conditions, the issue is probably buffering strategy, not generation correctness. If sync fails everywhere, inspect the media timeline and rendering path first.
Conclusion
Realtime avatar latency is best debugged as a pipeline problem, not a single metric. Measure each stage, make audio the timing reference, keep queues short, and verify that the client respects presentation timestamps instead of arrival order. For iOS, remember that the simulator is not a reliable proxy for device behavior.
If you are building this on top of a voice agent, the easiest path is to integrate the avatar where the turn is already known and the audio stream already exists. The docs at docs.protoface.com cover the current API details, and the GitHub examples are useful when you want to see a working integration rather than a conceptual diagram. Once you have the timestamps, the fix usually becomes obvious.