How to Keep Lip-Sync Stable While Scaling AI Avatars with Twilio
Learn timestamp-based lip-sync, jitter buffering, barge-in handling, and scaling patterns for AI avatars with Twilio/LiveKit.
Introduction
When an AI avatar starts talking over a realtime audio stream, lip-sync quality is mostly a timing problem. The face is only “right” if the mouth shape tracks the audio pipeline closely enough that the viewer never notices drift, jitter, or delayed closure at phrase boundaries. That becomes harder as you scale from a single demo to many concurrent sessions: more agents, more network hops, more event loops, more opportunities for timestamps to get out of alignment.
This post is about keeping lip-sync stable under real production conditions. By the end, you should be able to reason about where synchronization breaks down, what to measure, and which architectural choices keep an avatar’s mouth movement locked to speech even as you scale voice agents and interactive sessions. I’ll also show where Protoface fits as a developer-facing avatar layer for these systems.
What actually causes lip-sync drift
For a realtime avatar, “lip-sync” is not a single thing. It’s the composition of at least three clocks:
Audio generation time: when the TTS or voice agent emits audio frames.
Transport time: WebRTC, RTP, or another stream moves those frames to the client.
Avatar animation time: the renderer decides which visemes or mouth shapes to show for each audio segment.
If those clocks are not anchored to a common timeline, you get classic failures: the mouth opens late on sentence starts, overhangs after a phrase ends, or “swims” when network jitter causes a burst of late audio.
The most common mistake is treating audio arrival time as the truth. It isn’t. Arrival time includes network jitter, buffering, and scheduler noise. What you want is a stable mapping from audio timestamps to animation state, with enough buffering to smooth jitter but not so much that the avatar feels laggy.
Build around timestamps, not packet arrival
Stable lip-sync starts with timestamped audio. Whether your stack uses a voice agent, TTS service, or speech-to-speech pipeline, the avatar should consume a stream that carries timing metadata, not just raw samples.
Practically, that means:
Generate audio in fixed-size frames or chunks.
Attach monotonically increasing presentation timestamps.
Render visemes/mouth shapes against that timeline.
Buffer just enough to absorb jitter, then play out steadily.
If you are integrating with WebRTC, remember that audio jitter buffering already exists in the media stack. Your avatar layer should not fight it by making independent assumptions about “now.” Instead, it should anchor animation to the same playout clock the audio path uses, or to a derived timeline that advances with the audio buffer.
A useful mental model is this: the avatar should follow audio playout, not audio production. Production can burst, stall, or reorder under load. Playout should be smooth.
Control latency budget before you chase perfect mouth shapes
Developers often over-focus on the mouth model and under-focus on end-to-end delay. In practice, a slightly simpler viseme model with a tight latency budget looks better than a high-fidelity model that arrives late.
Your latency budget should include:
LLM or dialogue planning time
TTS synthesis time
Audio encoding and transport
Avatar frame generation and rendering
Once that budget gets too large, the avatar starts talking “after the fact.” Users notice that more than they notice a less detailed mouth shape. For conversational agents, aim for consistency over maximum detail. A stable 150 ms delay often looks better than a variable 50–250 ms delay.
Two specific trade-offs matter:
Buffer depth: deeper buffers reduce jitter but add latency.
Animation aggressiveness: highly expressive mouth motion can exaggerate timing errors; more conservative motion is often more robust.
Handle interruption and barge-in explicitly
Realtime avatars are not just “talking video.” In a real product, the user interrupts, the agent stops mid-sentence, and the mouth must stop cleanly. If your system doesn’t support barge-in, the avatar may keep articulating stale audio for a few hundred milliseconds, which looks broken even if the backend is healthy.
The practical pattern is:
Keep a session-level cancellation signal.
Stop new animation as soon as the agent is interrupted.
Flush or fade the current audio segment rather than letting it drain unpredictably.
Make the next speaking turn begin from a clean neutral pose if the previous utterance was cut off.
That last step matters. If the avatar is interrupted while the mouth is wide open, snapping to a neutral resting pose can be visually harsh. A short transition is usually better than a hard cut, but the transition must be bounded; otherwise you create the appearance of sync loss.
Scale sessions without scaling inconsistency
When you go from one demo session to hundreds, problems usually show up as variance, not absolute failure. One session is fine; the next one has a 300 ms stall because it landed on a crowded worker, hit a cold cache, or suffered a transient WebRTC negotiation delay.
The remedy is to make the avatar pipeline stateless where possible and session-scoped where necessary:
Stateless rendering workers are easier to autoscale and restart.
Session-scoped state should hold only what is needed for the current conversation: speaking turn, animation timeline, and interruption state.
Backpressure should be explicit. If the system can’t render in real time, it is usually better to skip to the latest valid frame than to queue stale frames.
For lip-sync, stale frames are the enemy. A slightly dropped intermediate mouth shape is almost always less noticeable than a late frame that shows up after the audio has already moved on.
Instrument the right metrics
If you are operating avatar sessions at scale, don’t rely on “looks okay” as your test. Track the timing signals that correlate with visible artifacts:
Audio-to-animation skew: difference between audio playout time and avatar pose time.
Jitter: variance in chunk arrival or playout timing.
Turn-start delay: time from agent intent to first visible mouth motion.
Turn-end tail: how long the mouth keeps moving after speech should have stopped.
In debugging sessions, I usually start by plotting these across a full utterance. If the skew is stable but offset, you have a calibration problem. If the skew wanders, you have buffering or scheduling noise. If the mouth trails only at turn boundaries, your stop/cancel path is the likely culprit.
How Protoface fits into the stack
Protoface is useful when you want to add the avatar layer without building the synchronization machinery yourself. For voice agents, the LiveKit integration is the cleanest path: the livekit-plugins-protoface plugin drops a talking face into a LiveKit agent so the agent can emit synchronized video alongside audio. The key benefit is that the avatar is integrated at the agent boundary, where timing information is already available, rather than bolted on later as a separate video effect.
If you are wiring this up in Python, the shape of the code is straightforward; exact parameters live in the docs, but the integration pattern looks like this:
For lower-level session management, the REST API lets you create and manage avatars and realtime sessions from your backend. Authentication uses API keys, so your server can keep secrets out of the browser. A minimal request looks like this:
The main operational point is that the session lifecycle belongs on the server. That is where you can enforce rate limits, pick quality tiers, manage interruption, and keep timing state consistent across retries or reconnects. If you prefer to script this, the Python SDK gives you the same control surface from code.
Practical checklist for stable lip-sync
Before shipping, I’d verify the following:
Audio and animation share a common timestamp source.
Jitter buffering is present, but bounded.
Turn interruption cancels both audio and mouth motion immediately.
Late frames are dropped rather than queued.
Session state is isolated so one slow conversation does not affect others.
You measure skew and tail latency, not just end-user impressions.
If you get those right, the avatar will usually feel stable even under moderate load. If you get them wrong, no amount of visual polish will hide the timing issues.
Conclusion
Stable lip-sync is mostly an engineering discipline: timestamp everything, keep the latency budget under control, treat interruption as a first-class event, and measure the timing signals that users actually perceive. Once the pipeline is structured that way, scaling from one avatar session to many becomes an operations problem instead of a synchronization problem.
If you’re building on LiveKit or managing sessions directly, the docs at docs.protoface.com are the best place to confirm the exact API shapes and integration details. For a working starting point, the GitHub examples in the quickstarts are also worth skimming. The goal is simple: keep the mouth aligned with the voice, even when the system underneath it is busy.