How to Sync TTS Timing with Avatar Animation for Natural Lip-Sync
Learn how to sync streaming TTS with avatar animation using one clock, visemes, and turn-state timing for natural lip-sync.
Introduction
Natural lip-sync is mostly a timing problem, not a “make the mouth move vaguely with speech” problem. If your avatar animation lags the audio by 200 ms, or starts before the first phoneme is actually spoken, users notice immediately. The same is true if pauses, prosody changes, and turn boundaries don’t line up with what the face is doing.
In practice, you need to synchronize three streams of information:
the synthesized audio
the animation timeline used to drive mouth shapes and facial motion
the conversational state, so the avatar knows when it is actually speaking versus listening
By the end of this post, you should be able to reason about where timing drift comes from, how to align TTS with avatar animation, and how to build a pipeline that stays stable in a realtime agent instead of only looking good in a prerecorded demo.
What “good lip-sync” actually means
For realtime avatars, lip-sync quality is usually dominated by four things:
Onset timing: the mouth should start moving close to the first audible speech energy, not when the text was generated.
Phoneme/viseme alignment: the mouth shape should track speech units with the right delay and duration.
Prosody awareness: stressed syllables, pauses, and sentence endings should affect facial motion, not just the jaw.
Stream continuity: the face should not “reset” between chunks unless the audio stream actually resets.
The key point is that text-to-speech is not a single event. Most systems emit audio incrementally, sometimes with timing metadata, and avatar animation must follow that stream rather than a static transcript.
If you are building a voice agent, treat speech as a timeline with at least two layers:
coarse turn state: speaking, listening, interrupting, idle
fine-grained timing: visemes, jaw motion, blink cadence, head motion
Start with a shared time base
The simplest way to get bad lip-sync is to let audio and animation use different clocks. One subsystem measures time from when the text was submitted, another from when the first audio packet arrived, and a third from when the WebRTC sender began transmitting. Drift is inevitable.
Instead, pick one authoritative timeline for the speaking turn. In realtime systems, that is usually the audio playback or media capture clock, because it reflects what the user actually hears. Every animation event should be scheduled relative to that clock.
There are two common patterns:
Audio-driven animation: infer visemes from the TTS audio stream, then render animation slightly ahead of playback to account for buffering.
Metadata-driven animation: use TTS-provided alignment events, such as phoneme timestamps, and map them to facial blend shapes.
Audio-driven is more robust when the TTS engine does not expose exact timing. Metadata-driven is usually better when the TTS system can provide reliable alignment. In both cases, you still need to compensate for transport latency, decoding delay, and any avatar rendering delay.
Align animation to the first audible frame, not the request time
A common mistake is to kick off facial motion as soon as the model finishes generating text. That works for a pre-rendered animation clip, but not for streaming TTS. The better trigger is the point where speech becomes audible or is about to become audible after buffering.
A practical approach:
buffer the first audio frames until you have enough to start playback smoothly
record the local playback start timestamp
start mouth motion using the same timestamp, with a small lead if your avatar renderer needs it
The lead time matters. If your renderer needs 80 ms to apply viseme changes, animation events should be scheduled 80 ms earlier than the visual frame you want the user to see. This is not cheating; it is how you align control signals to a display pipeline with latency.
For example, if the TTS engine emits phoneme timing for a syllable at 1.42 s after playback start, and your rendering path adds 60 ms, schedule the viseme transition at 1.36 s in local time.
Use visemes as the control surface, not raw phonemes
Phonemes are linguistically precise, but most facial rigs are not. You usually want to map speech into a smaller set of mouth shapes, or visemes, that your avatar can render consistently. That mapping reduces jitter and makes timing less sensitive to tiny alignment errors.
Typical implementation details:
group phonemes into a viseme set compatible with your rig
blend between visemes over short windows rather than snapping immediately
cap rapid switching to avoid “mouth chatter” on noisy alignments
apply separate timing for jaw open/close versus lip rounding and closure
For realtime systems, it is often better to slightly under-animate than to overfit every phoneme. Users are very sensitive to phase errors, but they tolerate modest simplification in the mouth shape itself.
Also, do not drive the mouth from the transcript alone. Two identical sentences can sound different depending on emphasis, speaking rate, and pauses. If the TTS engine can expose timing or audio features, use them.
Handle turn-taking and interruptions explicitly
In voice agents, lip-sync problems often come from turn management rather than animation code. If the assistant is interrupted, or if the user starts speaking while the avatar is still finishing a sentence, the face must change state immediately.
At minimum, maintain these transitions:
idle → speaking: start mouth motion when audio is actually about to play
speaking → listening: decay mouth motion and return to a neutral expression quickly
speaking → interrupted: cut off animation as soon as the audio is stopped
Do not wait for the current animation segment to finish. Real conversation is full of mid-sentence cuts, and a good avatar needs to reflect that.
If your pipeline supports streaming TTS, send shorter chunks with explicit boundaries only where you can preserve continuity. Overly aggressive chunking can create visible pauses between phrases, while giant chunks make interruption handling sluggish.
Practical debugging: find the source of drift
When lip-sync looks off, the bug is usually one of a few things:
clock mismatch: different subsystems are using different timestamps
buffering delay: audio starts later than animation
tagger lag: viseme or phoneme events arrive late from the TTS pipeline
render backlog: the animation system is overloaded and frames are dropped
bad chunk boundaries: the speech stream was split in a way that destroys timing continuity
The fastest way to debug is to log three times for every speaking turn: request time, audio playback start, and the timestamp of the first visible mouth movement. Then measure the delta between them over several turns. If the delta is stable but wrong, your offset is wrong. If it changes over time, you have drift or load-related backlog.
It also helps to inspect audio and animation separately. If the audio sounds smooth but the mouth is late, the bug is in timing alignment. If the mouth is smooth but speech sounds broken, the issue is in TTS chunking or transport.
How Protoface fits into the pipeline
This is where a realtime avatar platform is useful: you do not want to build the media plumbing, session orchestration, and face rendering stack from scratch just to discover you still need to solve synchronization.
With Protoface, the relevant integration point for many voice-agent stacks is the LiveKit plugin, livekit-plugins-protoface. The idea is simple: your agent keeps owning the speech pipeline, while the plugin injects a synchronized avatar video face into the session. In other words, you focus on the turn state and speech generation; the avatar surface handles the media side of presenting that speech as a talking face.
For teams that want to wire things up programmatically, the REST API and Python SDK expose session and avatar management, with auth handled by API keys. The exact request fields and session shapes are documented in the API reference, but the rough pattern is the same: create or select an avatar, start a realtime session, then attach it to your agent or embed flow.
If you are integrating at the media layer, keep one rule in mind: the avatar should follow the same speech turn events as the audio pipeline. Do not let the face infer its own speaking state from arbitrary timers.
Example: streaming TTS with aligned animation events
Here is the shape of a sane implementation, independent of vendor specifics:
In code, that usually means:
queue audio chunks into a jitter buffer
when playback actually begins, set
T0translate any phoneme timestamps into local animation times using
T0render visemes with a short lookahead if the frame pipeline needs it
If your TTS provider returns alignment metadata, use it. If it does not, infer coarse speech activity from the audio envelope and use a conservative mouth-open curve instead of pretending you know exact phonemes. That still looks better than unsynchronized animation.
For developers working in Python, the SDK is the most direct way to automate avatar/session setup. For agent frameworks, the plugin approach keeps the timing responsibility closer to the voice stack, which is usually where it belongs.
Conclusion
Good lip-sync is mostly about disciplined timing: one clock, one speaking turn, and animation events that track the audio actually heard by the user. The more streaming and realtime your system is, the less useful “start the mouth when text is ready” becomes.
If you are building a voice agent or interactive avatar, start by instrumenting your audio start time and animation start time, then work backward from there. Use visemes, not raw text, as the animation control surface. Handle interruptions explicitly. And keep the media path close to the speech pipeline so you can reason about latency instead of guessing at it.
For implementation details, examples, and current API shapes, check the docs and the relevant GitHub quickstarts linked from the project README. That will get you from “it talks” to “it feels synchronized” much faster than trying to tune the mouth in isolation.