Introduction

Realtime avatars are mostly a synchronization problem. The hard part is not “making a face move”; it is keeping the video, audio, and turn-taking aligned closely enough that a user perceives one coherent speaker instead of a slightly off, uncanny pipeline. If your avatar lags behind the audio, overreacts to noise, or clips consonants, the model quality is usually not the first thing to blame. The transport, buffering, timestamping, and resampling path matter just as much.

This post focuses on the practical side: how to improve lip-sync and audio quality in a realtime avatar pipeline, what to measure, and where the common failure modes come from. By the end, you should be able to reason about latency budget, align audio to generated mouth motion, and make better engineering trade-offs when integrating a voice agent with a talking face.

Start with the timing model, not the avatar model

Most realtime avatar systems are some variation of this flow:

speech input or agent text -> TTS or audio generation -> chunked audio streaming -> video frame synthesis or animation drive -> network transport to the client.

The important detail is that audio and video are produced and consumed in different chunk sizes, on different clocks, and often by different services. Lip-sync quality is therefore determined by how well you preserve timing metadata across the pipeline.

Three timing properties matter most:

• Capture-to-render latency: time from when the agent starts speaking to when the user sees the first mouth movement.

• Steady-state skew: the average offset between audio phonemes and visible mouth shapes once streaming is underway.

• Jitter: how much that offset varies over time. Small average error with high jitter still looks bad.

In practice, users tolerate a small fixed delay better than fluctuating desync. If the avatar is consistently 120 ms late but stable, it often feels more natural than an avatar that oscillates around perfect alignment.

Keep audio deterministic and bandwidth-friendly

Audio quality problems in avatar pipelines usually come from one of four places: sample-rate conversion, codec artifacts, clipping, or buffer underflow/overflow. The obvious fix is not always “use a higher bitrate.” Higher bitrate can help, but only if the audio is already clean and your pipeline preserves it.

Use a single canonical sample rate

Pick one sample rate end to end, then resample exactly once if you have to. Multiple resampling stages introduce artifacts and can subtly shift timing. For realtime voice agents, 16 kHz is common for telephony-oriented speech; 24 kHz or 48 kHz may be preferable if your system and transport support it and you care about richer timbre.

Where developers get into trouble is letting one service emit 22.05 kHz PCM, another expect 16 kHz, and the browser play 48 kHz. Every implicit conversion adds room for drift.

Prevent clipping before it reaches the transport

If the synthesized voice is too hot, clipping destroys high-frequency consonants and makes mouth motion feel “blurry” because the audio features used for alignment are less distinct. Normalize or limit at the source, and leave headroom for dynamic peaks. Realtime voice agents often sound better with a conservative target loudness than with aggressive normalization.

Useful checks:

• Peak levels should not routinely hit 0 dBFS.

• Transient consonants should remain intact after encoding.

• Silence segments should stay truly silent; noisy floor can confuse turn detection.

Prefer chunk sizes that balance latency and stability

Smaller audio chunks reduce startup latency, but if they are too small you can increase packet overhead and expose the client to jitter. Larger chunks are more stable but make mouth motion feel late. In most realtime systems, the right answer is to stream audio continuously in moderate chunks and rely on a jitter buffer rather than waiting for large blocks.

The practical lesson: if your avatar “speaks in bursts,” the issue may be buffer design, not the model. Check whether the downstream consumer is waiting for too much audio before beginning playback or animation.

Align mouth motion to audio, not to text

Text is not a reliable proxy for lip motion. Phoneme timing depends on prosody, stress, punctuation, pauses, and synthesis decisions. If your animation is driven by text tokens or raw character counts, you will eventually see the mouth move before the sound or continue moving after the word ended.

Better approaches derive animation timing from one of these:

• Audio features: energy, spectral cues, and speech activity derived from the actual waveform.

• Phoneme timing: timestamps from TTS or speech alignment metadata.

• Hybrid controls: text-level intent for expression plus audio-level signals for timing.

Watch for the “plosive problem”

Plosives like p, b, and m are where lip-sync bugs become most obvious. If the jaw opens too early, users notice. If the mouth closes too late, the avatar looks rubbery. When evaluating a pipeline, use test phrases rich in plosives and fricatives:

“Please bring Bob a blue paper bag.”

“Please bring Bob a blue paper bag.”

“Please bring Bob a blue paper bag.”

That sentence is useful because it exercises closed-mouth transitions, bilabial closure, and sharp consonant boundaries. If your system handles that cleanly, it usually handles ordinary speech well enough.

Measure, then tune the right buffer

You cannot improve realtime lip-sync by intuition alone. Log timestamps at each stage:

• speech or text generation start

• first audio chunk produced

• first audio chunk received by the client

• first frame rendered with mouth motion

• turn end / silence detection

From those, compute the gaps that matter. A common mistake is tuning the renderer when the network transport is the real bottleneck. Another is reducing video frame rate when the problem is actually audio jitter. The point of instrumentation is to determine where delay accumulates.

Trade-offs that usually matter in production

Once the pipeline works, the remaining choices are about consistency and user perception:

1. Lower latency vs. higher quality: faster turn start is often more important than perfect fidelity, but only to a point. If the avatar sounds synthetic or the lips are visibly out of sync, the experience degrades quickly.

2. Animation richness vs. robustness: expression layers, eye motion, and head movement make the avatar feel alive, but they also amplify timing mistakes. Start with stable mouth motion, then add expressiveness.

3. Server-side vs. client-side smoothing: server-side control gives you predictable timing, while client-side smoothing can help absorb jitter. Too much smoothing, though, makes the avatar feel sluggish.

In other words, do not optimize for the prettiest demo frame. Optimize for the least noticeable end-to-end behavior under real network conditions.

How Protoface fits in a LiveKit voice agent pipeline

This is where a realtime avatar layer can save you from building the synchronization plumbing yourself. If you are already using LiveKit for voice agents, the LiveKit-oriented quickstart examples show the general integration pattern, and the PyPI plugin pipecat-protoface demonstrates how to drop an avatar into a speech pipeline without inventing your own lip-sync transport.

The core idea is simple: your agent already emits voice; the avatar layer consumes that realtime speech stream and keeps a synchronized face on top of it. That lets you focus on the two things that actually move the needle: clean audio and stable timing. Exact initialization fields and session options are documented in the docs, but the implementation pattern is usually just “attach the avatar service to the existing voice flow and keep your audio path clean.”

from protoface import ProtofaceClient

from protoface import ProtofaceClient

from protoface import ProtofaceClient

If you are integrating at the agent layer rather than directly with an API, the same principle applies: keep the agent’s audio output stable, avoid extra resampling, and let the avatar service handle the synchronized presentation. That is usually much easier than trying to stitch face animation into a custom media stack after the fact.

Common failure modes and how to debug them

When an avatar looks “off,” the root cause is often one of these:

• Audio starts late: jitter buffer too large or first packet delayed by upstream synthesis.

• Mouth is always behind: fixed latency introduced by encoding, transport, or client-side buffering.

• Mouth stutters: unstable chunk arrival or frame scheduling.

• Speech sounds harsh: clipping or aggressive codec settings.

• Avatar overtalks silence: poor silence detection or a noisy background signal.

Debugging tip: isolate one dimension at a time. Test raw audio playback first. Then add the avatar layer with the simplest possible motion profile. Then introduce real agent turn-taking. If you start with the full stack, every issue looks like everything else.

Conclusion

Good lip-sync is mostly disciplined realtime engineering: consistent sample rates, stable buffering, measured latency, and animation driven by the actual speech signal rather than by text alone. If your audio is clean and your timing is predictable, the avatar will usually look much better even before you touch the visual model.

If you are building a voice agent, start by instrumenting the pipeline, then tighten the audio path, then tune the synchronization behavior. For implementation details, examples, and the supported integration surfaces, see docs.protoface.com and the relevant repositories linked above.