Introduction

When a realtime AI avatar looks “off,” the cause is usually not the model itself. It is almost always a media pipeline issue: the audio clock and video clock drift apart, the encoder makes the video too aggressive for the available bandwidth, or the SFU is forwarding streams that were never tuned for this delivery path.

If you are shipping voice agents with talking faces, you need two things to be stable at the same time: lip-sync and bitrate. Lip-sync is a timing problem. Bitrate is a transport problem. They interact, because when video gets starved or rebuffered, the visual timing can become visibly wrong even if the avatar animation is mathematically “correct.” By the end of this post, you should be able to reason about where the drift comes from, how to diagnose it, and how to keep an SFU-delivered avatar looking synchronized under real network conditions.

What “lip-sync” actually means in a realtime avatar pipeline

In a conversational avatar system, audio is the source of truth for timing. The speaking model, TTS, or voice agent emits audio frames on a well-defined timeline. The avatar video is then generated or rendered to match that timeline. In a WebRTC system, audio and video are usually separate tracks, but they must share a common notion of playout time.

There are three common failure modes:

• Clock drift: audio and video are produced by different components with slightly different timing assumptions. Over time, the offset accumulates.

• Jitter and buffering: the network or SFU delivers packets with variable delay. If one track is buffered longer than the other, lips can move late or early relative to speech.

• Encoder-induced latency: video compression settings can add enough delay that the avatar is always behind the audio, even though the source animation is aligned.

For human perception, audio lead is usually tolerated better than audio lag. If the lips are behind the voice, the mismatch is obvious. If the lips are slightly ahead, the brain is more forgiving. That does not mean you should bias the system to lead; it means your debugging priority should be eliminating systematic video lag first.

Why SFU delivery makes bitrate tuning more important, not less

An SFU does not “fix” your media; it routes it. That matters because avatars often sit at the awkward intersection of low-motion video, high perceptual sensitivity, and bursty conversational audio.

Unlike camera video, an avatar face has a lot of compressible static content: skin, background, hair, and only a small region with motion. That can tempt you to drive bitrate down aggressively. The problem is that lip motion lives in a small spatial area, so when the encoder is starved, the first visible artifact is often mouth detail. The result is not merely uglier video; it can look out of sync because the mouth shapes become smeared or delayed across frames.

Bitrate tuning for avatars should optimize for:

• Temporal consistency: avoid quality settings that cause frequent keyframe pressure or large frame-to-frame variance.

• Face-region fidelity: preserve mouth and jaw detail, even if the background can be softer.

• Predictable latency: a stable 25 fps stream at moderate bitrate is usually better than an occasionally beautiful stream that spikes and stalls.

In practice, this means avoiding “highest quality possible” defaults and instead choosing a tier that matches your network envelope and user expectations. A realtime avatar is judged by responsiveness first, resolution second.

How to debug lip-sync problems systematically

Start by separating generation issues from transport issues.

1. Verify the source timeline. If the audio is generated from TTS or a voice agent, confirm that the audio start time and the animation start time are derived from the same session event. If your stack emits the avatar video only after the first audio packet arrives, you can end up with a permanent offset that looks like “the avatar is always late.”

2. Measure end-to-end latency per track. Do not rely on eyeballing. Inspect the RTP/RTCP metrics, local render timestamps, and server-side timestamps if you have them. You want to know:

• How long after audio capture or synthesis does the first audio packet reach the client?

• How long after video generation does the first frame reach the client?

• Does the offset stay stable or drift over time?

3. Check for render queue buildup. A surprisingly common issue is the client getting video frames on time but rendering them late because the browser main thread or the player pipeline is overloaded. If you decode video on one thread and render on another, a backlog can create apparent lip lag even when network delivery is fine.

4. Confirm that audio is the pacing source. In mixed realtime systems, the video pipeline should generally follow the audio clock, not the other way around. If the avatar animation is driven by “wall clock plus frame count” while the audio uses actual playout timestamps, they will eventually diverge.

Practical bitrate controls that usually matter

There is no universal bitrate number for avatar video. The right value depends on resolution, frame rate, motion complexity, codec, and the amount of network congestion. Still, a few rules hold up well.

Keep the frame rate stable. For conversational avatars, a steady frame rate is more important than chasing high fps. A stable 24–30 fps stream with proper pacing is easier to sync than a variable stream that sometimes bursts and sometimes drops.

Prefer moderate resolution over oversized frames. The face occupies most of the frame in many avatar UIs, so you get little benefit from sending a much larger image than the viewport actually needs. Oversized frames waste bandwidth and increase encoder cost.

Reserve budget for motion detail. If the pipeline allows quality tiers or encoder profiles, choose settings that preserve small facial motions instead of maximizing background sharpness. The viewer notices mouth shape errors long before they care about subtle texture loss.

Expect network adaptation. In an SFU environment, congestion control can lower bitrate when bandwidth is constrained. That is good, but it means your system should degrade gracefully. A well-designed avatar stream can keep audio intact while allowing video to soften slightly, rather than allowing both tracks to become unstable.

Here is a simple example of the sort of API flow you would use to create a session programmatically. Exact request fields depend on the API version, so treat this as illustrative and check the docs for the current schema:

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

Where the LiveKit plugin fits

If your app already uses LiveKit for voice agents, the fastest path is the LiveKit Agents plugin from the plugin repository or the published package on PyPI. The important architectural point is that the avatar becomes another media participant in the agent session, so the audio and video can be paced together instead of being bolted on later.

That reduces the number of places where timing can drift. Instead of synthesizing audio in one subsystem and attaching a face in another, the plugin integrates the avatar into the same realtime agent flow. For developers, that usually means fewer “why is the mouth behind the voice?” surprises and less custom glue around track publication.

from livekit.plugins.protoface import ProtofacePlugin

from livekit.plugins.protoface import ProtofacePlugin

from livekit.plugins.protoface import ProtofacePlugin

If you are using Pipecat instead of LiveKit directly, the same general idea applies. The avatar service sits in the media graph as a realtime video producer, and the integration handles the timing and transport details for you. See the Pipecat guide for the integration shape: Pipecat Protoface service reference.

Operational gotchas that cause “it works in dev, fails in prod”

Browser autoplay and audio unlock rules. If the client starts video playback before the audio path is unmuted or user-initiated, the browser may buffer or pause one side. That can create a sync issue that only appears on certain browsers or device classes.

Different network paths for different users. A controlled test on a wired office network can hide congestion-control behavior that appears immediately on mobile or home Wi-Fi. Realtime avatars should be tested on a real range of uplinks and downlinks, not just on localhost or a datacenter link.

Over-aggressive bitrate caps. Teams sometimes cap video to “save cost” and then wonder why the avatar looks mushy during speech. If the mouth region collapses under compression, the product feels broken even if the transport metrics look healthy.

Misaligned session lifetimes. If the video session starts, pauses, or reconnects independently from the voice session, you can accumulate offset on resume. Make reconnect behavior explicit: either re-anchor both tracks to a fresh session boundary or restore both from the same timing state.

How Protoface approaches this in practice

Protoface is built to keep the avatar side of the media pipeline boring, which is what you want in production. For developers using the REST API or Python SDK, the session model is explicit, so you can create avatars and realtime sessions programmatically and keep timing and identity tied to a concrete session boundary. For LiveKit users, the plugin integrates the avatar into the voice-agent flow rather than asking you to splice a separate video system in after the fact.

The useful part here is not the brand name; it is the integration point. If the avatar service is already aligned with your voice agent’s realtime timeline, you avoid a large class of synchronization bugs. And if you need to inspect or manage sessions, the developer dashboard gives you a place to look at usage, keys, and active sessions without spelunking through logs first. The public docs at docs.protoface.com cover the current API and integration details.

Conclusion

Most lip-sync issues in SFU-delivered avatars are timing problems wearing a video-compression costume. Debug them by checking the audio clock, the video clock, and the buffer between them. Most bitrate issues are not about raw quality; they are about keeping mouth motion legible and latency predictable under real network conditions.

If you keep audio as the pacing source, keep video bitrate conservative but not starved, and verify the behavior end to end instead of at a single layer, you will eliminate most of the visible sync failures people associate with “realtime avatars.” For implementation details, start with the docs, then wire up the surface that matches your stack: REST API, Python SDK, or the LiveKit plugin.