Introduction

Low-latency avatar lip-sync is a systems problem, not just a media problem. If you want a talking face to feel responsive in a realtime voice agent, the pipeline has to keep end-to-end delay low and predictable across text generation, speech synthesis, transport, and animation. The common failure mode is not “the avatar is slow” in isolation; it is that each stage adds a little buffering, and the user ends up hearing the end of the sentence after they have already seen the mouth settle out of sync.

This post focuses on the engineering side of that pipeline: how to structure TTS for streaming output, how to align audio and facial motion without over-buffering, and what to watch for when you connect the system to a realtime agent. By the end, you should have a mental model for keeping lip-sync tight enough that the avatar feels live rather than animated after the fact.

Start with the latency budget

The first step is to stop thinking about “TTS latency” as a single number. What users experience is the sum of several smaller delays:

• Text availability: when the agent has enough text to start speaking.

• Audio generation startup: the time until the TTS service emits the first chunk of PCM or encoded audio.

• Transport delay: how long it takes to move audio to the client or media server.

• Playback buffer: the amount of audio queued before playout.

• Avatar motion delay: the frame or phoneme lookahead used for visual smoothing.

If you treat all of these as “safe buffering,” your lip-sync will drift. The goal is to minimize hidden queueing and keep one explicit, controlled buffer that the entire pipeline agrees on.

In practice, the best baseline is:

• stream text to TTS as soon as you have semantically stable chunks, not whole paragraphs;

• request streaming audio from the synthesizer, not a full file;

• forward audio in small frames with timestamps;

• let the avatar renderer consume the same timing model as the audio player;

• avoid adding another jitter buffer unless you absolutely need it.

The key idea is that the avatar should not infer timing independently from the audio path. The more the facial animation is tied to the actual audio timeline, the less likely it is to wobble under load or network jitter.

Stream the text, not the paragraph

For conversational systems, TTS should start before the model has finished composing a complete response. If your LLM emits tokens incrementally, you can batch them into short clauses and synthesize each clause as it stabilizes. That gets you a faster first byte and gives the avatar something to animate while the rest of the answer is still forming.

There are trade-offs. Smaller chunks reduce startup latency, but they also increase the risk of awkward prosody at chunk boundaries. Very large chunks sound natural, but you pay in delay. A practical compromise is to chunk at punctuation or clause boundaries and keep each chunk short enough that the TTS engine can begin quickly.

When you do this, pay attention to the point at which text becomes immutable. If the language model is still revising the sentence, do not send it to TTS prematurely unless your architecture tolerates barge-in and cancellation. Once audio has been emitted, you should treat it as committed; otherwise the visual and audible streams will diverge.

Use streaming audio with timestamps

Once TTS starts producing audio, do not collapse it into “just play this blob.” For realtime lip-sync, the important property is that each audio frame can be placed on a timeline. Whether you are using WebRTC, a live media server, or a browser audio pipeline, the client needs enough timing information to render speech and face motion together.

At a high level, the pipeline should look like this:

1. Text chunk arrives.

2. TTS begins streaming audio frames.

3. Frames are timestamped or sequence-numbered.

4. The avatar renderer maps audio progress to viseme or mouth-shape updates.

5. Playback and animation share the same clock or a tightly synchronized offset.

The exact mechanism depends on your stack, but the principle is constant: do not let the avatar “free-run.” If the face animation is driven by a separate timer, any small transport hiccup will show up as desynchronization. Instead, derive facial state from audio progression, even if the final viseme smoothing is computed locally.

How to avoid accidental latency inflation

Most lip-sync problems come from buffering that was added for a good reason and never revisited. Here are the usual suspects:

• Waiting for full sentences: easy to implement, bad for first response time.

• Overlarge TTS chunks: fewer requests, but slower start and more bursty audio delivery.

• Double buffering: one queue in the audio service, another in the media layer, each trying to be “safe.”

• Client-side smoothing with no timing source: makes motion look polished but not synchronized.

• Slow model handoff: if the LLM and TTS are coupled synchronously, TTS inherits the LLM’s tail latency.

A good way to debug this is to instrument each boundary separately: token availability, TTS first audio frame, first playout, and first visible mouth movement. If the numbers are close together but the experience still feels sluggish, the problem is probably a hidden buffer or a bad sync source.

Python example: start TTS as soon as you have a stable chunk

The code below shows the shape of a low-latency pipeline. The details vary by provider, but the pattern is what matters: stream text in chunks, start synthesis early, and keep the audio timeline explicit.

async def synthesize_and_stream(agent_text_stream, tts_client, avatar_session):

await avatar_session.send_audio(frame)

async def synthesize_and_stream(agent_text_stream, tts_client, avatar_session):

await avatar_session.send_audio(frame)

async def synthesize_and_stream(agent_text_stream, tts_client, avatar_session):

await avatar_session.send_audio(frame)

Two practical notes:

• If your TTS provider supports cancellation, wire it up. Users interrupt themselves constantly.

• If your synthesizer emits compressed audio, confirm the decoder does not introduce a large startup buffer on the playback side.

Where Protoface fits

Once you have a streaming voice agent, the remaining problem is to attach a face without introducing another sync layer. That is the role of Protoface. For LiveKit-based agents, the Pipecat integration and the LiveKit plugin make it straightforward to drop a synchronized avatar into an existing voice stack, so the agent’s speech and face stay aligned without you building a custom lip-sync renderer from scratch.

If you are building the flow programmatically, the Python SDK and REST API are the surfaces to look at. The SDK is useful when your application creates sessions or avatars as part of a backend workflow; the REST API is what you use when you want explicit control from your own service. The exact request and response fields are documented in the public docs, but the important architectural point is that avatar/session lifecycle stays server-managed, while the audio stream remains realtime.

import os

print(session)

import os

print(session)

import os

print(session)

For implementation details, the most useful references are the docs and the relevant integration repo: docs.protoface.com and the GitHub organization. If you are using Pipecat, the server-service reference is also worth reading because it makes the timing assumptions explicit.

Practical sync checks before you ship

Before you put a realtime avatar behind users, run a short checklist against your pipeline:

• Measure first-audio latency from token availability to synthesized frame availability.

• Measure visual onset from the same token boundary to first mouth motion.

• Test interruption by cutting off speech mid-response and ensuring both audio and animation stop cleanly.

• Test jitter on a slower network and confirm the face does not race ahead of the audio.

• Watch CPU load on the client if lip-sync is being rendered locally; animation quality is irrelevant if frames drop.

Also remember that voice quality and latency are often in tension. Higher-quality TTS models can produce better naturalness but may take longer to start. If your product is conversational and interruption-heavy, a slightly less rich voice that starts quickly may outperform a more expressive one that lags behind the user.

Conclusion

A good realtime lip-sync pipeline is mostly about disciplined streaming. Keep text chunks small enough to start early, stream audio instead of buffering it into a blob, and drive the avatar from the same timing model as the sound. When you do that, the face feels connected to the conversation instead of merely attached to it.

If you want a faster path from prototype to working system, start with the integration surface that matches your stack, then verify the timing in your own app. The public docs are the right place to check exact request shapes, session semantics, and current integration details: docs.protoface.com.