How to Stream Audio and Facial Animation Together with LiveKit
How to sync LiveKit audio, visemes, and turn timing for real-time voice agents with Protoface and WebRTC.
Introduction
If you are building a voice agent, the hard part is usually not generating speech. It is keeping the audio stream, the avatar’s lip motion, and the application state in sync well enough that the result feels live rather than stitched together. In practice, that means dealing with WebRTC transport, real-time audio timing, latency budgets, buffering, and the agent pipeline itself.
This post walks through the mechanics of streaming audio and facial animation together with LiveKit, and shows how to think about synchronization from the agent’s point of view. By the end, you should know where the timing comes from, what can go wrong, and how to integrate a talking avatar into a LiveKit-based voice agent without breaking the realtime loop.
What “synchronized” actually means
For a talking avatar, synchronization is not just “the face changes when audio plays.” You need three things to line up:
Audio playout time: when the user actually hears synthesized speech.
Viseme or mouth-shape timing: when the avatar opens, closes, and shapes the mouth.
Turn boundaries: when the agent starts, pauses, interrupts, or resumes speaking.
In a LiveKit voice agent, audio typically moves as a media stream over WebRTC. The avatar side needs to consume the same speech events or the same synthesized audio in a way that preserves timing. If the mouth animation is derived from text rather than audio, latency can be lower, but accuracy depends on the TTS model and the animation pipeline. If the animation is derived from audio features, the lip-sync is usually more faithful, but you must manage buffering and frame timing carefully.
From an engineering perspective, the goal is simple: the video face should appear to be “speaking the same utterance” as the audio, with minimal drift over a conversation, even when the agent is interrupted or the network jitters.
How the realtime pipeline fits together
A useful mental model is:
The user speaks into a LiveKit room.
Your agent consumes the audio, runs ASR or a speech pipeline, and decides what to say next.
The agent synthesizes speech or receives synthesized audio from a TTS component.
The avatar renderer produces a talking face that matches that speech stream.
Both streams are delivered back to the client with low enough latency to stay conversational.
The important part is that audio and animation do not live in separate universes. They need a shared notion of turn state and time. In a well-designed system, the avatar renderer is not guessing when speech starts; it is attached to the same output event that begins audio playout or receives the same speech token / audio segment boundary the user hears.
There are a few practical trade-offs here:
Lower latency vs. better sync: starting animation early can make the avatar feel more responsive, but if the audio pipeline stalls, the mouth can get ahead of the sound.
Text-driven vs. audio-driven lip sync: text-driven animation can begin as soon as the model emits a response, but audio-driven timing usually looks more natural.
Single stream vs. multiple streams: sending audio and video independently over WebRTC is normal, but you still need application-level coordination so the face does not lag behind the voice.
Working with LiveKit agents
If you are already using LiveKit Agents, the cleanest integration point is the agent pipeline itself. The avatar should be treated as part of the speech output path, not as a separate UI component bolted on afterward. That is the easiest way to keep turn-taking correct.
At a high level, your agent code emits a speaking event, hands off the synthesized speech, and the avatar layer turns that into a synchronized video face. In Python, that often looks like a plugin-style integration rather than a full custom media stack.
That example is intentionally schematic. The real code depends on how you structure your LiveKit agent, but the pattern matters more than the exact method names: the avatar is introduced at the point where the agent begins producing speech, so its motion can remain aligned with the utterance lifecycle.
A few gotchas are worth calling out:
Do not start the avatar on raw text alone unless your stack is built around text-timed animation. Otherwise, the face may move before audio is ready.
Handle interruption cleanly. If the user barges in, the agent should stop both audio playout and mouth animation immediately.
Keep an eye on buffering. If your TTS adds a noticeable startup delay, the avatar should either wait or show a neutral idle state instead of “talking into silence.”
Streaming audio and video over WebRTC
LiveKit gives you the transport layer, but the transport layer does not solve semantics. WebRTC will happily carry an audio track and a video track, yet those tracks can still drift in perceived time if you ignore the source timing.
What tends to work well in practice is:
Keep the agent clock authoritative. Use the agent’s speaking lifecycle as the source of truth for when the turn starts and ends.
Start both outputs from the same event. Audio playout and avatar animation should be triggered by the same speech boundary.
Let the media stack handle network transport. Do not invent your own synchronization protocol on top unless you need very custom behavior.
Also remember that video faces have a frame rate, while audio is continuous. The avatar renderer must choose a frame to display for each point in time based on the current phoneme or viseme state. If you see obvious desync, the bug is often not “WebRTC is slow”; it is usually that your application emitted state changes too early, too late, or out of order.
How Protoface fits in
Protoface is designed to sit directly in that speech-output path. For LiveKit-based agents, the relevant surface is the LiveKit plugin, pipecat-protoface for Pipecat users, and the underlying docs at docs.protoface.com. The useful detail is not that it “adds video”; it is that it gives your agent a synchronized talking face without forcing you to build and operate a separate avatar pipeline.
For example, in a LiveKit agent you can install the plugin and let it take care of the avatar side while your agent continues to manage the conversation logic and audio:
If you need to create or manage avatars and realtime sessions outside the agent runtime, the REST API is the right fit. Authentication uses API keys in the usual Bearer-token pattern, and the dashboard at app.protoface.com is where you manage sessions, avatars, keys, and usage. Keep API keys server-side; do not expose them to the browser.
The exact request shape depends on the endpoint and fields you use, so treat that as illustrative. The main point is that session creation and avatar management can be automated from your backend when you need dynamic provisioning.
Operational concerns: latency, failure modes, and quality
Once the demo works, the real work is making it stable under real traffic. The common failure modes are predictable:
Startup latency: the first utterance takes too long, so the avatar looks like it is thinking before it speaks.
Turn drift: the mouth continues moving after the audio has been interrupted.
Quality mismatch: the avatar is sharp, but the audio is low quality, or vice versa.
Session leaks: realtime sessions are not torn down cleanly after disconnects or timeouts.
In practice, you want to measure end-to-end conversational latency, not just TTS latency or video encode latency in isolation. Track time from user stop-speaking to agent response start, and time from response start to visible mouth motion. Those are the numbers that determine whether the interaction feels live.
Protoface bills by quality tier, which makes it worth being explicit about where you need fidelity and where you can accept a cheaper path. A support bot in a browser embed may tolerate a different visual quality than a premium sales assistant or a game NPC. Architect around that early so you do not paint yourself into a corner.
Conclusion
Streaming audio and facial animation together is mostly a synchronization problem: your agent, your audio pipeline, and your avatar renderer need a shared notion of turn timing. LiveKit gives you the realtime transport, but you still need to attach the avatar at the right layer and avoid treating it as an afterthought.
If you are building this for a production agent, start with the plugin integration, verify the turn-taking behavior under interruption, and measure latency end to end. Then move to session management and provisioning once the realtime path is solid. For implementation details, examples, and the current API surface, check the docs and the relevant quickstarts linked from the Protoface GitHub resources.