Introduction

If you are building a voice agent, the first version is usually just audio: the model speaks, the user speaks back, and the system stays stateful over a live session. That works, but it leaves a lot on the table. In practice, a talking face changes how users interpret turn-taking, latency, and presence. It also gives you a place to render nonverbal signals like nods, gaze, and mouth motion that make an AI agent feel synchronized instead of disjointed.

This post is about the specific realtime problem: how to add a lip-synced avatar to a live conversational app without breaking the low-latency properties of your existing voice pipeline. By the end, you should understand the moving pieces behind simulcast-style realtime avatar delivery in LiveKit, what you need to coordinate between audio and video, and where Protoface fits into that architecture.

What simulcast means in this context

In WebRTC, simulcast usually means sending multiple encodings of the same video stream so downstream consumers can choose a quality level based on bandwidth and viewport constraints. For an AI avatar, the interesting part is not just bandwidth optimization. The harder part is that the avatar’s video must stay synchronized with the agent’s speech generation, the transport path, and the client’s playback timing.

At a high level, the avatar system has three time-sensitive inputs:

• Text or token stream from the language model or agent orchestrator.

• Audio stream from TTS, or a timing signal derived from the audio pipeline.

• Session control messages that start, stop, and update the avatar behavior.

The video output is not a generic prerecorded stream. It is a generated stream that needs to remain aligned with speech onset, phoneme timing, and the agent’s turn boundaries. If that alignment slips, users notice immediately: lips drift from audio, head motion becomes uncanny, or the avatar appears to “think” after it has already started talking.

Why LiveKit is a good fit for this

LiveKit already gives you the primitives you want for realtime voice: rooms, participants, audio tracks, and a production-grade WebRTC transport. If your agent is already in a LiveKit room, the avatar should behave like another realtime participant rather than a sidecar service. That means the avatar video track should join the same media graph, use the same session lifecycle, and follow the same latency budget as the rest of the call.

That integration point matters because the avatar is not just decoration. In a LiveKit agent, the “source of truth” is usually the agent runtime: it decides when the assistant speaks, when to interrupt, and when to stop. The avatar layer needs to subscribe to that state rather than infer it indirectly from network traffic or audio levels. Put differently: keep media transport and agent orchestration separate, but tightly coordinated.

Implementation shape: keep the agent loop authoritative

A practical architecture looks like this:

1. The user joins a LiveKit room and speaks to the agent.

2. The agent runtime receives audio, transcribes or reasons over it, and decides on a response.

3. TTS generates the spoken response.

4. The avatar subsystem receives the same response context and emits a synchronized talking face.

5. LiveKit distributes the resulting audio and video tracks to the client.

The key thing to avoid is double inference about speech state. Do not let the video side independently decide when the avatar should open or close its mouth. Instead, feed it the same turn metadata and timing cues that drive audio playback. If the avatar service supports streaming updates, send them as soon as the model response starts, not after the full sentence is complete.

Two implementation details usually matter more than people expect:

• Latency budget: if video starts too late relative to audio, the face looks detached. Keep the avatar session warm if possible, and avoid adding extra hops between agent output and video generation.

• Turn finalization: the end of a turn is as important as the beginning. Cleanly signal stop/pause so the avatar does not keep animating through silence or user interruption.

Using the LiveKit plugin in a voice agent

For LiveKit-based agents, the lowest-friction integration is the Protoface plugin for Pipecat / LiveKit agent pipelines. The plugin drops an avatar into the existing agent flow so you do not need to build your own media bridge from scratch.

The exact API surface is documented, but the shape is typically straightforward: configure your avatar/session credentials, attach the avatar service to the agent pipeline, and let it mirror the assistant’s spoken output. A minimal example might look like this:

from livekit.plugins.protoface import ProtofaceAvatar

agent.add_output_sink(avatar)

from livekit.plugins.protoface import ProtofaceAvatar

agent.add_output_sink(avatar)

from livekit.plugins.protoface import ProtofaceAvatar

agent.add_output_sink(avatar)

The important point is not the exact method names; those can vary by framework version. The pattern is what matters: the avatar is a downstream consumer of the agent’s generated speech, and it publishes a synchronized video track back into the LiveKit room. If you want a reference implementation, start from the examples in the repository linked above and adapt it to your agent runtime.

When to use the REST API or Python SDK directly

If you need to create avatars, manage sessions, or wire avatar state into your own orchestration layer, the REST API is the right tool. It is also the cleanest way to integrate non-LiveKit systems or to build admin tooling. The authentication model is the usual bearer-token setup; keep API keys server-side only.

A simple session-creation request looks like this in principle:

curl https://api.protoface.com/sessions \
}'

curl https://api.protoface.com/sessions \
}'

curl https://api.protoface.com/sessions \
}'

For Python, the SDK is useful when you want to automate avatar/session lifecycle from your backend or from an agent control service. Again, keep this illustrative and check the docs for exact object names and fields:

from protoface_sdk import Client

)

from protoface_sdk import Client

)

from protoface_sdk import Client

)

The SDK and REST API are most useful when you need deterministic control over creation, teardown, and per-session configuration. That includes customer support flows, game NPC sessions, and systems that need to attach business metadata to a session. For manual testing and ops work, the developer dashboard is also handy, but it should not be in your production path.

Quality, bandwidth, and failure modes

Once the basic integration works, the main trade-off is quality versus cost. Higher-fidelity video helps the avatar hold up at larger sizes and in more visible placements, but it also increases compute and delivery cost. In a product setting, that usually means mapping quality tier to user-facing context:

• Lower tier for small embedded assistants or internal tooling.

• Higher tier for visible customer-facing agents where the face is central to the experience.

From a transport perspective, watch for the same failure modes you would expect in any realtime media system:

• Choppy video on poor networks: if the client cannot keep up, simulcast helps it downshift gracefully instead of dropping the stream.

• Audio/video skew: usually a session coordination problem, not a rendering problem.

• Long first-frame delay: often caused by cold starts, avatar initialization, or waiting too long to create the session.

• Interruptions: if the user barges in, the agent should stop the current utterance and the avatar should reflect that immediately.

In practice, the cleanest debugging approach is to instrument the agent lifecycle separately from the media lifecycle. Log when the assistant turn starts, when TTS starts, when the avatar session starts, and when the final media track is published. If those timestamps drift, you will find the source of the lag quickly.

How Protoface fits here

Protoface is the avatar layer that plugs into this workflow without forcing you to rebuild your voice stack. For LiveKit deployments, the plugin surface is usually the fastest path: the agent stays in control, and the avatar subscribes to that control flow instead of becoming a separate application. If you need lower-level control, the REST API and Python SDK cover session management directly, and the docs at docs.protoface.com are the right place for exact request shapes and integration details.

For teams already using LiveKit Agents, this is a pretty natural division of labor: LiveKit handles realtime transport and conferencing semantics, your agent handles conversation logic, and the avatar service handles synchronized face generation. That separation keeps the integration understandable and makes it easier to swap models, voices, or presentation tiers later.

Conclusion

The core idea is simple: treat the avatar as a realtime media participant, not a decorative afterthought. If you keep the agent authoritative over turn state, align video generation with speech timing, and let WebRTC do what it is good at, you can add a convincing talking face without sacrificing conversational latency.

If you are starting from scratch, begin with a LiveKit agent quickstart and wire in the avatar plugin. If you already have backend orchestration, use the REST API or Python SDK to create sessions and test the lifecycle independently. In either case, keep the integration small at first, measure end-to-end turn latency, and then decide what quality tier makes sense for your product.

For implementation details and current examples, start with the documentation and the relevant GitHub examples. That will save you from guessing at field names and let you focus on the part that actually matters: making the agent feel present in realtime.