A Practical Guide to LiveKit Agents Media Topology for Conversational Avatar Apps
LiveKit Agents media topology guide for conversational avatars: sync audio, video, and session state with low-latency server-side rendering.
Introduction
When you add a talking avatar to a voice agent, you are really solving a media topology problem: audio, video, and agent state all need to move through the right paths with the right timing. The common failure modes are familiar to anyone who has built realtime products: audio arrives before the mouth motion that should match it, frames are dropped under load, the agent blocks on rendering, or the video pipeline adds enough latency that the interaction feels disconnected.
This guide focuses on the practical topology choices behind conversational avatar apps built on LiveKit Agents. By the end, you should be able to reason about where the avatar lives in the media graph, what data should cross which boundary, and how to avoid the most common architecture mistakes when connecting a voice agent to a synchronized talking face.
Understand the media graph before you wire anything up
A conversational avatar app is not “a chatbot with video.” It is usually a small realtime system with at least four moving parts:
User audio: captured by the client, sent to the agent for ASR and turn detection.
Agent output audio: synthesized or generated speech that must be streamed back with low delay.
Avatar video: the visual layer that should stay aligned to the spoken audio stream.
Session state: turn state, instructions, voice selection, and whatever metadata your app needs to persist.
The key topology question is where the avatar rendering happens. In practice, you want the avatar video generation to sit as close as possible to the agent’s speech output, not at the browser edge. If you render the face in the same service that already knows the TTS timing, phonemes, or speech segments, you reduce coordination overhead and avoid a second, lossy synchronization channel.
For LiveKit-based systems, that usually means the avatar becomes another media participant in the session, rather than a separate websocket stream you try to align in the client. The agent emits speech, the avatar service uses that speech timing to generate lip-synced frames, and the resulting video track is published back into the room. That keeps the browser simple: it subscribes to one coherent realtime session instead of trying to infer synchronization across multiple independent connections.
Keep the synchronization boundary close to the speech pipeline
The most important implementation detail is timing ownership. Audio and video do not need identical transport paths, but they do need a shared notion of turn timing. If the agent is generating text incrementally, or the TTS service streams chunks, the avatar renderer should consume the same stream boundary information that the audio player uses. Otherwise you end up with one of two bad outcomes:
The avatar starts speaking too early, before audible audio is ready.
The avatar lags behind audio because it waits for an entire sentence or response to complete.
For responsive agents, you generally want incremental rendering. That means the agent can begin talking as soon as the first speech segment is available, and the avatar can start mouth motion on that same segment. The practical trade-off is that you need sane buffering and backpressure handling. Too little buffering and you will get jitter; too much and the interaction feels delayed.
A useful mental model is:
ASR determines when the user finished speaking.
LLM / orchestration determines the response content and turn transitions.
TTS turns text into audio frames.
Avatar renderer uses the same turn and chunk timing to generate video frames.
Do not make the browser responsible for reconciling these layers. Browsers are excellent at playback, but poor as synchronization authorities when the media sources are independent.
Choose a topology that preserves low latency and failure isolation
There are three topology patterns I see most often.
1. Browser-only composition
This is the simplest to prototype: the client receives audio and video separately and tries to keep them aligned locally. It works for demos, but it becomes fragile quickly. Packet loss, tab throttling, autoplay restrictions, and client CPU variance all create drift. This pattern is usually the wrong choice for production conversational avatars.
2. Agent-owned media publication
This is the pattern you want for most realtime voice agents. The agent runs the conversation loop, the avatar is attached to the agent process or session, and both audio and video are published into the realtime room. The browser stays a thin subscriber. This gives you a single control plane for turn timing and much better failure isolation: if the client has a hiccup, the server-side session can continue cleanly.
3. Split control plane, server-side avatar renderer
In more complex systems, your business logic, agent orchestration, and avatar rendering may live in separate services. That can be useful when you need to scale avatar generation independently from conversation logic. The cost is more coordination: session IDs, lifecycle events, and retries must be explicit. If you go this route, make sure the avatar service subscribes to the same authoritative session state as the agent, not a derived or lagging copy.
For all three patterns, watch for two failure modes:
Clock drift: even small timing mismatches become visible in mouth motion.
Duplication of turn logic: if both the agent and the browser think they own when speech starts or ends, synchronization will eventually break.
Operational details that matter in production
Once the architecture is sound, the remaining issues are mostly operational.
Session lifecycle matters more than people expect. You need a clear start state, a clean teardown path, and a place to recover from reconnects. If the agent restarts while the browser stays connected, the system should either reattach cleanly or end the session explicitly. Silent partial failure is what hurts user trust.
Rate limits and quality tiers should be part of your design, not an afterthought. Video generation is not free, and you will usually want to match quality tier to use case: lower latency and lower cost for short-lived support interactions, higher fidelity for branded experiences or demo surfaces. That decision affects buffering, frame generation cost, and perceived polish.
Security boundaries are also different for each surface. A browser embed should never need an API key. A server-to-server session creator absolutely should. Keep long-lived credentials out of the client, and use short-lived session-specific controls where possible. If you expose a browser-facing avatar surface, it should be constrained by origin allowlists, per-embed instructions, and reasonable rate limits.
Observability is not optional. At minimum, log session IDs, start/stop times, turn transitions, avatar attach/detach events, and media errors. When a user says “the avatar was out of sync,” you need to know whether the issue happened during ASR, TTS startup, video generation, or room publication.
How Protoface fits into this topology
Protoface is useful here because it gives you a concrete avatar layer without forcing you to redesign the rest of the voice stack. For LiveKit Agents, the LiveKit plugin and the related Pipecat integration make it straightforward to drop a synchronized talking face into an existing agent flow, so the avatar sits on the same timing boundary as the agent’s speech rather than being bolted on later.
If you are wiring this up programmatically, the Python SDK is the cleanest way to manage avatars and realtime sessions from your backend. The exact request and response fields are documented, but the flow is straightforward: create or select an avatar, start a session, attach it to the agent conversation, and tear it down when the room ends.
If you want to inspect the raw API flow, the REST surface is equally direct. The important part is to keep it server-side:
For examples and setup details, the docs at docs.protoface.com are the right place to start.
Practical integration pattern for LiveKit agents
If you are already using LiveKit Agents, the cleanest implementation is usually:
Let the agent own conversation state and speech generation.
Attach the avatar at the point where the agent produces streamed speech.
Publish the resulting video back into the LiveKit room as part of the same session.
Keep the browser as a subscriber, not a coordinator.
That pattern minimizes synchronization work and makes reconnect behavior much easier to reason about. It also keeps your app flexible: you can swap TTS providers, change the language model, or adjust voice settings without changing the basic media topology.
If you prefer to follow a known-good reference implementation, the plugin and quickstart repos linked from the Protoface GitHub organization are a better starting point than building the plumbing from scratch. The important thing is not the exact provider combination; it is preserving the invariant that speech timing and avatar rendering share the same authoritative session state.
Conclusion
For conversational avatars, the core design choice is not visual fidelity; it is where synchronization lives. Keep avatar rendering close to the speech pipeline, publish media from the server-side agent session, and avoid making the browser the source of truth for turn timing. That gives you lower latency, fewer sync bugs, and a topology that is much easier to operate.
If you are implementing this with LiveKit Agents, start with a server-owned session model and a plugin or SDK path that keeps the avatar attached to the agent’s speech stream. Then validate the edge cases: reconnects, partial responses, and teardown. For concrete setup steps, sample code, and integration notes, see the docs and the relevant quickstarts in the GitHub organization.