How to Use LiveKit Agents for SFU vs Peer-to-Peer Media in Realtime Avatar Apps
LiveKit Agents guide to SFU vs peer-to-peer media for realtime avatar apps, sync, scaling, and session control.
Introduction
If you are building a realtime avatar app, the first architectural choice is usually not “which avatar model?” It is “how does the media move?” In practice, you are choosing between SFU-based transport and peer-to-peer transport, and that choice affects latency, scaling, debugging, and how much of the media graph you control.
For voice agents with synchronized talking faces, the avatar is not a separate UI concern. It is part of the media pipeline: the agent produces speech, the avatar needs the audio and timing signals, and the client needs a video track that stays in lockstep with the conversation. This post breaks down how SFU and peer-to-peer fit into that picture, when to use each, and how LiveKit Agents can be wired up so the avatar behaves like a first-class media participant rather than a bolted-on widget.
By the end, you should be able to reason about the transport layer for your avatar app, understand the trade-offs between direct and mediated media paths, and see where Protoface fits when you need a synchronized avatar inside a LiveKit voice agent.
What changes when an avatar is part of the media graph
In a conventional voice agent, the media graph is simple: microphone input goes to the agent, synthesized audio comes back to the client, and maybe the browser renders a waveform or captions. Once you add a realtime avatar, you introduce a second output that has to stay aligned with the audio stream:
The avatar needs to respond to the agent’s speech cadence, not just text.
The client needs a stable video track that can tolerate jitter without visible desync.
Session state matters: reconnects, interruptions, barge-in, and turn transitions all affect what the face should do.
This is why “just stream a video file” does not work. A realtime avatar is effectively a live media endpoint. Whether you deliver that endpoint over an SFU or directly peer-to-peer changes who terminates the transport, who can observe/modify the tracks, and how well the system scales when you have many sessions.
SFU vs peer-to-peer: the practical difference
Peer-to-peer media means the participants send media directly to each other. That works well when there are only two endpoints, NAT traversal is manageable, and you want the shortest possible path. In a simple avatar demo, that can look appealing: fewer moving parts, no media server, and a direct audio/video connection.
But peer-to-peer has some hard limits:
Every participant needs a separate connection to every other participant.
Topology gets messy quickly if you add a human, an agent, an avatar, transcription, recording, or monitoring.
Server-side media processing is awkward because there is no central place to subscribe, fan out, or inspect tracks.
An SFU, by contrast, is a selective forwarding unit. It does not decode and re-encode everything like a MCU would. Instead, it receives media from publishers and forwards the relevant tracks to subscribers. For realtime avatars, that matters because the avatar can join the same room as the voice agent and humans, subscribe to the agent’s audio, and publish a video track back into the room. The SFU becomes the coordination point for the session.
When peer-to-peer is enough
Use peer-to-peer only when the media topology is genuinely small and stable. A common example is a one-off web demo where a single browser talks to a single backend service and you do not need other participants, recordings, or server-side routing. If the avatar is rendered entirely client-side and the only transport requirement is “send audio to one endpoint and get video back,” peer-to-peer can reduce infrastructure.
The downside is that the moment you want any of the following, peer-to-peer starts to look like the wrong abstraction:
Multiple participants in the same session.
Server-side agent orchestration.
Recording or observability.
Hot-swapping voices, prompts, or avatar state during a call.
Web experiences that should not expose backend credentials.
For realtime avatar apps, those needs show up quickly. You often start with one-to-one interaction, but the production version ends up needing room semantics, session management, and a controlled server-side media plane.
Why SFU is usually the better default for voice agents with avatars
SFU-based architectures fit the shape of voice agents better because the agent is already a server-side participant. The agent needs access to the user’s audio, and the avatar needs access to the agent’s synthesized audio or turn-taking signals. An SFU lets you model each of those as tracks in a room rather than inventing custom signaling between browser, agent, and avatar renderer.
From a systems standpoint, SFU gives you three things that matter for avatar applications:
Scalable fan-out. One published avatar video track can be forwarded to many subscribers without duplicating work in the avatar renderer.
Centralized session control. You can add/remove participants, handle reconnects, and coordinate turn-taking in one place.
Server-side hooks. The agent can subscribe to audio, produce speech, and keep the avatar synchronized with the call state.
There is also a quality angle. For conversational avatars, the visible artifact users notice first is desync, not bitrate. SFU does not magically solve desync, but it gives you the right place to manage timing and state under load.
How LiveKit Agents fit into the picture
LiveKit Agents are a natural place to attach an avatar because they already sit inside the realtime media flow. The agent receives audio, does its inference, emits speech, and can publish tracks into the room. If the avatar is another published track, then the agent and avatar stay coupled at the session layer instead of being stitched together by a separate web socket or ad hoc streaming endpoint.
Conceptually, the flow looks like this:
The important detail is that the avatar is not “just a UI overlay.” It is media produced in response to the agent’s turn state. That means the avatar plugin should live close to the agent runtime where it can share timing, interruption handling, and session lifecycle.
If you are using LiveKit Agents in Python, the integration pattern is usually straightforward: install the plugin, instantiate it in the agent process, and let it manage the avatar side of the session. Exact constructors and fields vary by integration, so treat the snippet below as illustrative and check the current docs for the real options.
For the LiveKit-specific plugin and examples, the quickest place to start is the plugin repository: GitHub. If you are using Pipecat instead of raw LiveKit Agents, the integration is documented separately in the Pipecat service guide.
Where Protoface solves the avatar side cleanly
This is the part that tends to get messy if you build it yourself. Documentation aside, the useful surfaces are the ones that map to how developers actually ship: a LiveKit Agents plugin for dropping in a synced avatar, a REST API for session creation and management, and a Python SDK when you want to drive avatar/session lifecycle programmatically.
A minimal REST flow is usually:
The exact endpoints and payload fields depend on the API surface you are using, so rely on the docs for the authoritative schema. The main point is that session and avatar state live server-side, which is what you want when your media pipeline is mediated by an SFU and your agent is the thing coordinating the conversation.
If you prefer to create sessions in Python instead of curling the REST API directly, the Python SDK gives you the same programmatic control from your backend. That is especially useful when your app already has a session orchestration layer and you want avatar provisioning to be just another step in that workflow.
Trade-offs and gotchas that matter in production
1. Keep the avatar close to the agent. The more hops you insert between speech generation and avatar rendering, the more chances you have for perceptible lag. If the agent and avatar are part of the same server-side runtime, you can coordinate turn boundaries more accurately.
2. Do not expose secrets in the browser. If you are building a browser-based experience, never ship long-lived API keys client-side. For full web embeds, a customer-managed iframe is the safer pattern because it keeps credentials off the page and lets you enforce origin and rate limits.
3. Treat reconnection as a first-class behavior. In a realtime avatar session, reconnects should preserve session identity and preferably the user-visible state. If your transport choice makes session continuity hard, users will see the avatar “reset” more often than you expect.
4. Measure quality in context. An avatar can have perfectly acceptable video quality and still feel wrong if the audio/video sync is off. When choosing a quality tier, evaluate the full conversational loop, not just frame resolution.
Choosing the architecture
A useful rule of thumb:
Choose peer-to-peer when you have a tiny topology, no server-side media processing, and no need for shared room semantics.
Choose SFU when the avatar is part of a real voice agent, when the server needs to participate in the media flow, or when you expect to add monitoring, recording, or multiple participants later.
For most production avatar agents, SFU is the safer default. It aligns with how voice agents are built, it keeps media control centralized, and it makes the avatar a normal participant in the session instead of a special case.
Conclusion
The main decision is not just transport; it is where you want control to live. Peer-to-peer can work for minimal demos, but SFU is usually the better fit for realtime avatar apps because it matches the lifecycle of a voice agent and gives you room to scale the interaction model.
If you are building this on LiveKit, start by treating the avatar as part of the agent’s media graph, not a separate frontend feature. Then wire in the avatar/session layer from the server side and keep credentials out of the browser.
For implementation details, refer to the public docs at docs.protoface.com, and use the GitHub plugin or SDK repositories when you want concrete integration examples. The quickest way to validate the architecture is still the old-fashioned way: spin up a small session, inspect the track flow, and make sure the avatar stays in sync under interruption and reconnect.