When to Choose SFU Over Peer-to-Peer in LiveKit Agents for Voice + Video Avatars
Choose SFU over peer-to-peer for LiveKit voice/video avatars when you need sync, reconnects, multi-party support, and room observability.
Introduction
When you add a talking avatar to a live voice agent, the first architecture decision is usually not “which model” or “which renderer.” It is whether the audio/video path should be peer-to-peer or routed through an SFU.
That choice matters because an avatar is not just a video stream. It is a synchronized output of TTS audio, lip motion, and sometimes head or facial animation, all updated in real time while the conversation is still unfolding. If you get the transport wrong, you will see drift, awkward latency spikes, broken reconnections, or a system that becomes hard to scale past a single user.
By the end of this post, you should be able to decide when peer-to-peer is good enough, when an SFU is the safer default, and how to wire a Protoface avatar into a LiveKit voice agent without turning your media path into a debugging exercise.
Peer-to-peer vs SFU: what actually changes
In a peer-to-peer setup, the client and agent exchange media directly. The upside is simplicity: fewer moving parts, no central media broker, and potentially lower overhead for a one-to-one session. If you are building a small demo where one browser talks to one agent, peer-to-peer can be perfectly reasonable.
The downside is that every participant becomes a media endpoint. For a voice agent with a video face, that means the browser has to handle:
the microphone uplink,
the agent audio downlink,
the avatar video track,
reconnection logic,
network variability, and
any future expansion to monitoring, recordings, or additional participants.
An SFU changes the topology. Clients publish media to the SFU; the SFU forwards selected streams to other participants. It does not decode and re-encode like a traditional MCU, so it keeps latency lower and preserves more of the original media characteristics. For live voice + avatar work, that usually means a cleaner separation of concerns: your agent generates audio/video, and the SFU handles distribution, subscription, and reconnect behavior.
When peer-to-peer is enough
Choose peer-to-peer when the product constraints are narrow and unlikely to change:
Strictly one-to-one sessions. No supervisors, no spectators, no shared room participants.
Prototype or internal tool. You want the shortest path to “it works” and can tolerate edge cases.
No need for room-level features. You do not care about server-side recording, multi-party orchestration, or participant metadata.
Very controlled networks. For example, a local demo, kiosk, or an environment where NAT traversal and connectivity are predictable.
Even then, be honest about your future requirements. The most common mistake is picking peer-to-peer because “there is only one user today,” then discovering a few weeks later that support teams want to join calls, product wants recordings, and the avatar needs to appear in a shared workspace. At that point, the migration cost is not just transport. It is also session semantics and lifecycle management.
When SFU is the better default
For live avatars embedded in real products, SFU is usually the more robust choice. The reason is not abstract “scale.” It is the set of practical problems an SFU solves for realtime media:
Stable fan-out. One agent stream can be delivered to many viewers without turning the agent into a broadcaster to each peer individually.
Better session control. You can manage joins, leaves, and reconnects centrally instead of relying on each client to repair its own transport state.
Cleaner browser behavior. Browsers are good at one media connection; they are better when the room infrastructure absorbs complexity.
Room observability. It is much easier to reason about who is connected, what streams exist, and where failures occurred.
Room for growth. Support agents, supervisors, recording bots, analytics taps, or multi-client experiences become possible without redesigning the media layer.
For voice + video avatars specifically, SFU also helps with synchronization discipline. Your agent typically produces audio first, and the avatar video is derived from that speech cadence. The transport should preserve timing as cleanly as possible, because lip sync errors are more noticeable than ordinary video jitter. An SFU does not magically fix bad generation logic, but it avoids adding unnecessary point-to-point complexity on top of it.
Media and synchronization gotchas developers usually hit
There are a few failure modes worth calling out because they are easy to misdiagnose.
1. Audio latency and avatar drift are not the same problem. If the model or TTS pipeline is slow, the whole conversation feels laggy. If the avatar video lags behind audio, the conversation feels uncanny. The fix differs. In the former case, optimize generation latency. In the latter case, inspect buffering, jitter, and track scheduling.
2. Do not treat video as an afterthought. A talking face is not a passive overlay. It is part of the conversational contract. If the video track drops and reconnects independently from audio, users notice immediately.
3. The browser is not the whole system. Your server may be generating media in order, but transport can reorder delivery under congestion. Any architecture that assumes “sent order == rendered order” will eventually disappoint you.
4. Room lifecycle matters. If the agent joins late, leaves early, or restarts mid-session, you need a deterministic way to restore the avatar state. That is much easier when the session is represented as a room with server-side coordination rather than as an ad hoc peer connection.
5. Quality tier affects the product experience. If your avatar is the interface, the transport path is part of quality. Lower bandwidth or lower fidelity may be fine for some use cases, but the user experience should be intentionally chosen, not accidental.
Protoface in a LiveKit agent: why the SFU path fits
This is where Protoface fits naturally. The LiveKit Agents plugin drops a synchronized talking face into a voice agent, so the agent can speak and appear as a realtime avatar without you building a custom media pipeline from scratch. The integration is specifically aligned with the room/SFU model: the agent lives in the LiveKit session, and the avatar rides alongside the voice interaction as a media participant rather than as a separate browser hack.
In practice, that means you keep your agent logic where it belongs and let the media stack do the hard work. A minimal shape looks like this:
The point is not the exact constructor signature; check the docs for the fields you need. The important bit is architectural: the avatar is attached to the agent in the LiveKit room, so the agent’s speech and visual output stay synchronized under the same media transport.
If you are using the Python SDK or REST API to create sessions ahead of time, keep the lifecycle explicit: create the avatar/session, start the agent, join the room, and tear everything down when the conversation ends. For API-driven workflows, the REST surface is also useful when you want to orchestrate sessions outside the agent process.
Again, the exact payload shape is in the docs; the example is just to show the pattern. The value is in treating avatar/session setup as infrastructure, not UI glue.
A practical decision rule
If you are building a one-off demo, a peer-to-peer path can be acceptable. If you are building anything users rely on, default to SFU unless you have a strong reason not to.
Use this heuristic:
Peer-to-peer for one-to-one, short-lived, tightly controlled demos.
SFU for production agents, multi-party rooms, browser clients you do not fully control, or any avatar experience where reconnects and observability matter.
For voice + video avatars, the extra coordination an SFU provides is usually worth more than the simplicity you give up. The avatar is part of the conversation, so its transport should be boring, reliable, and easy to reason about.
Conclusion
The core trade-off is simple: peer-to-peer minimizes infrastructure, while an SFU minimizes risk once the avatar becomes a real product surface. If your goal is a stable live avatar inside a voice agent, the SFU path is usually the one that scales better operationally and produces fewer media surprises.
If you are integrating this with LiveKit Agents, start from the plugin flow, keep session lifecycle explicit, and verify lip sync under real network conditions before you ship. The docs at docs.protoface.com are the right place for exact API fields, setup steps, and current integration details.