Using Agora for Realtime Avatar Streaming: When to Pick SFU Over Peer-to-Peer
Realtime avatar streaming with Agora: when to use SFU vs peer-to-peer, plus latency, sync, scaling, and session control trade-offs.
Introduction
If you are building a realtime avatar product, the first architectural choice is usually not “which model?” or “which renderer?” It is “how do I move media between participants?” That decision determines latency, bandwidth cost, failure modes, and how much control you have over the user experience.
In practice, you are usually choosing between peer-to-peer media paths and an SFU-based topology. Peer-to-peer can be the simplest possible path for one-to-one sessions. SFUs add a media server in the middle, which increases control and scalability at the cost of more infrastructure. For avatar streaming, that trade-off matters because you are not just sending audio: you are synchronizing audio, video, and often agent state under tight latency constraints.
By the end of this post, you should be able to decide when P2P is sufficient, when SFU is the better default, and what implementation details matter when you are streaming a talking face rather than a plain video call.
Peer-to-peer vs SFU: what actually changes
Peer-to-peer media means each client establishes a direct connection to every other client. For a single human talking to a single avatar, that sounds appealing: fewer moving parts, lower server cost, and potentially lower end-to-end latency.
But for realtime avatars, P2P has a few sharp edges:
Connectivity depends on both endpoints. NAT traversal, firewalls, and mobile networks can make direct negotiation brittle.
Scaling is quadratic. The moment you need one speaker and multiple viewers, every viewer adds another direct connection.
Server-side control is limited. If you want moderation, recording, server-side fan-out, or late-joining participants, you end up rebuilding media plumbing around the P2P path.
State synchronization is harder. With an avatar, the important artifact is not just the raw media stream; it is the alignment between audio, lip motion, and session state. Direct client links make centralized orchestration harder.
An SFU, or Selective Forwarding Unit, sits in the middle and forwards media streams between participants. It does not decode and re-encode like a full MCU. That distinction is important: an SFU is usually much cheaper and lower latency than transcoding everything centrally, while still giving you a control point.
For avatar streaming, the SFU model tends to fit better because the avatar is often one media producer serving many consumers:
a user talking to an agent in a support flow
an AI sales rep embedded on a site with multiple concurrent visitors
a game NPC or assistant that may be watched by more than one client
a conversational video agent that needs to survive reconnects cleanly
Why avatars are not just “video calls with a face”
A realtime avatar pipeline usually has three timing-sensitive components:
Speech generation — audio produced from TTS or a voice agent.
Visual generation — lip sync, facial motion, and sometimes gaze or expression synthesis.
Transport and playback — moving the resulting media to the client with minimal jitter and maintaining sync.
The transport layer is where media architecture starts to matter. If audio arrives 200 ms later than the corresponding mouth shape, the experience feels broken even if the raw render quality is high. If a client reconnects and loses session state, the avatar can desynchronize or restart awkwardly.
SFU-based architectures help because they support centralized coordination without forcing all media through a heavyweight transcoding server. In plain terms: the application can keep one authoritative session, but the media still flows efficiently. That becomes especially useful when you need:
One-to-many delivery for broadcasts or shared experiences
Late joiners that should receive the current stream immediately
Operational visibility into active sessions and media health
More predictable reconnect behavior across browsers and mobile networks
When peer-to-peer is the right choice
P2P is still valid. I would pick it when all of the following are true:
the session is strictly one-to-one
you control both endpoints reasonably well
you do not need fan-out, recording, or server-side moderation
you are optimizing for the simplest possible prototype
If you are doing a quick internal demo or validating the product concept, P2P can get you to “something visible” faster. It is also a fine default if the avatar is only ever shown to a single user and you can tolerate the occasional connectivity failure.
The moment your requirements start to include session management, visibility, or multiple consumers, the calculus changes. The operational simplicity of P2P is deceptive; you often trade one set of problems for another set that only appears at scale.
When SFU is the better default
Use an SFU when you need any of the following:
Reliability across networks. Corporate firewalls, restrictive NATs, and mobile handoffs tend to be easier to support with a media server in the loop.
Multiple viewers or participants. One avatar stream can be forwarded to many clients without N separate direct media negotiations.
Session control. You want to start, stop, inspect, or route sessions from your backend rather than from a browser tab.
Operational tooling. Session dashboards, usage accounting, and quality-based billing are easier to expose when the platform owns the media lifecycle.
Reusable integrations. A voice agent can gain a synchronized talking face without your app having to manage low-level video plumbing.
There is a performance cost to routing through an SFU, but for avatar products that cost is usually offset by lower integration complexity and better production behavior. In other words: if your product is “a realtime avatar,” the transport is part of the product, not just a networking detail.
Implementation details that matter in practice
Regardless of topology, a few engineering details are worth getting right:
Use a single session authority. The server should own session lifecycle, token issuance, and allowed participants. Do not let the browser become the source of truth.
Keep API keys off the client. Browser-exposed keys are a liability. Use backend-issued session data or an iframe/embed flow that never exposes credentials.
Model avatar state explicitly. “Speaking,” “listening,” “thinking,” and “idle” are useful states because they map cleanly to UI and orchestration.
Budget for jitter. Even good networks have short stalls. Your avatar pipeline should tolerate minor buffering without visibly breaking lip sync.
Plan for reconnects. A user refreshing the page should not require recreating the entire backend session if the media path can be resumed.
If you are implementing the backend yourself, a simple pattern is: create a session on the server, issue the client only the minimal credentials or session metadata it needs, and let your media layer manage the stream. For example, a backend can create a session with the REST API and then hand the frontend a short-lived session reference rather than a long-lived secret.
The exact request/response fields depend on the API docs, but the shape is the important part: session creation happens server-side, and the browser receives only what it needs to join.
How Protoface fits into this architecture
This is where Protoface is useful as a building block rather than a full-stack replacement. If you are already using LiveKit for your voice agent, the plugin ecosystem includes a LiveKit Agents integration that drops a synchronized avatar into the agent flow, so your voice agent gets a talking face without you wiring media synchronization yourself.
That matters most when you are choosing SFU over P2P. The plugin lets you stay focused on agent behavior while the media path stays aligned with the LiveKit session model. If you are using Pipecat instead, there is also a dedicated integration path documented in the Pipecat guide and package docs, but the architectural point is the same: the avatar becomes part of the realtime session, not a separate video toy bolted on afterward.
If you prefer to keep the browser completely free of backend credentials, the customer-managed iframe embed is the other practical option. It is especially useful when you want to add an interactive avatar to a website without standing up a custom media backend.
Pitfalls and trade-offs
A few common mistakes show up repeatedly in avatar systems:
Over-optimizing for latency at the expense of reliability. A slightly slower but stable SFU path often looks better than a marginally faster P2P path that drops on network changes.
Ignoring concurrent viewers. Even if your initial use case is one-to-one, product requirements tend to grow.
Mixing transport concerns into business logic. Keep session management, auth, and avatar behavior separate from the media transport layer.
Letting the frontend control critical state. Browsers are good clients, not good authorities.
If you are unsure, start by asking a simple question: “Will this ever need to be more than one user watching one avatar?” If the answer is maybe, SFU is usually the safer starting point. The extra infrastructure pays for itself quickly once you add even modest product requirements.
Conclusion
For realtime avatar streaming, peer-to-peer is best treated as a narrow optimization for simple one-to-one cases. SFU is the more durable default when you care about reliability, fan-out, operational control, or clean session lifecycle management. The bigger your product gets, the more the media transport behaves like product infrastructure rather than plumbing.
If you are building an avatar into a voice agent or interactive web experience, start from the session model and the network topology you actually need, not the one that looks simplest in a demo. For implementation details, SDK examples, and integration guides, see docs.protoface.com. If you want a working reference, the quickstarts on GitHub are the fastest way to map the concepts to code.