Introduction

If you are building realtime avatars, the first architecture choice is usually the one that determines whether the system feels simple or turns into a latency and ops problem later: do you send media peer-to-peer, or do you route it through an SFU?

This matters more for avatars than for ordinary video chat. A realtime avatar is not just “a camera stream.” It is usually a synthesized, lip-synced video track driven by an agent loop that also has to ingest audio, maintain conversational state, and keep latency low enough that turn-taking feels natural. By the end of this post, you should be able to decide when P2P is enough, when an SFU is the better fit, and what the scaling trade-offs look like in practice.

What you are actually scaling

The first mistake teams make is thinking they are scaling “video.” In an avatar system, you are scaling a combination of:

• Media transport: audio in, video out, sometimes both directions.

• Realtime control: session setup, auth, avatar selection, instructions, and state.

• Compute: speech-to-text, LLM, TTS, lip sync, and video generation or compositing.

• Fan-out: one avatar often needs to appear in many rooms or many browser sessions.

The transport layer is where P2P and SFU diverge. P2P optimizes for directness. An SFU optimizes for distribution. For an avatar that only ever talks to one user, P2P can be perfectly adequate. For anything that needs to scale across rooms, participants, or observability, the balance changes quickly.

Peer-to-peer: simplest path, narrowest envelope

In a P2P setup, the browser connects directly to the avatar session endpoint. If the avatar is generated server-side, the backend still exists, but the media path is direct between two peers rather than relayed through an intermediary.

That gives you a few nice properties:

• Lower infrastructure cost for one-to-one interactions.

• Fewer moving parts if the application is a single user talking to a single agent.

• Potentially lower latency when network conditions are good.

But P2P does not scale linearly with users. The number of direct connections grows with each peer relationship, and browser NAT traversal gets less predictable as the topology gets more complex. More importantly for avatars, P2P makes it harder to support situations where you want one avatar stream consumed by multiple participants or mirrored across several surfaces.

In practical terms, P2P is fine when you have:

1. One user, one avatar.

2. Short-lived sessions.

3. No need to fan out the same media to multiple clients.

4. Minimal operational requirements around recording, moderation, or observability.

The second you want a room, a shared conversation, a support queue, or an avatar embedded in multiple places at once, P2P starts to look like the “prototype that became architecture.”

SFU: better fit for fan-out and session control

An SFU, or selective forwarding unit, sits in the middle of the media graph. It does not transcode everything the way a traditional MCU would. Instead, it forwards streams between participants and can selectively route tracks based on subscription state, active speakers, or quality constraints.

For realtime avatars, that middle layer is usually the right trade-off when you care about any of the following:

• Many viewers, one avatar: one synthesized face can be consumed by multiple clients without each client maintaining a separate media relationship.

• Room topology: the avatar behaves like a participant in a call or a conversation room.

• Dynamic routing: you can attach, detach, or move sessions without rebuilding the media model.

• Operational visibility: session lifecycle and media topology are easier to reason about centrally.

The cost is obvious: an SFU adds another server hop, which adds some latency and another component to operate. In a purely one-to-one demo, that can be overkill. At scale, it is usually the more stable place to pay the complexity tax.

The important nuance is that an SFU does not magically solve avatar generation latency. If your speech pipeline takes 700 ms, moving media through an SFU will not make the avatar feel more immediate. What it does is keep the transport layer from becoming the thing that breaks once you add users, rooms, or multiple consumers.

Scaling comparison: where each architecture bends or breaks

There are three questions I would use to choose an architecture.

1. How many clients need the same avatar stream?

If the answer is one, P2P is attractive. If the answer is many, SFU is usually the correct default. Broadcasting a realtime avatar to multiple browsers through separate P2P sessions duplicates media handling and complicates session coordination. An SFU centralizes that fan-out.

2. Is the avatar embedded in a conversation system?

If the avatar is part of a voice agent, a support workflow, or a team room, the media topology should match the conversation topology. SFUs tend to map better to “participant in a room” semantics. P2P tends to map better to “single point-to-point interaction.”

3. Do you need operational control over sessions?

Once you need per-session rate limits, isolation, attach/detach semantics, or a clean auth boundary between your app and the browser, the media path should not be doing extra work. A central session layer plus an SFU makes it easier to keep those concerns separated.

There are also a few edge cases worth calling out:

• Mobile and flaky networks: P2P can be fine, but it becomes harder to reason about quality when every client negotiates separately.

• Multi-tenant SaaS: SFU-based session models generally make tenant isolation and usage accounting cleaner.

• Web embeds: if you want a browser-native avatar surface that can be dropped into arbitrary sites, you usually want the server to own more of the topology.

Latency, bandwidth, and compute: the real bottlenecks

People often over-focus on transport and under-focus on the avatar pipeline itself. A believable realtime avatar is sensitive to end-to-end latency, but the biggest contributors are usually:

• ASR turn detection and transcription delay.

• LLM response latency.

• TTS startup time and chunking behavior.

• Lip-sync and face rendering delay.

An SFU adds a modest transport hop. P2P removes that hop, but it does not remove the rest of the pipeline. If you have a 1-second generation budget, shaving 20 to 40 ms of media routing is nice, but not transformative. If you have to support 50 concurrent viewers of the same avatar, the network and connection overhead of P2P becomes the bigger problem than the extra hop through an SFU.

Bandwidth tells a similar story. P2P can be efficient for one client because it sends one direct stream. Once you have multiple clients, the sender or server often ends up duplicating work. SFUs reduce duplication by forwarding a single inbound stream to multiple subscribers in a topology designed for that job.

Where Protoface fits

Protoface is useful here because it gives you a clean separation between avatar/session management and your own agent logic. If you are wiring an avatar into a LiveKit voice agent, the plugin and the broader integration pattern are the shortest path to a synchronized talking face without rebuilding the media plumbing yourself. The same idea applies when you manage sessions directly through the REST API or Python SDK: your app controls the session, while the avatar layer handles the synchronized visual side.

For direct API-driven control, a basic session flow looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

The exact request fields depend on the API version in the docs, but the pattern is stable: create a session, attach it to your agent or embed, and let Protoface handle the avatar-specific realtime state. If you are working in Python, the SDK follows the same shape:

from protoface import Client

)

from protoface import Client

)

from protoface import Client

)

For a browser embed, the key operational detail is that the iframe keeps the API key off the client entirely, which matters if you want a customer-facing avatar on an arbitrary site without exposing credentials. That is the right model when the browser is just a rendering endpoint, not a trusted control plane.

Practical decision guide

If you are choosing today, I would use this rough rule set:

• Use P2P for a small, single-user proof of concept where you control both endpoints and do not need fan-out.

• Use an SFU when the avatar participates in rooms, needs to reach multiple clients, or you expect the topology to evolve.

• Use a managed session layer when auth, rate limiting, and operational separation matter more than shaving a small amount of routing latency.

One subtle but important point: “scales better” does not just mean more users. It also means fewer architectural rewrites when product requirements change. If you know the avatar will move from a demo to a production surface with multiple consumers, an SFU-based design is usually the safer long-term bet.

Conclusion

P2P is the simpler starting point for a single realtime avatar session, but SFU-based architectures scale better once you care about fan-out, room semantics, or operational control. For avatar systems, the transport choice should follow the conversation model, not the other way around.

If you are implementing this now, read the docs, start with the smallest topology that matches your use case, and measure end-to-end latency before optimizing transport prematurely. The quickest path from there is usually a managed avatar/session layer plus the media topology that fits your product, not the one that looks simplest on a whiteboard.

For implementation details and current API shapes, see docs.protoface.com.