Best Media Topology for Realtime AI Avatars on Agora: SFU, Peer-to-Peer, or Hybrid
Compare Agora media topologies for realtime AI avatars: SFU vs P2P vs hybrid, with latency, lip sync, and scaling trade-offs.
Introduction
If you are putting a realtime AI avatar into a product, the media topology matters almost as much as the model or the rendering pipeline. For a voice agent with a talking face, you are usually moving at least two realtime streams: audio and a video track that is synthesized from the agent’s speech. The question is how those tracks should flow between client, media server, and avatar renderer.
This post compares three common choices on Agora: SFU, peer-to-peer, and hybrid. By the end, you should be able to choose a topology for your use case, understand the trade-offs around latency and scale, and avoid the usual traps around lip sync, congestion, and browser resource usage.
What you are actually optimizing for
For realtime avatars, “best” usually means a balance of four things:
End-to-end latency from user speech to avatar reaction.
Lip sync correctness between audio and the generated face video.
Fan-out cost when one avatar needs to reach many viewers.
Operational simplicity when the system grows from prototype to production.
The important detail is that the avatar is not just another webcam stream. It is a synthesized output of the agent pipeline, and that means the network topology affects both transport and coordination. A topology that looks fine for a one-to-one demo can become awkward once you need multiple participants, agent handoff, screen sharing, or a support queue with watchers.
Peer-to-peer: simplest path, narrowest fit
Pure peer-to-peer is the lowest-ceremony option. If you have one human user and one AI avatar, and the avatar stream is generated locally or in a very direct server-to-client path, you can avoid a media server entirely. That gives you the fewest moving parts and often the shortest route to a working prototype.
For realtime avatars, the downside appears quickly:
No efficient fan-out: each additional viewer adds another connection.
Harder NAT traversal at scale: WebRTC can work well P2P, but reliability degrades as network conditions get uglier.
Poor fit for multi-party sessions: once you want the avatar in a room with several humans, P2P becomes a mesh problem.
Less control over routing: it is harder to centralize recording, moderation, or compositing.
In practice, P2P is best when the avatar is truly private, the session is one-to-one, and you are optimizing for minimum infrastructure rather than operational flexibility.
SFU: usually the default for production avatar rooms
An SFU (Selective Forwarding Unit) is generally the right starting point for realtime avatar sessions that involve more than one participant or more than one downstream consumer. Each publisher sends one upstream stream to the SFU, and the SFU forwards the appropriate media to subscribers without transcoding the whole room in the middle.
That matters for avatars because the avatar is often a publisher of a synthesized video track plus audio, while the human participant subscribes. If you also want moderators, observers, or recorded sessions, the SFU can forward the avatar stream to all of them without turning your app into a custom relay.
Why SFUs work well for avatars
SFUs fit realtime AI avatars for a few concrete reasons:
One outbound stream, many viewers. A sales demo with one avatar and ten attendees is much easier on an SFU than on P2P.
Room-level coordination. You can keep the avatar, user audio, and any other participants in a shared session model.
Lower client complexity. Browsers subscribe to the media they need instead of negotiating a full mesh.
Better control over quality adaptation. The server can forward the most appropriate representation for each subscriber.
The trade-off is that SFU does not magically fix latency. It removes a category of scaling problems, but your avatar still depends on the speed of your speech-to-response pipeline, model inference, rendering, and the network path to the room. For lip sync, the avatar renderer should stay tightly coupled to the audio timing source, and the SFU should be treated as transport, not as a place to “fix” timing.
Hybrid: the practical answer for most real products
Hybrid usually means using P2P where it is appropriate and an SFU where scale or coordination demands it. For avatars, that often translates to one of these patterns:
Direct path for a private one-to-one interaction, then upgrade to SFU when extra participants join.
SFU for the main session, with selective local rendering or bypass paths for special clients.
Server-side avatar generation plus client-side WebRTC subscription, while the rest of the application stays outside the media path.
This is usually the right mental model if your product might start as a single-user assistant and later become a shared workspace, a customer-support room, or a game lobby. You do not want to bet on a fully meshed topology if you already know you will eventually need observability, moderation, or multiple viewers.
Decision guide: which topology should you choose?
Use this as a rough rule of thumb:
Choose P2P if the avatar is one-to-one, the session is ephemeral, and you want the smallest possible infrastructure footprint.
Choose SFU if you need multiple participants, observers, recording, or reliable fan-out for the same avatar session.
Choose hybrid if you expect the product to evolve, or if some paths are private while others need room-level distribution.
For most developer products, SFU or hybrid is the safer default. P2P is attractive early on, but avatars tend to accrete requirements: transcripts, analytics, co-browsing, handoff to a human, and customer-facing embeds. Each of those makes a room-oriented topology more valuable.
Gotchas that matter in production
1. Lip sync is not just a UI concern. The avatar’s video timing should be derived from the speech timing source used by the agent. If you buffer audio too aggressively or let video drift independently, the face will feel off even if the transport is otherwise healthy.
2. Don’t confuse transport latency with inference latency. If users hear a slow response, the bottleneck may be LLM generation, TTS, rendering, or WebRTC transport. Measure each stage separately before you start tuning the media topology.
3. Keep the avatar stream small and predictable. Realtime generated video is expensive. If you can avoid sending full-resolution video everywhere, do it. SFUs help because they let you forward one published stream to many subscribers instead of duplicating work in the application layer.
4. Design for session lifecycle. Avatar sessions are stateful: voice choice, persona instructions, quality tier, and room membership all need to line up. A topology choice that is awkward to resume, reconnect, or hand off will cost you later.
How Protoface fits into the media path
Protoface sits on the avatar side of the problem: it gives you the realtime talking face that you plug into your voice agent or embed into your product, while you choose the surrounding media topology. If you are building with LiveKit, the Quickstart examples and the public docs at docs.protoface.com are the most relevant places to see the integration patterns in context.
For a LiveKit-style agent flow, the shape is typically simple: your agent generates speech, Protoface turns that into synchronized avatar video, and your room topology decides how that stream reaches users. The plugin path is handy when you want the avatar to behave like part of the agent stack rather than a separate rendering service.
If you prefer to manage sessions directly, the REST API and Python SDK let you create avatars and sessions programmatically. Keep API keys server-side, and use the browser-facing iframe embed only when you want a zero-backend integration with origin allowlisting and rate limits enforced by the embed itself.
The practical takeaway is that Protoface handles the avatar layer; Agora-style topology choices still determine how that avatar is delivered to one user, many users, or a mixed room.
Conclusion
For realtime AI avatars, topology is a product decision, not just an infrastructure decision. P2P is fine for a narrow one-to-one case. SFU is the better default for rooms, viewers, and reliability. Hybrid is what you end up with when the product needs both.
If you are starting today, pick the simplest topology that matches your near-term session shape, but do not box yourself into a mesh if you already expect fan-out or multi-party interaction. For implementation details, integration examples, and session/quality controls, see docs.protoface.com.