How to Build a Realtime AI Avatar App with Agora: Choosing SFU vs Peer-to-Peer Media
Build realtime AI avatars with Agora: compare SFU vs peer-to-peer media for low-latency, sync, scaling, and production sessions.
Introduction
If you are building a realtime avatar app, the media transport choice matters almost as much as the avatar model itself. The core question is not “can I stream video?” but “how do I route audio and video so latency stays low, sync holds, and the system scales without turning into a brittle pile of special cases?”
This post breaks down the practical trade-off between SFU and peer-to-peer media for AI avatar applications, with an eye toward voice agents, conversational video, and embedded web experiences. By the end, you should be able to choose a transport architecture, understand the latency and scaling implications, and wire a basic integration without exposing unnecessary secrets or creating a future migration headache.
What you are actually optimizing for
Realtime AI avatars are not just “video calls with a bot.” The pipeline usually looks like this:
speech in → ASR or text input → LLM reasoning → TTS → synthesized audio out, plus a lip-synced facial video stream that must stay aligned with the audio. That alignment is the hard part. Users tolerate a slightly slower response more than they tolerate a face that talks out of sync with the voice.
That means your transport layer needs to preserve a few invariants:
Low end-to-end latency so the avatar feels conversational.
Stable synchronization between audio and video.
Predictable scaling when many sessions run concurrently.
Simple client connectivity across browsers, mobile, and sometimes server-side renderers.
Once you frame the problem that way, SFU vs peer-to-peer becomes a concrete engineering trade-off instead of a vague architecture preference.
Peer-to-peer: simplest path, narrow ceiling
Peer-to-peer media means each endpoint sends media directly to the other endpoint. In the best case, that removes a relay hop and can shave some latency. For a two-party session, the topology is simple and the bandwidth path is easy to reason about.
For avatar apps, the catch is that “two endpoints” is often a fiction. Even if the user only sees one avatar, the system behind it may include a browser, an agent process, a TTS service, a video compositor, and sometimes a monitoring or recording path. Once you need to add another consumer of the same media, p2p gets awkward fast.
Common p2p constraints:
Network traversal: NAT and firewall behavior can force fallback signaling or TURN usage, which adds latency and cost.
Topological rigidity: adding a second participant often means renegotiation or a separate path.
Poor fan-out: one sender, many receivers is exactly where p2p loses its appeal.
Operational fragility: debugging connectivity becomes a support burden, especially across heterogeneous client networks.
For a demo or a tightly controlled one-to-one session, p2p can be fine. For production avatar workloads, it usually becomes the wrong default once you care about observability, reliability, or future extensibility.
SFU: the practical default for realtime avatars
An SFU, or selective forwarding unit, receives media from one participant and forwards it to the others without decoding and re-encoding the stream. That matters because it keeps the media path low-latency while still centralizing routing.
For avatar apps, SFU is usually the better fit because it gives you:
A single connection model for clients, which simplifies browser and mobile integrations.
Room semantics for adding monitors, recordings, human handoff participants, or analytics consumers later.
Better scaling when one session needs to be observed from more than one endpoint.
More predictable media control for muting, switching tracks, or handling reconnects.
The important nuance is that an SFU is not “more latency” in the naive sense. It adds a forwarding hop, but in practice that hop is usually outweighed by the reliability and routing benefits. For conversational AI, the bottleneck is often synthesis, model inference, or client jitter, not the SFU itself.
When peer-to-peer still makes sense
P2P is still a reasonable choice if all of the following are true:
You have exactly one user and one avatar endpoint.
You do not need recording, supervision, or additional participants.
Your deployment environment is controlled enough that connectivity failures are rare.
You are optimizing for the shortest possible path and are willing to accept operational friction.
That describes a prototype more often than a product. If you expect to add coaching, moderation, or “join as a human” workflows later, start with an SFU. Rewriting transport under a live avatar product is usually more painful than paying a small complexity cost up front.
Designing the avatar session flow
For a sane production architecture, separate the concerns:
Control plane: create sessions, authenticate, assign policies, and manage lifecycle.
Media plane: carry audio/video with low latency and synchronization guarantees.
Application plane: the agent logic, prompts, tool calls, logging, and business rules.
This separation helps you avoid leaking API keys into the browser, makes it easier to rotate credentials, and lets you change the transport layer without rewriting your avatar logic.
A useful operational rule: never tie session creation directly to client-side media establishment unless the client can only ever act with scoped, short-lived credentials. In browser-based avatar products, that usually means a backend-issued session token or an embedded surface that keeps secrets off the page entirely.
Practical transport selection criteria
If you are deciding between SFU and p2p for a realtime avatar app, use a shortlist like this:
Need to support observers, moderators, or handoff agents? Choose SFU.
Need recordings or downstream processing? Choose SFU.
Need the absolute minimum moving parts for a temporary prototype? P2P can work.
Need predictable browser connectivity at internet scale? Choose SFU.
Need to keep the client simple and the server authoritative? Choose SFU.
For most AI avatar products, SFU is the engineering default. P2P is the exception you justify when the product scope is intentionally tiny.
How Protoface fits into this
This is where Protoface is useful in practice: it gives you a developer-facing avatar layer that can sit inside the realtime system you already have, rather than forcing you to build media, lip sync, and session management from scratch. If you are using a voice-agent stack, the LiveKit integration is the shortest path to a synchronized talking face; if you are building your own control plane, the REST API and Python SDK cover session and avatar management.
For example, if you already have a LiveKit voice agent, the plugin approach keeps the avatar attached to the agent process instead of creating a separate browser-only workaround. The exact wiring depends on your existing agent architecture, but the shape is straightforward: create or configure the avatar, then attach the Protoface media component to the agent session.
If you are provisioning sessions from your backend instead, the REST API is a better fit. Keep the API key server-side, create the session there, and hand the client only the minimum data it needs.
The Python SDK is useful when your backend orchestration is already in Python and you want to keep session logic close to the rest of your agent code. Exact method names and payload fields are documented in the SDK and docs, but the pattern is the same: create avatar/session objects server-side, then connect your media stack to them.
For browser embeds, customer-managed iframe flows are a better answer than trying to expose an API key to the frontend. That matters because the browser is the least trustworthy place to store long-lived secrets. An iframe also lets you enforce parent-origin allowlists and per-embed limits without forcing each customer to implement their own media backend.
Operational gotchas that matter in production
Three issues show up repeatedly in realtime avatar systems:
Audio/video drift: if audio and lip motion are produced by different clocks or different delivery paths, sync will slip under jitter.
Reconnect behavior: mobile browsers and flaky networks will disconnect; your session model should recover cleanly without duplicating state.
Backpressure: if your model or TTS path gets slow, the media layer should degrade gracefully rather than piling up stale frames.
SFUs usually make these easier to manage because there is a centralized session topology and a clearer place to apply policy. P2P can still work, but each failure mode becomes more client-specific.
Also remember that avatar quality tier affects cost. If you are experimenting, profile both perceived quality and total system latency before committing to the highest tier everywhere. In many products, the “good enough” tier is the one that lets you ship without sacrificing responsiveness.
Conclusion
For realtime AI avatars, the transport decision is mostly about operational shape. Peer-to-peer is a narrow, optimized path for a single controlled session. SFU is the architecture that survives real product requirements: more participants, better recovery, cleaner server authority, and fewer connectivity surprises.
If you are building a production avatar app, start with SFU unless you have a strong reason not to. Keep your media plane separate from your control plane, keep secrets out of the browser, and choose integration points that match how your agent actually runs.
For implementation details, API shapes, and quickstarts, see docs.protoface.com and the relevant examples in the GitHub organization. If you are integrating with an existing voice agent stack, start from the LiveKit plugin or the Python SDK and build outward from there.