Introduction

When you add a realtime avatar to a voice app, the hard part is not generating video frames. The hard part is getting media flow right: who publishes audio, who subscribes to video, where the AI inference sits, how the avatar stays synchronized to speech, and what changes when you move from a local demo to a multi-user production system.

This post walks through the practical architecture choices behind realtime avatars with WebRTC: peer-to-peer versus SFU, how media flows through a voice agent stack, and what to watch for when you attach a talking face to an assistant. By the end, you should be able to reason about latency, synchronization, scaling, and deployment trade-offs for avatar sessions in a way that maps cleanly to real systems.

WebRTC media flow in a realtime avatar system

A realtime avatar pipeline usually has four logical stages:

1. Capture or receive audio from the user.
2. Send that audio to an agent for ASR, reasoning, and response generation.
3. Render or synthesize avatar video that is aligned to the response audio.
4. Deliver the media back to the client with low latency and stable synchronization.

WebRTC is a good fit because it handles jitter, packet loss concealment, congestion control, NAT traversal, and realtime media transport. But it does not define your product architecture. You still need to decide where media mixes, who forwards it, and how the agent gets attached to the call.

In a voice-agent avatar setup, there are usually two media tracks to think about:

• Inbound audio: user speech that feeds ASR and agent logic.

• Outbound audio/video: the assistant’s spoken response plus the synchronized face.

If the video is lip-synced to the spoken output, then the video renderer must be driven by the same utterance boundaries, timing model, or audio timestamps as the speech generator. If those clocks drift, the mouth motion will look off even if the transport is healthy.

Peer-to-peer: simplest path, narrowest envelope

Peer-to-peer WebRTC is the straightest line between two endpoints. The browser sends media directly to the agent, and the agent sends media back directly to the browser. For a prototype, this is often the fastest path to first frame.

The advantages are obvious:

• Low hop count and low latency.

• No media server to operate.

• Simple mental model for one-to-one conversations.

But peer-to-peer has sharp edges in production:

• Every client must maintain a direct transport path to the agent.

• Scaling to many viewers or multi-party rooms is awkward.

• Recording, observability, and server-side moderation become harder.

• Network conditions vary widely across residential NATs and mobile networks.

For avatars, peer-to-peer is usually only attractive when the agent is the only participant and the session is strictly one-on-one. Once you want supervisors, room recordings, or a separate inference worker, a server-mediated topology tends to be easier to operate.

SFU: the practical default for production media routing

An SFU, or selective forwarding unit, receives published tracks and forwards them to subscribed participants without decoding and re-encoding the media in the way an MCU would. For realtime avatars, that matters because you want to preserve latency and avoid putting a transcode bottleneck in the middle of your session.

A common pattern looks like this:

Client publishes microphone audio to the room.
Agent subscribes to that audio, runs ASR/LLM/TTS, and publishes assistant audio plus avatar video back into the same room.
Other clients, if any, subscribe to the assistant tracks as needed.

The SFU gives you several useful properties:

• Role separation: the browser is just a publisher/subscriber; the agent can be a backend participant.

• Multi-subscriber support: one avatar stream can be viewed by many participants without duplicating inference.

• Operational visibility: rooms, participants, track state, and session metadata are easier to inspect.

• Network resilience: forwarding tends to be more robust than direct connectivity across many client environments.

The main trade-off is that the SFU adds one more hop. For an avatar, that hop is usually worth it because the system is already dominated by ASR, LLM, TTS, and video synthesis latency. In practice, shaving a few milliseconds of transport by going direct often matters less than having a topology you can reason about and scale.

Synchronization: audio is the source of truth

A talking avatar is not just a video stream with a mouth that opens and closes randomly. The video must be synchronized to the assistant’s speech in a way that feels coherent. The easiest mental model is to treat the generated audio as the timeline anchor and drive the avatar renderer from that timeline.

That usually means:

• Generate the response audio first, or at least in chunked segments.

• Use audio timestamps or token/phoneme timing to schedule mouth shapes and facial motion.

• Keep the outbound video playout aligned to the same utterance boundaries as the audio.

Three things tend to break this alignment:

• Variable inference latency: if your TTS or video generation takes inconsistent time, the face can lag the speech.

• Buffer mismanagement: too little jitter buffering causes stutter; too much buffering increases perceived lag.

• Clock drift: if audio and video are generated on different timing assumptions, sync degrades over longer sessions.

For developers, the practical rule is simple: optimize for consistent end-to-end delay, not just raw throughput. Users care more about a stable conversational turn than a slightly faster but jittery one.

Scaling and topology choices

Once a realtime avatar leaves the demo stage, topology becomes an engineering problem, not a product detail. The right answer depends on whether the avatar is embedded in a single user session, shared across a room, or attached to a backend agent that may need to scale independently.

Use peer-to-peer if:

• the interaction is one user and one agent,

• you do not need server-side recording or moderation, and

• you want the simplest possible first implementation.

Use an SFU if:

• you expect multiple participants or observers,

• you want the agent to live as a backend participant, or

• you need better operational control over sessions and media tracks.

As soon as you add one of these requirements, the extra complexity of an SFU typically pays for itself. The agent integration becomes cleaner, and your client apps stop needing to know about backend media internals.

How Protoface fits into this architecture

This is the layer where Protoface is useful: it gives you a developer-facing avatar surface without forcing you to design your own renderer, session service, and delivery plumbing from scratch. In practice, you can attach an avatar to a voice agent, manage avatar and session lifecycles through the REST API or Python SDK, or use the LiveKit plugin if your agent already runs in that ecosystem.

For example, if you are building a LiveKit voice agent, the plugin path is the most direct way to add a synchronized talking face. The agent keeps handling conversation logic, while the avatar layer focuses on video delivery and timing. The exact configuration fields vary by integration, so treat the snippet below as illustrative and check the docs for the current parameter names.

from livekit.plugins import protoface

)

from livekit.plugins import protoface

)

from livekit.plugins import protoface

)

If you are creating sessions programmatically, the REST API and Python SDK fit better. A minimal API call looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

And the same shape in Python is straightforward:

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

If you are choosing between integration paths, the rule of thumb is:

• LiveKit plugin for voice agents already built around LiveKit.

• REST API or Python SDK for custom orchestration and backend control.

• Iframe embed when you need a customer-managed browser experience with no API key in the frontend.

The important part is that the media architecture stays consistent: the avatar is still just another realtime participant in the session, with audio-driven video sync and a transport layer that can be reasoned about like any other WebRTC system.

Implementation gotchas that actually matter

A few mistakes show up repeatedly in realtime avatar systems:

• Generating video independently of speech: this produces uncanny motion and broken lip sync.

• Assuming transport latency is the main problem: often the bigger issue is response buffering inside the agent stack.

• Exposing API keys to the browser: if the client can create or manage sessions directly, you need a safe embedding model.

• Ignoring rate limits and session limits: long-running avatar sessions can create cost surprises if you do not bound them.

For browser embeds specifically, a customer-managed iframe is a clean boundary when you want to avoid backend code on the host site. The parent origin can be allowlisted, and operational controls like voice selection, custom instructions, and session limits can stay server-side instead of leaking into the page.

Conclusion

The core architecture question for realtime avatars is not “can I stream video?” It is “how do I keep audio, avatar motion, and conversational state synchronized while staying deployable under real network conditions?” Peer-to-peer works for the narrowest case. An SFU is usually the production choice because it gives you a sane media topology for agents, observers, and operational control.

If you are implementing this now, start with the simplest topology that matches your product constraints, then move the media into an SFU-backed session once you need scale or observability. For integration details, current SDK examples, and quickstarts, see docs.protoface.com and the relevant example repos linked from there.