Introduction

If you are building a voice agent with a talking face, the transport choice matters as much as the model choice. The question is usually not “WebRTC or not,” but whether the avatar should connect directly to each client peer-to-peer, or whether you should route media through an SFU. That decision affects latency, browser complexity, scaling behavior, recording, observability, and how painful it is to debug when the video and audio drift apart.

This post is a practical guide to those tradeoffs. By the end, you should be able to decide when P2P is the simplest correct choice, when an SFU is worth the added infrastructure, and what to inspect when a realtime voice+video agent feels “off” in production.

Start with the actual media graph

For a realtime avatar, the interesting path is usually:

speech input or ASR → agent reasoning → TTS or speech synthesis → avatar animation → rendered video + audio to the browser.

In a pure P2P design, the agent or media server negotiates a direct WebRTC connection with the browser. In an SFU design, both sides publish/subscribe to a media router, and the SFU forwards selected streams to each participant.

The key distinction is this:

• P2P optimizes for the shortest media path and minimal moving parts.

• SFU optimizes for fan-out, control, and easier handling of multiple consumers.

For one user and one avatar, P2P often looks attractive. For anything that needs more than one downstream consumer, or that needs operational control, SFU usually wins.

P2P: the lowest-friction default

With a single browser and a single avatar, P2P is hard to beat on simplicity. There is no central media router to scale, and if the transport is healthy you get a direct path between endpoints. That makes it easier to reason about latency: your round-trip consists mostly of encoding, network RTT, decode, and rendering. If you are close to the user geographically and the session is truly one-to-one, this is often enough.

For voice agents, P2P is especially appealing when:

• the session is one user talking to one avatar,

• you do not need server-side recording or transcription fan-out from the media path,

• you are optimizing for quickest time to first prototype,

• you can tolerate some browser/network edge cases without a central place to inspect media.

But P2P has two common failure modes in real products.

First, NAT traversal is not guaranteed to be boring. Some networks are friendly; some are not. WebRTC’s ICE machinery usually makes it work, but “usually” is not the same as “predictably across enterprise Wi-Fi, mobile, and restrictive proxies.”

Second, debugging is distributed. When lip sync looks wrong, you have to ask whether the problem is upstream synthesis timing, packet loss, jitter buffering, codec mismatch, browser render cadence, or the avatar pipeline itself. With no shared media infrastructure, you often end up chasing the issue at both ends.

SFU: more structure, more leverage

An SFU sits in the middle and forwards media without fully decoding and re-encoding every stream. In practice, that gives you a few important capabilities:

• Fan-out: one avatar stream can be delivered to many viewers without establishing N separate peer connections from the producer.

• Control: you can attach recording, switching, subscriptions, and metadata handling around a centralized media plane.

• Operational visibility: stats, stream state, and participant lifecycle are easier to observe when they pass through a shared system.

For voice+video agents, SFU architecture becomes compelling when the avatar is not just a one-off interactive widget but a reusable media source: customer support queues, group demos, multiplayer NPCs, live classrooms, sales rooms, or anything where one agent may need to be consumed by several clients or services.

There is a cost, though. SFUs add a network hop and another failure domain. If you do not need fan-out, that extra hop is pure overhead. Also, because an SFU is a shared control point, you need to think about provisioning, auth, and stream lifecycle more carefully than with a direct connection.

How to debug the tradeoff in practice

When teams say “the video feels laggy” or “the avatar is out of sync,” the root cause is usually one of a few buckets. You can debug the architecture choice by asking which bucket dominates your workload.

1) Measure glass-to-glass latency, not just network RTT.

For avatars, users care about when the face motion arrives relative to the audio they hear and the words they expect. A 40 ms network improvement can be irrelevant if your TTS or animation pipeline buffers 200 ms to stay smooth. If you can only instrument one thing, instrument end-to-end latency from agent output to rendered frame.

2) Separate transport issues from synthesis issues.

If audio arrives late but the video is on time, the problem is probably not “WebRTC” in the abstract. It might be TTS chunking, encoder buffering, or renderer scheduling. If both are late together, look at upstream generation or a congested media path. If they drift apart over time, check clocking, buffering policy, and whether one stream is being timestamped differently.

3) Watch packet loss and jitter before you blame the avatar model.

WebRTC will conceal some packet loss, but if the network is unstable you will see it as audio artifacts, video freezes, or larger jitter buffers. SFUs can help here because they give you a single place to inspect transport health. P2P gives you fewer knobs and less aggregate visibility.

4) Decide whether you need multi-consumer media now or later.

This is the biggest architectural fork. If your roadmap includes recording, live monitoring, audience view, supervisor view, or “same agent, multiple surfaces,” start with an SFU or at least design for one. Retrofitting fan-out into a P2P-only system is where migration pain shows up.

What to choose, concretely

A useful rule of thumb:

• Choose P2P if you are shipping a one-to-one avatar, want the fewest moving parts, and can accept that debugging is mostly end-to-end and browser-specific.

• Choose SFU if you need scale-out delivery, centralized visibility, recording, or any sort of shared session topology.

There is a middle ground too: prototype on the simplest path that exercises your product shape, but keep the media contract stable enough that you can swap the transport later. In other words, do not let “direct browser connection” leak into your business logic. Keep auth, session creation, and media identity separate from the transport itself.

Where Protoface fits

If you are building a LiveKit-based voice agent, the lowest-friction way to add a synchronized avatar is the LiveKit Agents plugin. The plugin turns an existing voice agent into a video agent without making you wire the avatar pipeline by hand; the agent keeps doing the conversational work, and the avatar surface stays synchronized with it.

In practice, this is the kind of place where transport tradeoffs stop being theoretical. You can start with a working agent, then evaluate whether the current deployment shape wants direct connectivity or a media router underneath. If you are using the LiveKit path, the plugin and examples are here: https://github.com/protoface-ai/protoface-plugin-pipecat and https://docs.pipecat.ai/api-reference/server/services/video/protoface for the Pipecat integration guide.

A minimal Python-side setup looks like this conceptually:

from livekit.plugins import protoface

from livekit.plugins import protoface

from livekit.plugins import protoface

If you need to create avatars or sessions programmatically, the REST API and Python SDK are the right surfaces. The API is authenticated with bearer tokens, so your backend can provision sessions without exposing secrets to the browser:

curl -X POST https://api.protoface.com/sessions \

curl -X POST https://api.protoface.com/sessions \

curl -X POST https://api.protoface.com/sessions \

Exact request fields vary by endpoint, so use the docs for the canonical schema: https://docs.protoface.com.

Common gotchas when debugging

Three mistakes show up repeatedly:

1. Assuming low RTT means low latency. Media buffering can dominate everything else.

2. Treating audio and video as independent systems. For an avatar, they are coupled; the user notices desync immediately.

3. Ignoring session topology until production. If you need supervised playback, recording, or multiple viewers, the architecture should reflect that from day one.

Also remember that browser behavior matters. CPU throttling, tab suspension, autoplay policies, and video element scheduling can all make a clean backend look broken. If the issue only appears on certain devices, inspect client decode and render performance before you start redesigning the transport.

Conclusion

P2P is the simplest path when you have one user, one avatar, and a narrow product surface. SFU is the better fit when you need fan-out, central observability, or production control over media delivery. The right choice is less about ideology and more about what you need to debug when things go wrong.

If you are implementing this in a real agent stack, start with a single path that matches your topology, measure end-to-end latency, and keep the media contract transport-agnostic. Then expand only when your product requirements force the issue. For implementation details, session APIs, and quickstarts, see the docs and the linked examples in the repository and integration guides.