Debugging Latency in LiveKit Agents: SFU Routing vs Peer-to-Peer Media
Debug LiveKit agent latency by separating SFU routing, P2P media, and avatar sync with WebRTC timing and stats.
Introduction
When a voice agent feels “slow,” the root cause is often not the model. It’s media routing. In LiveKit-based systems, latency can come from the WebRTC path itself: whether media is flowing peer-to-peer, relayed through an SFU, or being transcoded/bridged by an application service. If you’re adding a talking avatar to a LiveKit voice agent, you need to know which hop is actually burning time before you start tuning the wrong layer.
This post walks through a practical debugging approach for separating SFU routing delay from peer-to-peer media behavior. By the end, you should be able to identify where latency is introduced, reason about the trade-offs in LiveKit deployments, and recognize when the avatar pipeline is the issue versus the transport path.
First, define the latency budget
For conversational agents, end-to-end latency usually means:
capture time on the client
uplink encoding and network transit
server-side audio handling and agent inference
downlink media delivery
rendering and playout on the receiver
When an avatar is involved, there’s an additional synchronization problem: the face video needs to stay aligned with synthesized speech. That means you care about both absolute latency and A/V skew. A system can be “fast enough” for audio but still feel broken if the avatar lip-sync lags by a few hundred milliseconds.
The useful mental model is: measure the media path independently from the agent path. If the model is fast but the user hears/see the agent late, transport and routing are your suspects. If the transport is clean but the response starts late, the agent pipeline is.
P2P vs SFU: what actually changes
In a direct peer-to-peer setup, media flows between endpoints with no media relay in the middle. That can minimize hop count, but it’s fragile in real networks: NAT traversal, firewall behavior, and topology changes can make direct paths unreliable or impossible. It also doesn’t scale well beyond small rooms.
An SFU (Selective Forwarding Unit) changes the problem. Instead of every participant sending media to every other participant, each endpoint sends a single upstream stream to the SFU, which forwards it to other participants. In LiveKit, that’s the standard architecture for rooms. The SFU generally does not “understand” your content; it forwards packets and may adapt routing, layer selection, and congestion handling. The upside is scalability and simpler connectivity. The trade-off is extra hop latency and more places for queueing, bandwidth adaptation, or packet loss concealment to show up.
In practice, the latency differences you’ll see are often not the pure forwarding delay of the SFU itself. They come from:
ramp-up time while WebRTC selects candidate pairs and establishes transport
server-side bitrate adaptation under constrained uplink/downlink conditions
packet jitter buffering on the receiver
additional transcoding or media processing in your app layer
So when someone says “SFU is slow,” the more accurate question is: which stage of the SFU-mediated path is slow?
How to debug latency without guessing
Start by isolating the transport from the agent. A good workflow is:
Measure room join time. If the participant sees the room immediately but audio/video arrives late, transport is less likely to be the issue.
Measure first packet to first render. WebRTC stats can tell you when packets start arriving versus when frames are decoded and rendered.
Check for direction-specific delay. One-way audio lag often points to uplink or server-side processing; one-way video lag can indicate encoding or decoder buffering.
Compare RTT and jitter across paths. A low RTT with high jitter is worse for live media than a slightly higher but stable RTT.
Useful signals to inspect:
jitterandpacketsLostfor media healthround-trip time from WebRTC stats
time from local publish to remote subscribe
frame decode delay and playout delay on the receiving side
If you control both ends, run a loopback test. Send a known audio burst or video cue and timestamp it at capture and render. This is much more trustworthy than trying to infer latency from user perception. If you can, test the same code path with and without the avatar layer, because avatar synthesis can introduce its own buffering for lip sync alignment.
Common debugging traps
1. Confusing signaling delay with media delay. Signaling gets the room set up; media delay is the time until packets are actually flowing and rendered. A fast signaling path can still hide a congested media route.
2. Measuring the server and the client as if they were the same clock. Unless you have synchronized clocks or a shared timestamp strategy, cross-host comparisons can mislead you. Prefer delta measurements within the same process or use an explicit timing marker end-to-end.
3. Ignoring adaptive bitrate behavior. If the connection is unstable, the SFU or sender may reduce quality to preserve continuity. That can improve smoothness while increasing perceived latency if buffers grow.
4. Assuming the avatar is just another video track. A synced talking face usually depends on speech timing. If the avatar service waits for enough audio context to generate accurate mouth motion, you may see a deliberate buffer. That’s not necessarily bad; it’s a trade-off between immediacy and lip-sync quality.
Where Protoface fits in this path
This is exactly where the LiveKit plugin is useful: it lets you drop a synchronized avatar into an existing voice-agent room without rebuilding your media stack. The relevant integration is the Python package published as pipecat-protoface or the LiveKit-oriented plugin surfaced in the quickstart repos, depending on your agent framework.
The key debugging benefit is that you can keep the LiveKit transport unchanged and treat the avatar as a separate media participant. That lets you answer a very specific question: is the lag coming from the room’s routing, or from avatar generation/synchronization?
If you’re wiring this into a LiveKit agent, keep the agent audio path and the avatar path instrumented separately. For concrete integration patterns and current SDK details, check the docs at docs.protoface.com and the Python SDK repo at github.com/protoface-ai/protoface-sdk-python.
A simple way to validate the path is to compare three timestamps:
audio chunk captured by the agent
avatar session accepted/started
first rendered frame or first synced mouth movement
If the gap between capture and avatar start is large, the issue is probably in the avatar pipeline or session orchestration. If the avatar starts quickly but the frame arrives late to the user, look at the LiveKit route, subscription timing, and client rendering.
A quick cURL sanity check for session setup
Sometimes the fastest way to rule out application logic is to verify the API path directly. If you’re creating sessions through the REST API, a minimal check can look like this:
The exact request shape may differ, but the point is to verify that session creation itself is fast and stable before you go back to the LiveKit room. If REST is quick but the live experience is not, transport or synchronization is still the likely culprit.
Practical tuning guidance
Once you know the bottleneck, the fixes are fairly mechanical:
If join/setup is slow: reduce room churn, reuse sessions when possible, and avoid unnecessary reconnects.
If media is jittery: inspect network quality, packet loss, and whether your client is on a constrained uplink.
If video lags audio: check decoder buffering and whether the avatar pipeline is intentionally holding frames for sync.
If the avatar lags the agent: profile the speech-to-avatar boundary, not just the agent model.
For developers using LiveKit Agents, the important thing is to treat the avatar as part of the realtime media system, not as a cosmetic overlay. That framing leads you to the right instrumentation and makes the differences between P2P and SFU routing visible instead of speculative.
Conclusion
Latency debugging in realtime agents is mostly about separating layers: signaling, transport, inference, and synchronized rendering. SFU routing adds scalability and reliability, but it also introduces hop-based behavior that can be mistaken for “model slowness.” Peer-to-peer paths remove some of that complexity but trade it for connectivity and scale constraints.
If you instrument each stage, compare transport with and without the avatar layer, and measure first-packet-to-first-render instead of relying on gut feel, you’ll usually find the bottleneck quickly. For integration details, current SDK behavior, and practical setup examples, start with docs.protoface.com and the relevant plugin or SDK repository.