Troubleshooting Agora Realtime Avatar Calls: SFU Handoffs, Peer-to-Peer Fallbacks, and Session Drops
Debug Agora avatar call issues: distinguish SFU handoffs, P2P fallbacks, and real session drops in WebRTC.
Introduction
Realtime avatar calls look simple from the outside: connect a voice agent, stream audio, render a face, and keep the session alive. In practice, the failure modes are usually in the transport layer, not the model layer. With WebRTC-based systems, you can get connected audio and then lose video when an SFU rebalances. You can get a clean peer-to-peer path that later falls back to relayed media. You can also see abrupt session drops that are actually NAT, ICE, or token-expiration problems rather than application bugs.
This post is a practical troubleshooting guide for those cases. By the end, you should be able to separate SFU handoffs from true disconnects, recognize when a P2P session is being replaced by a relayed session, and instrument your app so “the avatar disappeared” becomes an actionable event instead of a vague complaint. I’ll also show where Protoface fits when you need an avatar layer that plugs into an existing voice stack.
Start with the connection model, not the avatar
Most realtime avatar systems are built on top of a media transport that can change shape during a session. The avatar itself is just a synchronized video source driven by audio, lip-sync, and state. The thing that actually fails is usually the route between peers and media servers.
There are three paths worth distinguishing:
Peer-to-peer: media flows directly between client and agent when network conditions allow it.
SFU relayed: media is routed through a Selective Forwarding Unit when direct connectivity is not ideal or not possible.
Fallback / re-route: the session migrates from one transport mode to another due to NAT changes, path degradation, or policy.
For debugging, the important point is that these are not equivalent from a latency, observability, or failure semantics perspective. If your app assumes “connected once means connected forever,” it will misclassify a handoff as a drop. The UI symptom might be a frozen face, audio continuing without video, or a brief reconnection message with no recovery.
SFU handoffs: what they are and why they look like drops
An SFU handoff happens when a session moves between routing paths, often because the initial path is no longer the best option. In WebRTC terms, this can involve ICE restarts, candidate re-selection, or a change in upstream/downstream transport characteristics. In product terms, users just see a hiccup.
Common triggers include:
Mobile clients switching Wi-Fi to cellular.
Corporate networks that initially allow direct connectivity and later tighten UDP behavior.
Browser tab suspension or background throttling.
Transient packet loss that makes the current route look unhealthy.
The useful mental model is: an SFU handoff is usually a transport migration, not a session death. Your app should treat it as a state transition and preserve the conversation state, the avatar selection, and any local UI context.
What to look at:
ICE state changes: connected → disconnected → checking → connected is often recoverable.
Track replacement: audio or video track IDs changing without a full session teardown suggests a handoff or renegotiation.
Jitter / RTT spikes: sustained degradation before the event is a clue that the route was being abandoned.
If you own the WebRTC client, log the transport state machine, not just your app-level “session started / session ended” events. That gives you the evidence to distinguish “media rerouted” from “peer vanished.”
Peer-to-peer fallbacks: the path can change under you
Teams often optimize for the lowest-latency route and assume that if P2P succeeds once, it will stay that way. That’s not guaranteed. WebRTC stacks may choose a different candidate pair after a network change, or they may fall back to a relayed path when the direct one becomes unreliable.
From a developer standpoint, the main gotcha is that a P2P fallback can alter both performance and observability:
Latency increases slightly, which may affect turn-taking or lip-sync perception.
Bandwidth behavior changes, so a previously stable avatar stream can appear “blurrier” or slower.
Debugging gets harder because the session is still “up,” but its characteristics changed.
When investigating, compare:
Pre- and post-fallback RTT.
Audio packet loss versus video packet loss.
Whether the fallback affected only the avatar video track or the entire media session.
If only the video side degrades, the underlying audio path may still be healthy, and the issue could be codec renegotiation, encoder backpressure, or a render-side problem rather than transport collapse.
Session drops: when it really is a disconnect
A true session drop is different from a handoff. The session ends because one side stops sending keepalives, the signaling channel closes, credentials expire, or the client cannot re-establish ICE within the allowed window.
Typical causes in production:
Expired auth: short-lived session credentials or a stale token.
Signaling loss: WebSocket or control channel closed unexpectedly.
Browser limits: page navigation, tab discard, or autoplay restrictions preventing media from resuming cleanly.
Server-side cancellation: your agent process intentionally shut down or hit a timeout.
When a session truly drops, you should see a clean end-state at the protocol level rather than just a frozen face. The remediation is also different: instead of retrying the media path, you may need to mint a new session, refresh credentials, or restart the agent.
Practical debugging checklist
When a caller reports “the avatar disappeared,” work through these in order:
Confirm whether audio continued. If yes, the problem may be video track-specific.
Check transport state transitions. A disconnect/reconnect sequence points to handoff or fallback.
Inspect timing. If the problem aligns with network changes, browser backgrounding, or token expiry, that’s usually the root cause.
Verify session lifecycle. Make sure the session object outlives the UI component, and that cleanup code is not unsubscribing tracks too early.
Log media and signaling separately. One can fail while the other remains healthy.
Two implementation details matter more than most teams expect:
Keepalive and timeout values should reflect real network conditions, not just local dev behavior.
Rejoin logic should be idempotent. If a reconnect happens twice, your app should not create duplicate avatars or duplicate agent turns.
If you are building a browser UI, don’t key your rendering solely off a “connected” boolean. Track a richer state machine such as: connecting, media-established, degraded, handoff-in-progress, reconnecting, and ended. That makes your logs and UX much more accurate.
How Protoface fits into this
Where this becomes especially useful is when you add a synchronized avatar to an existing voice agent rather than building the whole media stack yourself. The LiveKit Agents plugin, livekit-plugins-protoface, is the cleanest path if your agent already lives in LiveKit. It gives the agent a talking face without forcing you to wire lip-sync and video generation by hand. The integration points and examples are in the plugin repo and the docs.
For example, in a Python agent you would typically initialize the avatar service and attach it to the session lifecycle. The exact constructor fields and session payload shape belong in the docs, but the pattern looks like this:
If you prefer direct API control, the REST API is the right surface for creating or managing avatars and sessions from your own backend. That is useful when you want explicit lifecycle control during retries or when you need to correlate avatar sessions with your internal call IDs.
And if you need a broader integration reference, the public docs at docs.protoface.com cover the exact session fields, auth model, and lifecycle events you should wire into your logs. For the LiveKit path specifically, the plugin repository is the most useful source of runnable examples: https://github.com/protoface-ai/protoface-quickstart-agora is a good reference for a transport-oriented quickstart, and the plugin examples show how to keep the avatar aligned with the agent even during reconnects.
Operational advice that saves time later
Three habits make this whole class of problems much easier to support:
Emit structured events for signaling, ICE, media track state, and application lifecycle. Correlate them with a session ID.
Surface degradations in the UI before you surface hard failures. Users tolerate “reconnecting” much better than a blank panel.
Test network transitions deliberately: switch networks mid-call, throttle bandwidth, background the tab, and let tokens expire in staging.
Do not assume your local success path tells you much. Realtime avatar systems tend to work well on a developer laptop with stable Wi-Fi and then fail under exactly the conditions your customers actually use: mobile, VPNs, flaky NATs, and long-lived sessions.
Conclusion
If you take one thing from this post, make it this: most “avatar dropped” incidents are transport-state problems, not avatar problems. Separate SFU handoffs from true disconnects, log the media state machine, and make your reconnect logic idempotent.
Once you can tell a path migration from a real session end, troubleshooting gets much faster. If you are integrating a voice agent, start with the LiveKit plugin or the REST API, and keep the session lifecycle explicit in your code. For the exact event shapes and configuration fields, refer to the docs. If you want working examples to adapt, start from the relevant quickstart in the GitHub org and test the failure cases early, not after users find them for you.