Troubleshooting WebRTC Session Drops in ElevenLabs Agent-Based Avatar Apps
Debug WebRTC drops in ElevenLabs avatar apps: ICE, SDP, tracks, idle timeouts, and agent lifecycle checks.
Introduction
When an ElevenLabs-powered voice agent drops its WebRTC session, the failure usually isn’t “the avatar crashed.” It’s more often one of a small set of transport, negotiation, or media-flow problems: the peer connection never finishes, ICE can’t find a viable path, audio is fine but video stalls, or the session is being torn down by the application because the agent thinks the conversation ended.
This post walks through the practical debugging loop I use for realtime avatar apps: how to tell signaling failures from media failures, where to look in browser and server logs, what WebRTC state transitions actually matter, and how to keep your avatar session alive long enough for a real conversation. By the end, you should be able to isolate whether the drop is caused by network negotiation, agent lifecycle, or misconfiguration in the avatar pipeline.
Start by separating signaling, transport, and media
WebRTC failures often get lumped together because the user sees the same symptom: the face disappears or the call disconnects. Internally, there are three different layers:
Signaling: exchanging SDP offers/answers and ICE candidates through your app or provider.
Transport: establishing the peer connection, including STUN/TURN discovery and ICE nomination.
Media: sending and receiving audio/video tracks once the connection is up.
For agent-based avatar apps, that distinction matters because the voice agent may continue running even if the avatar track is gone, or the session may be intentionally closed by the agent when it interprets the conversation as complete.
Use the browser’s WebRTC internals before you change application code. In Chrome, open chrome://webrtc-internals and inspect:
iceConnectionStateandconnectionStatecandidate pair selection and bytes sent/received
track lifecycle events
RTT and packet loss trends right before the drop
If the state goes connected to disconnected briefly and then recovers, that’s usually a transient network issue. If it goes to failed, the transport path is broken and you should focus on ICE/TURN. If the peer connection stays healthy but the avatar freezes, look at the media pipeline upstream of WebRTC.
Typical causes of session drops in agent-based avatar apps
In practice, the causes fall into a few buckets.
1. ICE never converges
If the app is behind restrictive NAT, corporate firewalls, or mobile networks, your peer may need TURN relay to establish a path. A classic failure pattern is an offer/answer exchange that completes, followed by iceConnectionState: checking forever, then failed.
Things to check:
Are you actually getting server-reflexive or relay candidates?
Are UDP packets blocked and forcing TCP/TLS relay?
Does the candidate pair selected in the browser have bytes flowing in both directions?
If only one side can send media, you may have a half-open connection that looks superficially alive but drops under load.
2. The agent lifecycle ends the session
With voice agents, “session drop” can be a business-logic event rather than a network event. The agent may decide the interaction is done, the server may time out an idle session, or your app may reuse an avatar session incorrectly after the underlying media connection was already closed.
For ElevenLabs-style agent workflows, verify that your agent process is not emitting a terminal event when it receives a long pause, an empty transcript window, or an unexpected interruption. If your app couples “no speech detected” to “end call,” the avatar will look unstable even when the WebRTC layer is healthy.
3. Audio/video track negotiation mismatches
It is common to get audio working and then discover the video track never attaches, or vice versa. That usually means the SDP negotiated a codec or directionality the receiving side didn’t expect. In avatar apps, this can also happen if the avatar track is created after the peer connection is already in a state that prevents new tracks from being added cleanly.
Look for:
recvonlyvssendrecvmismatchescodec mismatch between browser and agent stack
track replacement instead of renegotiation
autoplay restrictions blocking remote audio playback, which can be mistaken for a disconnected call
4. Idle timeouts and watchdogs
Realtime systems often have watchdogs for safety and cost control. If no audio is flowing, a session can be terminated after a configured idle window. That is good hygiene, but it can surprise you when the agent pauses to think or when a frontend tab is backgrounded.
If the symptom is consistent after roughly the same amount of silence, assume timeout first. If it only happens under packet loss or when switching networks, assume transport instability first.
How to debug the drop systematically
The fastest path is to instrument the session with timestamps and correlate events across client, server, and agent logs.
Record when the peer connection changes state. Log
iceConnectionState,connectionState, andsignalingStatetransitions.Record when tracks start and stop. A track ending cleanly is different from a transport failure.
Record agent events. If the conversation manager emits “end session,” “timeout,” or “turn complete,” that’s likely the root cause.
Correlate with network changes. Switching Wi-Fi, VPNs, or browser tabs often produces “random” WebRTC failures.
A minimal browser-side logger looks like this:
If you’re using a Python backend to coordinate sessions, keep the session ID and the WebRTC state in the same log context. Even if the exact SDK calls differ, the pattern is the same: create the session, attach the agent, capture the session identifier, and log any terminal events before tearing down the peer connection.
For server-side probing, the quickest sanity check is to confirm that the session object exists and hasn’t already transitioned to a closed state:
That kind of check doesn’t debug WebRTC itself, but it tells you whether the platform thinks the session is still valid. If the API says the session is closed while the browser still thinks it is connected, your app is likely closing the wrong layer.
What to inspect in the agent stack
If your app uses a voice-agent framework, inspect the place where the avatar session is attached to the agent rather than only the frontend. In many systems, the avatar is just one output track in a larger pipeline. The failure can originate when the agent restarts, the TTS stream resets, or the audio sink is replaced.
For LiveKit-based stacks, the relevant integration point is the agent plugin, not the browser. The ElevenLabs agents quickstart is a good reference for the full loop, and the plugin itself is the layer that injects the avatar into the agent pipeline. When debugging, confirm that the plugin is initialized once per session and that you aren’t accidentally creating a second avatar connection on a hot reload or worker restart.
A subtle failure mode here is double teardown: the agent decides to end, the plugin cleans up media, and then the outer app also force-closes the room. That can leave you with misleading logs that look like a transport error when the real issue was lifecycle duplication.
Common gotchas that look like WebRTC bugs
Background tab throttling: browser timers and autoplay policies can interfere with media startup.
Proxy/VPN path changes: ICE may reconnect or fail when the client network changes mid-session.
Hot reload during development: the frontend remounts while the backend still holds the original session.
Mismatched session ownership: one service creates the session, another service tries to close it, and the cleanup races the media layer.
Silent audio source: the avatar can appear broken when the upstream TTS is simply not emitting frames.
When in doubt, reduce the problem. Try a single browser, a clean incognito profile, no VPN, and a known-good network. If the issue disappears, the bug is probably environmental rather than protocol-level.
Where Protoface fits
Protoface is useful here because it gives you a clearer boundary between your app and the avatar session itself. You can create and inspect sessions through the REST API, or coordinate them from Python without having to expose any API key in the browser. In practice, that makes it easier to separate “our app ended the session” from “the WebRTC transport died.”
If you are integrating with a server-side voice agent, the LiveKit Agents plugin is the most relevant surface. Use the plugin to attach the avatar to the agent, then debug the agent lifecycle and peer connection independently. If you are working from Python, the SDK is a convenient way to inspect whether the session exists, whether it is active, and whether your cleanup path is firing too early. For exact fields and supported operations, use the documentation.
Conclusion
Most “session drops” in realtime avatar apps are not random. They are one of four things: ICE couldn’t establish a path, the agent ended the session, track negotiation was wrong, or an idle timeout cleaned up a live conversation. The debugging move is to identify which layer failed, then correlate browser state, agent logs, and session lifecycle events.
If you’re building on a voice-agent stack, start with the agent/plugin boundary, then verify transport in the browser, then confirm the session state server-side. Once you can categorize the failure reliably, the fixes are usually straightforward. For implementation details and examples, start with docs.protoface.com and the relevant quickstart in the Protoface GitHub organization.