Protoface REST API for Realtime Avatar Apps: Managing ICE Candidates from Python or TypeScript
Protoface REST API guide for realtime avatar apps: manage WebRTC ICE candidates with Python or TypeScript.
Introduction
Realtime avatar systems look simple from the outside: send audio in, get a talking face out. Under the hood, they are usually two coupled streams: a low-latency media path for audio/video and a control path for session creation, configuration, and lifecycle management. If you are wiring one into a voice agent or a browser app, the piece that usually causes the most friction is WebRTC signaling, especially ICE candidate handling.
Protoface gives you a clean way to attach a synchronized avatar to a realtime conversation without having to build the media stack from scratch. In this post, we will focus on the practical part developers tend to get stuck on: how ICE candidates work, why they matter, and how to move them through a REST API or SDK from Python or TypeScript.
By the end, you should be able to:
Understand where ICE candidates fit in a realtime avatar session.
Decide whether to use trickle ICE or a non-trickle exchange.
Use a REST API, Python SDK, or TypeScript client to pass candidates correctly.
Avoid the common failure modes that make WebRTC look “randomly broken.”
ICE candidates in a realtime avatar session
WebRTC connectivity is not magic. The browser or agent does not simply “connect” to a remote avatar endpoint. It first negotiates media parameters via SDP, then discovers network paths using ICE, and finally selects the best candidate pair that can actually pass packets. Candidates represent possible local and remote transport addresses: host, server-reflexive, and relayed.
For avatar apps, this matters because your session is not just a video stream. It is interactive, time-sensitive, and often behind NAT or firewall boundaries. In practice:
The client creates an offer and gathers local ICE candidates.
The server responds with an answer and its own candidates, or a way to fetch/submit them.
Both sides exchange candidates until ICE completes and media starts flowing.
If that exchange stalls, you may still see an established signaling session, but no audio or video. That is why reliable candidate handling is not an optional implementation detail; it is the connection itself.
Trickle ICE vs. batch ICE
There are two common ways to exchange ICE candidates:
Trickle ICE: candidates are sent as they are discovered.
Batch ICE: gather candidates first, then send them together with the SDP payload.
Trickle ICE is usually preferred for realtime avatar sessions because it reduces setup latency. The browser can start connectivity checks earlier, which often gets you to first audio faster. Batch ICE is simpler to reason about, especially in server-driven integrations, but it adds delay and can make poor-network cases slower to recover.
As a rule of thumb:
Use trickle ICE when your client or SDK supports asynchronous candidate exchange cleanly.
Use batch ICE when you want a simpler bootstrap flow or your runtime makes streaming signaling awkward.
For developers integrating avatars into voice agents, the actual trade-off is often “latency versus operational simplicity.” If your agent setup already has a signaling channel, trickle is usually worth the small extra implementation complexity.
What to send, and when
An ICE candidate message usually includes the candidate string plus a few routing fields such as the media section index, the username fragment, and sometimes the SDP mid. Exact schemas vary by implementation, and the Protoface docs should be treated as the source of truth for field names and endpoint behavior.
Two practical rules matter more than the exact JSON shape:
Send candidates after the offer/answer is established enough for the other side to associate them. Candidate exchange without a matching session context is useless.
Do not assume the first candidate is the best candidate. ICE will usually test multiple paths and settle on the best one.
In browser-based flows, you typically listen for local candidate events and forward them immediately. On the receiving side, you add candidates to the peer connection as they arrive. If the remote side is controlled via REST rather than a direct websocket or data channel, your app often becomes the signaling bridge: capture candidate events in one place, POST them to the API, and feed the returned remote candidates back into the peer connection.
Python: session setup and candidate forwarding
If your avatar session is managed from Python, the shape of the code is usually straightforward: create the session, then keep forwarding ICE candidates as they are discovered. The exact SDK methods depend on the object model, but this is the basic pattern.
The main thing to watch here is event ordering. In Python, especially if you are bridging between an async SDK and your own signaling loop, it is easy to enqueue candidates before the session object is fully initialized. Buffer them briefly if needed, then flush after the offer/answer state is ready.
Also watch for lifecycle cleanup. Realtime sessions should be explicit about teardown; stale sessions and unclosed peer connections are a common source of leaked resources and confusing ICE retries.
TypeScript: wiring candidate events cleanly
In TypeScript, the integration often sits inside a browser app or a Node signaling service. The key is to keep the candidate path asynchronous but deterministic: add local candidates to the API as they arrive, and add remote candidates to the peer connection only after the peer is ready to accept them.
If you are handling the remote side yourself, be careful not to call addIceCandidate before the remote description is set. In many browser implementations that produces an error, but in some environments it just fails silently. A small queue of pending candidates is often the cleanest fix.
REST API fallback: useful when you do not want SDK coupling
When you need full control over signaling, the REST API is often the simplest integration surface. It keeps your media client and your application backend decoupled, which is useful if the browser, voice agent, and session manager live in different processes.
From there, your application can exchange ICE candidates over whatever signaling channel fits your architecture: another REST endpoint, a websocket, or an internal job queue. The important part is consistency: keep the session identifier attached to every signaling message, and preserve ordering if your implementation depends on it.
One subtle gotcha: candidate payloads are not durable configuration. They are ephemeral connectivity hints. Do not cache and replay them across sessions. A fresh session should negotiate fresh media and fresh candidates.
Protoface in a LiveKit-style agent flow
If your avatar is attached to a voice agent rather than a browser UI, the integration often becomes simpler. The media plane is already there, so the avatar just needs to join the same realtime conversation and render the synchronized face. In that case, the LiveKit Agents plugin is the relevant surface: you drop the avatar into the agent pipeline and let the plugin handle the avatar-specific media wiring. The PyPI package and examples are the fastest place to start if your stack is Python-centric.
For that path, the practical value of the REST API and SDK is still the same: you create or manage the avatar/session, then let the agent and avatar exchange realtime media with the least amount of custom signaling possible. If you want a full reference implementation, the quickstarts linked from the main repo are useful starting points, and the docs at docs.protoface.com are where the exact request and event schemas live.
Common failure modes and how to debug them
Most ICE bugs are not random. They usually fall into one of a few categories:
Signaling mismatch: candidates are sent to the wrong session or before the peer connection is ready.
NAT traversal failure: host candidates are useless from the public internet; you need server-reflexive or relay paths.
Ordering issues: remote candidates arrive before the SDP answer is applied.
Firewall restrictions: UDP or TURN traffic is blocked, so connectivity checks never succeed.
When debugging, verify the basics in this order:
Session exists and matches the client you think it does.
Offer/answer exchange completes successfully.
Local candidates are being generated.
Remote candidates are being received and added.
ICE state moves to connected/completed, not just checking.
If you have access to browser devtools or SDK logs, look for the selected candidate pair. That will usually tell you whether the connection is using host, srflx, or relay and whether the issue is signaling or network reachability.
Conclusion
ICE handling is the unglamorous part of realtime avatar work, but it is also the part that determines whether your avatar actually appears and speaks in real time. The core model is simple: negotiate SDP, exchange candidates, and keep the session state aligned across your client and backend. Whether you do that from Python, TypeScript, or a REST-only integration, the same constraints apply.
If you are building against Protoface, start with the docs for exact schemas and session lifecycle details, then choose the integration surface that matches your architecture: SDK for application code, REST for explicit control, or the LiveKit plugin when the avatar should live directly inside an agent pipeline. The quickstarts in the GitHub org are also a good way to validate your signaling flow end-to-end before you wire it into production.