Building a Realtime AI Avatar with ElevenLabs Agents Over WebRTC in Python
Build a realtime AI avatar in Python with ElevenLabs Agents, WebRTC, LiveKit, and synced voice/video sessions.
Introduction
Realtime AI avatars are useful when a voice agent needs a face, not just audio. The core problem is that you have multiple asynchronous streams to keep aligned: user audio in, agent text and audio out, and a video face that must lip-sync to the outgoing speech without lagging behind or visibly drifting. If you get the timing wrong, the avatar looks uncanny even if the language model and TTS are good.
This post shows the practical shape of that system in Python: how a WebRTC-based agent keeps latency low, how the avatar side of the pipeline should be wired, and where the integration boundaries sit. By the end, you should be able to reason about the media path, choose an integration surface, and wire up a realtime avatar into a production voice agent without exposing credentials to the browser.
What WebRTC is doing in this stack
For developer-facing avatars, WebRTC is usually the right transport because it is optimized for realtime interactive media. It handles NAT traversal, jitter buffering, adaptive congestion control, and media timing so you do not have to build those pieces yourself. In practice, the avatar is just another participant in a session, receiving audio to drive speech timing and sending a video track back to the client.
The important architectural point is that the avatar renderer is not “playing a video file.” It is generating frames or a video stream synchronized to the speaking voice. That means your application has to think in terms of sessions, tracks, and state transitions:
Input path: microphone audio or text turns into agent speech.
Synchronization path: speech timing informs the avatar’s mouth and facial motion.
Output path: a live video track is published to the session and rendered in the client.
For the user, the avatar should feel like a single participant in the call. For you, that means minimizing round trips and keeping the media pipeline continuous. If you are already using a voice agent framework, the main question is how to insert the avatar without making the agent logic aware of video internals.
Starting from an ElevenLabs Agents voice pipeline
When you build on top of an ElevenLabs Agents flow, you generally already have the voice loop: transcribe audio, generate a response, synthesize speech, and stream that speech back to the session. The avatar layer belongs next to TTS, not inside your model logic. The avatar should consume the spoken output, not the raw text alone, because the actual audio timing is what determines lip sync.
In Python, the cleanest integration is to treat the avatar as a media plugin in the same agent process. The exact event hooks depend on your agent stack, but the pattern is stable: create the session, attach the avatar renderer, then feed the synthesized speech through the same realtime path the client hears.
The important idea is not the exact method names; it is the contract. The agent produces speech, the avatar consumes that speech timing, and the client renders the resulting video track in the same realtime session. If you preserve that boundary, you can change the LLM, change the TTS provider, or swap the transport layer without rewriting avatar logic.
Managing latency, sync, and failure modes
Most avatar issues in production are not “AI problems.” They are transport and synchronization problems. The common failure modes are predictable:
Audio/video skew: the avatar’s mouth keeps moving after speech ends, or starts too late.
Buffer buildup: the pipeline queues too much audio before the avatar consumes it, increasing perceived lag.
Turn-taking glitches: interruptions or barge-in cause the avatar to continue a previous phrase.
Session drift: reconnects or network jitter desynchronize the media state.
A few practical rules help:
Stream early, not late. Do not wait for full responses if your stack can stream TTS incrementally. The avatar needs timing, and users need the first syllable quickly.
Treat silence explicitly. A gap in output should usually mean the avatar is idle, not “still speaking.”
Keep one source of truth for speech state. If the agent believes it is done speaking but the avatar renderer is still draining audio, you will see visual artifacts.
Handle cancellation. If the user interrupts, cancel both the speech stream and the avatar playback path together.
WebRTC helps with transport latency, but it does not solve application-level timing mistakes. If your agent framework emits partial deltas, you still need to decide how aggressively to synthesize and how to gate the avatar’s mouth motion. In practice, you want the avatar to follow the audio clock, not the text clock.
Session and API design that holds up in production
It is tempting to make the browser talk directly to every service involved in the avatar pipeline. That is usually the wrong move. The browser should not know your API keys, and it should not need to coordinate avatar creation, session policy, and media permissions. Those belong in a backend or managed session layer.
A production-grade flow usually looks like this:
Your backend creates or looks up an avatar and starts a realtime session.
The backend hands the client a short-lived join token or session URL.
The browser connects over WebRTC and renders the video track.
Your agent process streams speech into the session and controls turn state.
If you need to create sessions programmatically, use the REST API from your server, not from the frontend. The exact fields vary by object type, but the shape is familiar: authenticate with an API key, create an avatar or session, then attach runtime parameters such as voice, instructions, or quality tier.
That server-side pattern matters because it keeps credentials out of the browser and makes your system easier to audit. It also makes it simpler to enforce per-session policies like duration limits, rate limits, or quality selection.
Where Protoface fits
This is the layer where the avatar problem stops being “custom media engineering” and becomes an integration choice. Protoface docs describe the realtime session and avatar surfaces, and for Python voice agents the most direct path is the LiveKit plugin in the ElevenLabs Agents quickstart. That plugin drops a synchronized talking face into the agent pipeline without forcing you to build your own WebRTC video publisher.
For teams already using LiveKit Agents, the practical value is that the avatar becomes another component in the agent graph rather than a separate web app or a custom browser integration. You keep your voice logic in the agent, your session management on the backend, and your media transport in LiveKit, while the avatar layer focuses on synchronized rendering.
Implementation notes and trade-offs
A few engineering choices are worth calling out before you build:
Quality tier versus latency: higher-fidelity avatars usually cost more compute and can add some startup delay. Choose the lowest tier that matches your product requirement.
Model independence: keep the avatar interface narrow so you can swap ElevenLabs, another TTS provider, or even a different agent stack later.
Browser compatibility: WebRTC playback should work in modern browsers, but your UI still needs fallback handling for denied camera/mic permissions or blocked autoplay.
Observability: log session start, connection state, speech start/stop, and interrupt events. Those timestamps are usually enough to diagnose sync issues quickly.
If you are debugging a system that “looks laggy,” measure the pipeline in segments: model response time, TTS start time, first audio packet, video track attach time, and client render time. Most teams only measure model latency and miss the rest.
Conclusion
A realtime avatar is mostly a synchronization problem dressed up as a media problem. If you keep the speech stream, avatar timing, and WebRTC session state aligned, the result is straightforward: a voice agent that feels present instead of disembodied.
For a production implementation, keep the browser thin, manage sessions from your backend, and integrate the avatar at the same level as your TTS output. If you are using LiveKit and ElevenLabs, start with the quickstart examples in the repository, then cross-check the session and API details in the docs before you ship.
Next step: read the documentation, wire up a minimal session in Python, and test the pipeline with interruptible speech so you can validate timing before you add product UI on top.