From Zero to Demo: Streaming a Voice and Video AI Avatar with Adaptive Simulcast
Build a realtime voice AI avatar with LiveKit, Protoface, and adaptive simulcast for synced lip sync and video streaming.
Introduction
Adding a talking avatar to a voice agent sounds straightforward until you actually wire the pipeline: audio turns, lip-sync timing, video transport, session lifecycle, and browser playback all have to stay in lockstep. If you get any of that wrong, the avatar looks delayed, uncanny, or just broken.
This post walks through the technical path from a blank project to a working demo: a realtime voice agent that streams a synchronized video face, with enough detail to understand where latency comes from and how to control it. By the end, you should know how to reason about the media pipeline, how adaptive simulcast fits into the picture, and where Protoface can slot into a production integration without exposing secrets in the browser.
What “realtime avatar streaming” actually means
A voice-and-video avatar is not a prerecorded clip with captions. It is a low-latency media session where:
the agent generates or receives speech audio in realtime,
the avatar service produces a lip-synced video stream from that audio, and
the client plays both streams with enough synchronization that the face appears to be speaking the audio the user hears.
In practice, the transport layer is usually WebRTC. That gives you built-in congestion control, jitter buffers, NAT traversal, and media negotiation. It also means you need to think about stream quality dynamically: a 720p avatar may be great on desktop fiber and wasteful on a throttled mobile connection. That is where simulcast matters.
Adaptive simulcast: why it matters for avatars
Simulcast means publishing multiple encodings of the same video track at different bitrates and resolutions. A receiver can select the layer that best matches current bandwidth and viewport size without renegotiating the whole session.
For avatars, this is especially useful because the perceived quality ceiling is not the same as for natural video. The face is synthetic, usually centered, and often shown in a fixed-size panel. You rarely need to push the highest layer continuously. Adaptive simulcast lets the client or SFU settle on a lower layer when network conditions degrade, then switch up when conditions improve.
The practical trade-off is simple:
Single encode: less complexity, but you either overprovision bandwidth or accept visible quality drops.
Simulcast: more encoder work and media plumbing, but better resilience and faster adaptation.
If you are building an interactive avatar product, adaptive simulcast is usually the right default because avatars are judged on responsiveness first, not pure frame fidelity.
Architecting the demo: audio, timing, and session state
The minimal architecture is:
a voice agent produces speech text or audio,
the avatar service turns that into a realtime speaking video stream,
the browser subscribes to the media session and renders the face, and
your backend keeps session state and credentials out of the client.
The two common implementation mistakes are:
treating video as a separate, loosely coupled asset instead of a media track that must stay synchronized with audio, and
putting service credentials into frontend code when an embed or backend-mediated session would avoid that entirely.
For production, the backend should own avatar/session creation, policy, and authorization. The browser should only receive ephemeral session material or an embed that is already constrained by your rules.
Using a LiveKit voice agent with a Protoface avatar
If your agent already runs on LiveKit, the fastest path is the LiveKit-oriented quickstart and the Protoface plugin for LiveKit agents. The idea is to drop the avatar into the agent pipeline so the agent’s spoken output is accompanied by a synchronized face without you writing a separate media stack.
Conceptually, your agent continues to own conversation logic, while the plugin handles avatar session attachment and video generation. The exact integration points depend on your LiveKit agent setup, but the shape is typically: initialize the plugin, create or select an avatar, then bind it to the agent session before streaming starts.
That example is intentionally minimal. The important part is the sequencing: establish the avatar before the conversation gets underway so the first spoken turn is already synchronized. If you attach late, you often end up with a disjointed first response or an awkward “face appears after the voice” effect.
If you are using Pipecat instead of LiveKit, the same pattern applies at a different layer. The Pipecat integration guide is the relevant reference point.
Creating and managing sessions from the API
When you need more control, use the REST API from your backend. This is the right choice when you want to create sessions on demand, enforce per-user policy, or track usage against your own application state.
A typical flow is:
your backend authenticates with an API key,
you create or select an avatar,
you create a realtime session,
you hand the client only the minimal session data it needs.
A request looks like this:
Fields will vary by endpoint, and the docs are the source of truth, but the important pattern is stable: API keys stay server-side, and the browser never talks directly to the management API.
The Python SDK is a good fit when you already have a Python backend and want typed, programmatic control over avatars and sessions.
Again, keep this illustrative. The exact method names and fields are documented in the SDK reference. The practical value is that your backend can create sessions as part of an application flow rather than through ad hoc manual API calls.
Adaptive simulcast in practice: what to watch for
Once the media pipeline is working, spend time on the operational details that actually affect perceived quality:
Initial layer selection: start conservatively on weak networks so the avatar appears quickly, then let the session scale up.
Viewport-aware rendering: do not waste bandwidth sending a high-res feed into a 320px panel.
Audio-first latency budgeting: users will tolerate modest video softness more than they will tolerate delayed speech.
Session cleanup: avatar sessions should end when the call ends; leaked sessions cost money and confuse usage tracking.
One useful mental model is that video quality should be dynamically negotiated, while conversational timing should remain stable. If the avatar “keeps up” with the voice but the resolution drops temporarily, that is acceptable. If the lip sync slips, it feels wrong immediately.
Where Protoface fits
Protoface is useful when you want the avatar layer to be a product capability, not a media R&D project. The developer-facing surfaces cover the common integration modes: a LiveKit plugin for voice agents, a REST API for session management, a Python SDK for backend automation, and customer-managed iframe embeds when you want to drop an interactive avatar into a site without exposing API keys in the browser.
For this specific demo, the most relevant pieces are the REST API and the LiveKit plugin: the former gives you explicit session control, and the latter lets an existing voice agent gain a synchronized video face with minimal wiring. If you want to go deeper, the public docs are the right place to confirm endpoint shapes, SDK signatures, and quickstart-specific setup details.
Conclusion
The core problem in streaming an AI avatar is not “render a face”; it is preserving the relationship between speech, video, and transport under realtime conditions. Adaptive simulcast helps you keep the experience responsive across changing network conditions, while a clean backend-owned session model keeps credentials and policy where they belong.
If you are building this for a voice agent, support flow, game NPC, or web experience, start with a small working path: create a session from your backend, attach the avatar to your agent, and validate lip sync and layer switching under real network constraints. Then harden the edges—auth, cleanup, and quality policy.
For setup details and the current API/SDK surface, start at docs.protoface.com.