Streaming a Real-Time Real Estate Avatar with WebRTC in JavaScript
Build a low-latency real-time avatar in JavaScript with WebRTC, lip sync, session control, and backend-signed playback.
Introduction
If you are building a voice agent, a customer-support assistant, or a conversational product demo, the missing piece is often the face. A realtime avatar does not need to be “cinematic”; it needs to be synchronized, low-latency, and predictable under WebRTC conditions. That means the hard parts are not just rendering video, but wiring audio, animation timing, transport, and session lifecycle so the avatar looks like it is actually speaking in the same realtime loop as the agent.
This post walks through the practical shape of a realtime avatar streaming pipeline in JavaScript: how the browser receives and renders a live video track, what WebRTC is doing for you, where lip sync comes from, and what integration boundaries matter if you want to ship something robust. By the end, you should be able to reason about the architecture, connect an avatar stream to a web app, and know where to put the backend logic versus the frontend plumbing.
The core model: WebRTC is a media transport, not the avatar system
It is useful to separate three concerns:
Conversation control: deciding what the agent says and when it speaks.
Avatar generation: turning speech into a synchronized talking face.
Transport and playback: getting the resulting video/audio stream into the browser with low latency.
WebRTC only solves the third piece. It gives you NAT traversal, congestion control, jitter buffering, and realtime media delivery. It does not generate the avatar, and it does not guarantee lip sync by itself. If the upstream system emits video frames that are not aligned to the audio timeline, the browser will faithfully play the mismatch.
For a realtime avatar, the practical requirement is that the video stream be generated from the same speech event that produced the audio, or at least from a shared timing source. In other words, “video follows audio” is not a UI detail; it is the product requirement. If the agent is interrupted, barge-in needs to stop the current utterance and the current mouth motion together. If the model streams partial tokens, the avatar pipeline needs a clear policy for when to begin viseme generation, when to hold, and when to flush.
How the browser actually plays a streamed avatar
In a JavaScript client, the browser usually receives a remote MediaStream via an RTCPeerConnection. The media is then attached to a <video> element or rendered through a custom compositor. For avatar use cases, a plain video element is often enough unless you need overlays, cropping, or background composition.
The basic flow looks like this:
Join or create a WebRTC session.
Negotiate the remote media tracks.
Attach the incoming video track to a video element.
Keep the element muted/playsinline/autoplay-friendly to avoid browser policy issues.
A minimal browser-side pattern looks like this:
Two details matter in practice. First, autoplay policies can block playback unless the element is muted or initiated from a user gesture. Second, network jitter can create a perceptible delay between spoken audio and the facial motion if the sender is buffering too aggressively. You want the avatar to feel “live,” not perfectly recorded.
Engineering for low latency and believable motion
The technical goal is not “high quality video” in the abstract. It is “consistent motion under variable network conditions.” That means you should care about the following:
End-to-end latency: time from agent response generation to visible mouth movement.
Jitter tolerance: how much timing variation the playback pipeline absorbs.
Interruptibility: whether you can cut off the current utterance cleanly.
Session isolation: whether one user’s avatar state can leak into another session.
For lip sync, the common approach is to drive facial motion from speech phonemes or visemes derived from the utterance. The exact implementation varies, but the important engineering invariant is that the animation timeline must be locked to the utterance timeline. If the audio starts 150 ms before the facial pose, users notice. If the avatar keeps moving after the agent has been interrupted, users notice that too.
From a browser perspective, you should treat the avatar video like any other live media track: expect it to stall, recover, and renegotiate. Handle track replacement without remounting the entire UI. If you are building a conversation widget, preserve the DOM element and swap the underlying stream rather than tearing down the player state on every turn.
Another useful practice is to design for degraded states. If the avatar stream is temporarily unavailable, show the text transcript or a static placeholder rather than a blank tile. Realtime systems fail in partial ways; your UI should too.
Backend responsibilities: session lifecycle and security
Do not expose privileged session creation logic in the browser. A realtime avatar system typically needs a backend-controlled session lifecycle: create session, authorize client, bind the session to a specific user or conversation, and revoke it when done. This is especially true if the system can be instructed with custom prompts, voice selection, or session-specific constraints.
At a minimum, your backend should be responsible for:
Generating session tokens or exchanging short-lived credentials.
Creating the avatar session with the right configuration.
Binding the media session to your application identity.
Cleaning up sessions on disconnect, timeout, or cancellation.
If you are integrating a voice agent, the backend usually also owns the agent orchestration. The agent decides when to speak; the avatar runtime renders that speech as live video. That separation matters because the avatar layer should not need to know whether the text came from an LLM, a rules engine, or a human operator.
For API-driven systems, this is often just a small number of HTTP requests. A generic example against a session API looks like this:
The exact request schema depends on the endpoint, but the security model is the part to copy: API keys stay on the server, sessions are created server-side, and the browser receives only the minimum information required to join its own live session.
Where Protoface fits: dropping a live avatar into an agent
Protoface is useful here because it sits at the boundary between the agent and the media transport. If you are already using a LiveKit-based voice agent, the LiveKit plugin lets you attach a synchronized video face to the agent without building a custom avatar renderer yourself. The plugin is designed to make the agent’s spoken output and the avatar’s visible speech part of the same realtime loop.
A typical Python integration looks like this:
If you prefer to orchestrate sessions directly, the REST API and Python SDK are the cleaner surfaces. The SDK is a good fit when you want to create or inspect sessions from application code, while the REST API is useful for server-side automation and tooling. The important thing is that the session lifecycle remains backend-owned, and the browser only participates as a client of an already-authorized realtime session. See the docs at docs.protoface.com for exact request fields and current examples.
Browser integration tips that save time later
Even if your avatar stream works in a demo, production issues tend to come from the edges: autoplay, reconnects, and state synchronization. A few practical tips:
Use a user gesture to start media when possible, especially on mobile browsers.
Keep a stable video element and replace the MediaStream in place on renegotiation.
Watch connection state and surface explicit UI when the WebRTC peer connection drops.
Align transcript state with media state so the UI does not claim the agent is speaking when the stream is paused.
If your product has multiple avatars or multiple concurrent calls, isolate everything by session. That means per-session signaling, per-session media objects, and clean teardown when the conversation ends. Most “mysterious” avatar bugs are really session leaks.
Conclusion
A realtime avatar is a media system, a conversation system, and a session system stitched together. WebRTC handles the transport, the agent handles turn-taking, and the avatar renderer turns spoken output into believable motion. The implementation details matter: keep the backend in control of session creation, keep the browser focused on playback, and make sure lip sync is driven by the same timeline as speech.
If you are integrating this kind of experience into a voice agent or web app, start with the docs, then use a small end-to-end prototype before you optimize for polish. The quickest path is usually: create a session server-side, connect the client with WebRTC, and verify that interruption, reconnect, and autoplay behavior are all sane. From there, you can layer in your own agent logic and UI. For current setup details and examples, go to docs.protoface.com.