How to Build a Realtime AI Sales Avatar in JavaScript with WebRTC and LiveKit
Build a realtime AI sales avatar in JavaScript with WebRTC, LiveKit, turn-taking, and lip-synced streaming audio.
Introduction
If you are building a sales agent that can actually hold attention, a text box or a voice-only experience is usually not enough. In practice, the interface has to feel present: the user speaks, the model answers, and a synchronized face on screen makes the interaction easier to follow and harder to ignore. The engineering problem is that this “simple” video face has to stay aligned with streaming audio, low latency turn-taking, and the browser’s realtime media constraints.
That is the gap this article addresses. By the end, you should understand how to wire a realtime AI sales avatar into a JavaScript application using WebRTC and LiveKit, what the moving parts are, where latency comes from, and how to keep the experience stable enough for a customer-facing workflow.
What a realtime avatar actually needs from your stack
At a systems level, a talking avatar is just another media participant in a realtime conversation. The avatar has to:
receive user audio with minimal delay,
produce agent audio in small chunks as the model speaks,
render a video stream whose lip motion tracks that audio, and
survive normal conferencing issues: reconnects, jitter, clock drift, and browser autoplay restrictions.
WebRTC is the right transport because it is built for low-latency interactive media, not file delivery. LiveKit gives you a practical realtime media layer: it handles rooms, tracks, signaling, and browser integration. Your AI agent can join that room as a participant, publish audio, and receive the user’s mic stream. The avatar then becomes a synchronized video track associated with the agent’s speech.
The important distinction is that you are not “generating a video.” You are running a realtime media pipeline where audio is the source of truth and video is a synchronized visual output. If the speech stream pauses, the face should pause. If the agent is interrupted, the avatar needs to stop cleanly and hand control back to the user without desynchronizing.
Architecture: browser, LiveKit room, and agent runtime
A clean implementation usually looks like this:
The browser creates or joins a LiveKit room.
The user publishes microphone audio from JavaScript.
Your voice agent subscribes to that track, runs STT, LLM, and TTS, then emits audio back into the room.
The avatar subsystem consumes the agent’s audio stream and publishes a video face track that is lip-synced to that audio.
There are a few practical implications:
Audio timing matters more than model latency. Even a strong model feels bad if your audio chunking and buffering introduce jitter.
Interruptions need explicit handling. If the user starts talking while the agent is speaking, you should stop or fade the response quickly and let the new turn win.
Video should follow the audio transport, not an independent clock. A separate rendering pipeline is how you get mouth drift and awkward pauses.
For a sales assistant, this architecture is useful because you can keep the experience conversational while still showing a visible “speaker” in the UI. That improves turn-taking cues: the user can tell when the agent is listening, thinking, or talking.
Implementing the browser side in JavaScript
In the browser, the main task is to join the room, publish the user’s mic, and render the agent’s remote tracks. The exact LiveKit APIs vary slightly by client version, but the shape is consistent: connect with a server-generated token, attach media, and subscribe to remote publications.
Two common gotchas:
Token generation belongs on the server. Never mint LiveKit tokens in the browser.
Autoplay policies still apply. If you expect audio playback without a user gesture, test that path explicitly in the target browsers.
For a sales flow, the UI should expose state transitions clearly: connecting, listening, thinking, speaking, and ended. Those states are not cosmetic; they help users tolerate the few hundred milliseconds of delay that realtime systems inevitably have.
Turn-taking, interruption, and quality trade-offs
The hardest part of a “natural” avatar experience is not the face; it is turn management. You need to decide when the agent should start speaking, when it should stop, and how aggressively it should react to user barge-in. In practice, a good implementation usually includes:
VAD or equivalent speech detection so you can detect when the user begins speaking.
Partial transcription or streaming text so the agent can respond without waiting for a full utterance in every case.
Response cancellation so the current agent turn can be interrupted cleanly.
Buffered playback on the agent side to absorb small network spikes without visible lip-sync issues.
You will also want to choose a quality tier intentionally. Higher visual fidelity and more advanced rendering can improve perceived quality, but they usually cost more and may introduce more latency or stricter bandwidth requirements. For a sales agent, the right choice depends on whether you care more about crisp presentation or maximum responsiveness.
One thing to avoid is over-optimizing the face at the expense of the conversation. Users forgive modest video imperfections much more easily than they forgive slow or interruptive speech. The avatar should make the call feel present, but the agent still has to sound fast and coherent.
Where Protoface fits: dropping an avatar into an existing LiveKit agent
If you already have a LiveKit-based voice agent, the cleanest way to add a synchronized face is the LiveKit plugin surface. The plugin is designed to slot into the agent process so the agent’s audio drives the avatar video without you having to build a separate lip-sync pipeline yourself. For a team already using LiveKit Agents, that is usually the shortest path from “voice bot” to “video bot.”
A minimal Python sketch looks like this:
If you are wiring this in for real, use the docs and the plugin repository examples as the source of truth for current arguments and lifecycle hooks: docs.protoface.com and the relevant integration repo on GitHub. The key point is architectural: your voice agent remains the agent, and the avatar becomes a media participant synchronized to the agent’s output.
For teams that need to create or manage avatars and sessions programmatically, the REST API is the other useful surface. A typical workflow is:
create an avatar or session with your backend,
store the returned identifiers,
hand short-lived room/session data to the browser or agent runtime,
keep API keys server-side only.
That request is intentionally schematic; the exact resource names and fields are documented in the API reference. The important operational detail is that the browser should never see your API key, and session creation should happen in your backend or another trusted environment.
Practical deployment notes for a sales use case
For a sales avatar, reliability matters more than cleverness. A few deployment rules tend to pay off quickly:
Keep the backend authoritative. Session creation, token minting, and policy decisions belong on the server.
Instrument latency end to end. Measure mic capture to first token, first token to first audio, and audio to first visible mouth movement.
Design for failure modes. If the avatar disconnects, the agent should degrade gracefully rather than leaving the user in a dead room.
Test in the browser you ship. WebRTC behavior, autoplay rules, and media device permissions vary enough that local success is not enough.
If you want to see a working implementation pattern, the quickstarts in the public repository are a good reference point for how the pieces are typically assembled, especially when you are trying to adapt an existing voice agent rather than starting from a blank slate.
Conclusion
The core idea is straightforward: use WebRTC for low-latency media, LiveKit for room and track management, and a synchronized avatar pipeline so the agent’s face tracks its speech instead of drifting behind it. Once you think about the system as a realtime media graph rather than a “video generation” problem, the implementation becomes much more manageable.
Start with a narrow vertical slice: one browser client, one LiveKit room, one voice agent, and one avatar. Get turn-taking and interruption handling right before you worry about polish. From there, use the docs at docs.protoface.com to fill in the exact API details and integration options for your stack.