Introduction

Realtime avatars are deceptively simple at the product layer and annoyingly subtle at the engineering layer. The user clicks “start,” expects to hear audio, see a face begin speaking, and then carry on a natural conversation. In practice, you are coordinating browser autoplay policy, microphone permissions, WebRTC device setup, agent startup time, and the warm-up behavior of your model pipeline. If any one of those steps is slow or blocked, the experience feels broken.

This post focuses on the three failure modes I see most often when developers add a talking avatar to a voice agent: autoplay failures, mic-access friction, and start latency. By the end, you should have a practical mental model for building a startup flow that reliably gets to first audio and first frame without surprising users.

Understand the startup path before you optimize it

A realtime avatar flow usually has four phases:

1. The browser loads your UI and negotiates media policy.

2. The user grants microphone permission, or the app confirms it already has access.

3. The agent session starts and a media connection is established.

4. The avatar begins rendering video synchronized to the agent’s audio stream.

The important part is that these phases are partially independent. You can ask for mic access without starting the agent. You can preconnect a websocket or WebRTC session before a user clicks “talk.” You can even warm up the avatar side of the pipeline while the UI is still visible. The best implementations do as much work as possible before the user’s explicit action, but only within browser policy and product constraints.

Autoplay: assume the browser will block you until a user gesture

Modern browsers are conservative about media playback with audio. If you try to play audio or an autoplaying video with sound before a trusted user interaction, you will often get a rejection or a silent element. For avatar products, this means “just start speaking on page load” is usually not a valid assumption.

The reliable pattern is:

• Render the avatar element muted or idle initially.

• Wait for a user gesture such as clicking “Start” or “Talk to the avatar.”

• Start media playback and unmute only after that gesture succeeds.

• Keep the interaction atomic so the same click can also kick off agent/session startup.

For video avatars, even if the video itself can render, the browser may still require a user gesture for audio output. In practice, your “start” button should be the point where you both request playback and begin the realtime session. Do not split those into separate buttons unless you have a specific reason; every extra click increases the chances the user gets stuck in a half-started state.

A good UI pattern is to make the initial state explicit. For example:

// Pseudocode: exact fields depend on your UI and SDK

}

// Pseudocode: exact fields depend on your UI and SDK

}

// Pseudocode: exact fields depend on your UI and SDK

}

The order can vary depending on your stack, but the key is to keep both operations under the same user gesture when possible. If your media library supports “resume” semantics, use them. If it supports a muted pre-roll, that can also help. What you want to avoid is a flow where the agent starts, emits audio, and then the browser drops it because playback was not user-initiated.

Mic access: treat permission as part of the product flow, not a technical afterthought

Microphone access is the other common place where realtime voice experiences fail. The browser permission prompt is not just a checkbox; it changes how much startup work you can safely do before the user commits. If your app depends on the user speaking, the cleanest flow is to request mic access only after you have shown the UI state and the user has clicked the start control.

Two practical points matter here:

• Ask late, but not too late. A premature permission request feels invasive. A delayed request makes the UI look unresponsive if the agent is already waiting for speech.

• Handle denial as a first-class state. Users deny mic access, device selection fails, and browsers sometimes return a transient error if another app holds the device.

If you want to reduce friction, preflight the permission state when the page loads and branch your UX accordingly. If the mic is already granted, you can move directly into session startup. If it is blocked or requires a prompt, you can show a short explanation before the click that triggers the prompt.

In a browser-based app, the mechanics usually look like this:

async function ensureMic() {
}

async function ensureMic() {
}

async function ensureMic() {
}

That pattern is useful when your first step is permission discovery rather than continuous capture. If your agent needs to begin capturing immediately, keep the stream open and hand it to your WebRTC stack instead of stopping it. The larger point is to make mic access explicit in your startup state machine:

• idle — user has not clicked start

• requesting-permission — browser prompt or device lookup

• connecting — media/session negotiation in progress

• live — audio and avatar are active

• blocked/error — show a concrete recovery path

That structure is boring in the best possible way. It prevents the most common failure: the agent is technically running, but the user cannot hear it or speak to it, and there is no obvious way out.

Start latency: optimize the first two seconds, not the whole session

For realtime avatars, perceived quality is dominated by startup time. Users will forgive a little jitter later if the conversation begins quickly. They are much less forgiving of a blank screen, a frozen mouth, or a long pause after they hit start.

There are three broad contributors to first-response latency:

1. Client-side setup — UI rendering, permission prompts, device selection, and media element readiness.

2. Session setup — creating or joining the realtime transport, exchanging credentials, and syncing state.

3. Agent warm-up — model initialization, prompt loading, and any avatar-specific synchronization work.

For the first two, you can usually improve perceived latency with parallelism. For example, fetch session credentials while the user is reading the start screen, or preload the avatar container before the click. For agent warm-up, focus on reusing infrastructure where possible. If your stack tears down everything between turns or between calls, you will pay the startup penalty every time.

One useful pattern is to distinguish between session establishment and conversation initiation. The session can be ready before the first spoken response exists. That means the avatar can connect, show a “listening” state, and then speak as soon as the model has the first token or first audio chunk. Users perceive that as much faster than waiting until everything is complete before displaying anything.

Another practical trick is to make your loading states informative. “Connecting avatar…” is better than a spinner, and “Mic permission needed” is better than an infinite wait. When the user understands what is happening, short delays feel intentional rather than broken.

How Protoface fits into this startup flow

Protoface is useful here because it gives you a clean boundary between your app and the avatar/session layer. If you are embedding Protoface into a LiveKit voice agent, the livekit-plugins-protoface plugin lets the agent gain a synchronized talking face without you hand-rolling the avatar transport yourself. That makes it easier to keep startup logic focused on one thing: get the voice agent connected under a user gesture, then let the plugin handle the avatar side of the media sync.

For teams that want direct control, the REST API and Python SDK are the right layers for creating avatars and managing realtime sessions programmatically. The exact request fields are documented in the API reference, but the shape is straightforward: authenticate with an API key, create or start a session, and attach that session to your realtime stack.

import requests

print(resp.json())

import requests

print(resp.json())

import requests

print(resp.json())

That example is intentionally generic; check the docs for the exact payload your workflow needs. The important part is that the server-side boundary keeps your key out of the browser and lets you control session creation before the client even touches media.

If you are implementing a website embed instead, the iframe approach is the simplest way to avoid browser-side key exposure entirely. That is especially helpful when you want a low-friction start flow: the parent page can present a button, the iframe handles the avatar session, and you can keep the browser permission and autoplay constraints isolated to the embed surface.

Implementation checklist

If you are building this from scratch, I would validate the following before shipping:

• The first user gesture both initiates playback and begins session startup.

• The app has an explicit denied/blocked/error state for autoplay and microphone access.

• Mic permission is requested only when the user has signaled intent.

• Session setup can begin before the first spoken response is ready.

• You measure time to first audio and time to first rendered frame separately.

Those metrics matter because the slowest part is not always obvious. Sometimes the avatar is fine and the browser audio policy is the real bottleneck. Sometimes the media path is fast and your agent is just slow to produce the first reply. Instrument both ends so you can tell the difference.

Conclusion

Building a good realtime avatar experience is mostly about respecting the browser and the user. Make autoplay part of the explicit start gesture, request mic access at the right time, and structure your session startup so the user sees progress before the first response is ready. If you do that well, the avatar feels immediate even when the underlying pipeline is doing a fair amount of work.

For implementation details, check the docs at docs.protoface.com. If you are using LiveKit, the plugin repo and quickstarts linked from the docs are the fastest way to validate the flow end to end.