Introduction

When you put a realtime AI avatar in the browser, you are no longer just streaming media. You are coordinating three independent systems with different failure modes: browser autoplay policy, microphone permission flow, and session concurrency/cost growth. The avatar may be technically “ready” before the browser will let audio play, the mic may require a user gesture before capture can begin, and your infrastructure may look fine in staging but fall over once hundreds of sessions become thousands.

This post is about planning for those constraints up front. By the end, you should be able to design a browser-based avatar experience that:

• starts reliably despite autoplay restrictions,

• handles microphone prompts without deadlocking the UI,

• scales sessions predictably with reasonable operational guardrails, and

• keeps latency and cost under control as usage grows.

For concreteness, I’ll use Protoface as the example avatar layer, since it exposes the same operational surfaces most teams end up needing: a realtime API, SDKs, embedded browser experiences, and LiveKit-based agent integrations.

Start with the browser’s two hard gates: autoplay and mic permission

Realtime avatar UX usually fails in one of two places before your backend is even interesting:

1. The browser refuses to play audio until there is a user gesture or the page has an established media engagement state.

2. The browser blocks microphone access until the user explicitly grants permission, and sometimes only after a trusted interaction.

These are not edge cases. They are the default behavior in modern browsers, and you should treat them as part of your product flow, not as errors.

Autoplay is a UX contract, not a technical afterthought

Most realtime avatar systems produce audio and video independently. The avatar can render its first frame as soon as the session starts, but audio playback is still governed by browser policy. If you try to auto-play speech before the user clicks anything, you will see one of three outcomes:

• audio starts normally, usually because the browser already has permission or engagement state,

• audio is suspended until a user gesture occurs, or

• your media element is created, but you never hear anything because no resume/play path was implemented.

The practical fix is simple: design a “Start” interaction that does all browser-gated actions together. That means:

• create or resume the audio context,

• attach the media element,

• request microphone access, and

• join the realtime session.

Do not split these across multiple buttons unless you want to debug timing bugs in production.

// Pseudocode: combine gesture-gated actions behind one button.

});

// Pseudocode: combine gesture-gated actions behind one button.

});

// Pseudocode: combine gesture-gated actions behind one button.

});

Mic permissions need a clear state machine

Mic prompts are the other place realtime avatar apps get stuck. A common anti-pattern is to begin session setup before you know whether the user will grant the microphone. That creates awkward failure states: the avatar connects, the agent speaks, and the UI is waiting on a permission prompt that the user may never answer.

A better model is to treat microphone access as a first-class state:

1. Idle — no session, no permission request.

2. Prompting — user clicked start; browser permission dialog is open.

3. Granted — capture is active; transport can stream audio up.

4. Denied — explain what failed and offer retry or text fallback.

This state machine matters because your system should degrade gracefully. If mic access is denied, the avatar can still be useful in a text-first mode, or you can route the user into a partial experience instead of leaving them on a spinner.

Implementation detail: if the product is conversational, the “ready” state should mean both sides of the conversation can actually flow. If your avatar can speak but the user cannot speak back, you have not started a realtime session; you have started a broadcast.

Session growth is usually a control-plane problem before it is a media problem

Once the browser UX is stable, the next failure mode is session growth. Realtime avatars consume resources in at least four places:

• server-side session orchestration,

• media transport and fanout,

• model or TTS/ASR latency budgets, and

• avatar rendering or synthesis cost, depending on your architecture.

Capacity planning here is less about raw CPU and more about concurrency envelopes and tail latency. A small increase in session count can push you over a threshold where queues build up, startup time grows, and users perceive the product as “slow” even though the system is technically healthy.

Plan around three rates, not one

When sizing infrastructure, separate these numbers:

• Session creation rate — how many new avatars/users per minute?

• Concurrent active sessions — how many are live at peak?

• Steady-state message/audio throughput — how much traffic each session generates once established?

Those three tend to scale differently. A product launch may spike creation rate for five minutes while concurrency stays moderate. A customer support deployment may have modest creation rate but long-lived sessions, pushing concurrency much higher. If you only optimize for one dimension, the other will surprise you.

Operationally, you want to know your hard limits before users do:

• maximum simultaneous sessions per environment or region,

• average and p95 session startup time,

• how long sessions live when idle,

• what happens when quota is reached, and

• whether you shed load at session creation or mid-conversation.

Design your fallback behavior before you need it

If your application exceeds capacity, the wrong behavior is to let browsers sit in an indeterminate connecting state. Realtime systems should fail fast and visibly. Good fallbacks include:

• queueing the user with an explicit wait message,

• downgrading to text-only interaction,

• serving a static avatar or cached intro while the session initializes, or

• rejecting new sessions cleanly once a tenant-specific limit is reached.

For web apps, this is especially important because browser state is sticky. Users will refresh, open duplicate tabs, and retry quickly. If your backend creates a second session every time they do that, your concurrency numbers will drift far above the behavior you planned for.

Two practical mitigations:

1. Bind a session to a stable user or tab identity when appropriate, so refreshes can reconnect instead of duplicating work.

2. Set explicit idle and max-duration timeouts so abandoned sessions do not quietly consume capacity.

Watch the quality-tier economics early

If you bill by quality tier, capacity planning is not just about uptime; it is also about cost predictability. Higher quality usually means more compute, more bandwidth, or longer synthesis time. That can be perfectly acceptable if it maps to conversion or retention, but you should make the trade-off explicit.

In practice, I would track:

• cost per minute of active session by tier,

• conversion or task completion by tier,

• startup latency by tier, and

• failure rate by browser/device segment.

Those numbers tell you whether a premium tier is actually improving the product or just increasing your bill.

How Protoface fits: keep the browser simple, push control into the service layer

For browser embeds, the cleanest pattern is to keep secrets and session orchestration off the client entirely. The iframe embed model is useful here because it avoids exposing API keys in the browser, while still letting you configure per-embed voice, instructions, origin allowlists, and rate limits at the embedding boundary. That is the right shape if you want to keep the frontend thin and predictable.

For server-side orchestration, use the REST API or Python SDK to create sessions, enforce your own admission logic, and observe usage. The exact request fields and session schema are in the docs, but the pattern is straightforward: create the avatar or session server-side, hand the browser only a short-lived session reference, and let the client join after the user gesture.

import requests

print(session)

import requests

print(session)

import requests

print(session)

If you are already using LiveKit for voice, the plugin path is the least disruptive way to give your agent a synchronized talking face. The agent keeps its existing audio conversational flow; the plugin adds the video layer without forcing you to rebuild the agent loop. For examples, see the repository linked from the quickstart docs or the plugin package on PyPI.

# Example shape only; check the plugin docs for exact setup

# Example shape only; check the plugin docs for exact setup

# Example shape only; check the plugin docs for exact setup

Practical capacity checklist

Before you ship a browser avatar to real traffic, verify these behaviors in a staging environment with real browsers:

• First-run autoplay behavior on Chrome, Safari, and Firefox.

• Mic permission accepted, denied, and dismissed flows.

• Tab refresh and duplicate-tab reconnect behavior.

• Idle timeout and explicit session teardown.

• Peak concurrent sessions with a load test that resembles real conversation duration.

Also test the unpleasant but common cases: users who never grant mic access, mobile browsers that are stricter than desktop, and transient network loss during a live conversation. Most of the “capacity” bugs people hit in production are really state-management bugs that only show up under partial failure.

Conclusion

Browser-based realtime avatars are easy to demo and harder to operate. The hard parts are not the avatar frames themselves; they are the browser policies, permission timing, and session lifecycle decisions around them. If you treat autoplay as a deliberate UX step, model mic permissions as a state machine, and plan for concurrency with clear limits and fallbacks, the system becomes much easier to run.

If you are implementing this stack, start with the relevant quickstart or integration guide, then validate the browser flow end-to-end under load. The docs at docs.protoface.com are the place to check the current API shapes and integration details, and the GitHub examples are useful when you want to see the surrounding agent or embed code in context.