Introduction

Designing an inclusive conversational video agent is mostly about controlling the same interaction model you would for any serious web app: focus order, keyboard reachability, live-region timing, and predictable state changes. The avatar itself can be visually rich, but the surrounding experience still needs to work for screen readers, keyboard-only users, low-vision users, and people who simply do not want motion-heavy UI getting in the way of the conversation.

This post walks through the practical pieces: how to structure the component, how to manage focus during a realtime call, how to expose state changes with ARIA without making the UI noisy, and how to avoid common traps when streaming audio, video, and captions. By the end, you should be able to build a conversational video agent in TypeScript that is operationally sound and accessible by default.

Start with the right mental model

A conversational video agent is not “a video player with a chat box.” It is closer to a realtime communication surface with multiple concurrent signals:

• speech input from the user

• streaming text or partial transcripts

• agent audio output

• avatar video output

• connection and turn-taking state

Accessibility problems usually appear when these signals are treated as decorative UI instead of interaction state. For example, if the avatar auto-plays video without a visible control or if the transcript updates every few hundred milliseconds inside a live region, screen readers can become unusable very quickly.

For inclusive design, separate the concerns:

1. Conversation state: idle, connecting, listening, thinking, speaking, reconnecting, ended.

2. Media state: audio/video tracks available, muted, failed, paused.

3. Accessibility state: what should be announced, what should receive focus, and when the user must be able to intervene.

That separation makes the UI easier to reason about and keeps ARIA from becoming a pile of ad hoc announcements.

Build a semantic control surface first

In practice, you want a small number of clearly labeled controls rather than a huge canvas of clickable regions. The root should behave like a dialog or region with an accessible name. The call controls should be real buttons. The transcript should be a readable region, not a stream of divs with random tabindex values.

A minimal structure might look like this:

<section aria-labelledby="agent-title" aria-describedby="agent-help">

</section>

<section aria-labelledby="agent-title" aria-describedby="agent-help">

</section>

<section aria-labelledby="agent-title" aria-describedby="agent-help">

</section>

There are a few important choices here:

• <video aria-hidden="true"> is usually correct if the avatar is decorative and the spoken content is represented elsewhere through audio/transcript. If the video conveys unique information, do not hide it; give it an accessible alternative.

• role="status" with aria-live="polite" is good for connection events and turn changes. It should not be used for full transcript streaming.

• Keep controls as native <button> elements. Custom div-based controls add work for no benefit.

Also make sure the initial keyboard focus lands somewhere sensible. If the user clicked “Start conversation,” keep focus on the start/end control cluster so they can immediately mute or hang up without tabbing across the page.

Focus management during a realtime session

Focus handling is where many conversational UIs break down. In a normal form, focus changes are user-driven. In a realtime agent, focus often changes because the app state changes underneath the user. That can be okay, but only if you make the rules explicit.

Use a focus trap only when the agent is truly modal

If the conversation runs in a dedicated panel or overlay, trap focus while it is open. If it lives inline on the page, do not trap focus. Trapping focus in an inline widget is frustrating and can create dead ends for keyboard users.

When the user opens a session modal, store the previously focused element and restore focus when the session ends:

let previouslyFocused: HTMLElement | null = null;

}

let previouslyFocused: HTMLElement | null = null;

}

let previouslyFocused: HTMLElement | null = null;

}

That restoration step matters more than most developers think. It preserves the user’s place in the page after a session ends.

Do not move focus for every new message

The agent speaking should not steal focus from the user. Neither should each partial transcript update. Focus shifts are appropriate only for high-priority transitions, such as:

• session starts and the user needs to authorize microphone access

• a blocking error occurs and a retry button appears

• the user explicitly opens a transcript, settings panel, or feedback form

For everything else, use live regions or stable visible text. The user should remain in control of where they are on the page.

Use ARIA for state, not decoration

ARIA is most effective when it reflects actual UI state. It is not a styling tool and it does not make a bad interaction accessible. For conversational agents, the most useful patterns are usually:

• role="status" for non-urgent changes, like “Connected” or “Agent is speaking”

• aria-pressed for toggle buttons like mute/unmute

• aria-busy="true" while waiting on network or model startup

• aria-disabled="true" when a control is temporarily unavailable, paired with a visible explanation

Keep announcements short. “Agent is generating a response” is enough. Do not announce every partial token. Screen readers can fall behind fast if you push low-value updates into the accessibility tree.

If you need to expose transcript updates, consider buffering them and announcing only complete utterances or sentence boundaries. For a live transcript panel, let the text itself be visible and scrollable, but avoid firing a live announcement for each interim result.

Keyboard and multimodal input should share the same state machine

A recurring bug in realtime UIs is having separate code paths for keyboard, mouse, and voice. That leads to inconsistent behavior: the button says “Mute,” the keyboard shortcut does something else, and the session state gets out of sync. Instead, define one state machine and route all inputs through it.

In TypeScript, that often means a reducer-like model with explicit events:

type CallState = "idle" | "connecting" | "listening" | "speaking" | "ended";

}

type CallState = "idle" | "connecting" | "listening" | "speaking" | "ended";

}

type CallState = "idle" | "connecting" | "listening" | "speaking" | "ended";

}

Once the state machine is centralized, your UI becomes straightforward: buttons dispatch events, media callbacks dispatch events, and ARIA text is derived from state. That keeps announcements and focus decisions deterministic.

Handle media permissions and motion carefully

Realtime voice agents require camera and microphone permissions in many setups, and that introduces its own accessibility issues. The permission prompt itself can disrupt focus, and some users will deny access or rely on alternate input devices. Treat permission failures as first-class states with clear recovery paths.

Two practical points:

• Never auto-start audio in a way that surprises the user. A visible start button is better than implicit autoplay.

• Respect reduced-motion preferences. If the avatar UI includes animated transitions, check prefers-reduced-motion and reduce nonessential motion.

For captions or transcripts, keep contrast and line spacing reasonable, and make the transcript area keyboard-scrollable. If the transcript is dynamic, avoid replacing the entire DOM subtree on each update; incremental updates are less disruptive for assistive tech and for performance.

Where Protoface fits

In a production app, the avatar/video layer should be something you can add without rebuilding your own media pipeline. That is where Protoface is useful: it gives you a realtime avatar surface you can attach to a voice agent, while you keep responsibility for the accessible shell around it.

If you are already using LiveKit Agents, the LiveKit-facing plugin can drop a synchronized talking face into the agent flow without changing your UI model. The important part from an accessibility perspective is that the plugin handles the avatar stream; your app still owns focus, keyboard shortcuts, status text, and transcript presentation.

For a quick integration, the shape usually looks like this in Python, though exact fields belong in the docs:

from livekit.agents import Agent

)

from livekit.agents import Agent

)

from livekit.agents import Agent

)

If you are wiring the service directly, the REST API can create sessions and avatars server-side with bearer authentication. Keep API keys out of the browser and let your backend mint sessions or manage embeds. A request shape is typically:

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"avt_123","voice":"..."}'

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"avt_123","voice":"..."}'

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"avt_123","voice":"..."}'

The exact schema varies by endpoint, so check the docs before hardcoding anything. The point is architectural: use the avatar service for media, and keep accessibility logic in your app where it belongs.

Conclusion

An inclusive conversational video agent is mostly a disciplined frontend state machine with realtime media attached. The key decisions are simple but important: keep focus stable, announce only meaningful state changes, use native controls, avoid noisy live regions, and treat the avatar as presentation rather than interaction authority.

If you do that, the experience stays usable for keyboard and assistive technology users without becoming awkward for everyone else. When you are ready to implement, start from the public docs at docs.protoface.com and adapt the media layer to your stack, whether that is a LiveKit agent, a Python backend, or a customer-managed embed.