How to Build a Real-Time AI Avatar Accessibility Layer in TypeScript for Screen Reader Users
Build a TypeScript accessibility layer for real-time AI avatars with ARIA live regions, state machines, and screen reader-friendly updates.
Introduction
Screen reader users can already get a lot out of conversational interfaces, but real-time AI avatars often break the accessibility contract: the visual layer updates continuously, while the semantic layer is either missing or so noisy that assistive tech can’t keep up. If you build a talking avatar on top of WebRTC or a streaming pipeline, you need to treat accessibility as a first-class real-time problem, not a post-processing step.
This post shows how to build an accessibility layer in TypeScript that sits alongside a real-time avatar stream and keeps screen reader users informed without overwhelming them. By the end, you should be able to:
Model avatar state transitions in a way that screen readers can consume reliably.
Debounce and prioritize updates so announcements stay useful under low-latency streaming conditions.
Expose the right ARIA semantics for transcript, turn-taking, errors, and connection state.
Integrate the layer with a live avatar session, whether you are driving it from your own agent or a hosted service such as Protoface.
What “accessibility layer” means in practice
A realtime avatar is not just a video element. It is usually the visible endpoint of a pipeline that includes microphone input, speech recognition, LLM inference, text-to-speech, lip sync, and transport over WebRTC or another streaming protocol. From an accessibility perspective, the critical state is not “frame 183 arrived”; it is:
who is speaking,
whether the agent is listening or responding,
what transcript has been finalized,
whether the session is connected or degraded, and
how to let the user interrupt or ask for repetition.
The accessibility layer is a small client-side state machine that converts low-level stream events into stable announcements and focusable UI. Think of it as a semantic projection of the avatar session. It should be independent from rendering details so you can reuse it whether the avatar is embedded in a canvas, a video tag, or an iframe.
Model the session as a finite state machine
The biggest mistake is to pipe raw event streams directly into an aria-live region. Streaming systems emit partial transcript updates, audio state changes, reconnect attempts, and visual animation events. If you announce all of them, you create noise. Instead, normalize events into a small set of states.
A practical model looks like this:
Connection: connecting, connected, reconnecting, failed.
Turn state: idle, listening, thinking, speaking.
Content: partial transcript, finalized transcript, error, retry message.
User controls: mute, stop speaking, repeat last response, skip.
Keep these separate. A speaking avatar can still be connected, and a connected session can still be waiting for the model. Conflating them makes announcements harder to reason about.
Once the state is explicit, you can decide what is worth announcing. For example:
Announce “Connected” once.
Announce “Listening” when the user can speak.
Announce finalized transcript chunks only, not every partial token.
Announce “Speaking” once, then update a visual transcript region silently.
That separation is the difference between usable and exhausting.
Use ARIA in a way that matches the interaction model
For screen reader users, the avatar should not be a mysterious video object. It should expose a concise control surface. The best pattern is usually:
a labeled container with the session name,
a status region for connection and turn state,
a transcript region that is readable and navigable,
buttons for interrupt, repeat, and mute if those actions are supported.
For live updates, prefer a dedicated polite live region for normal state changes and an assertive region only for failures that require immediate attention. Avoid making the transcript itself an endless aria-live feed. It is better as a normal DOM region with new finalized turns appended into a list.
Two implementation details matter a lot:
Debounce partials. Streaming ASR often emits rapidly changing text. Do not announce these unless they are meaningful to the user. Usually only finalized chunks should go to speech output, while partials stay visual.
Throttle state churn. If the pipeline switches listening/thinking/speaking multiple times per second, batch updates so screen readers hear the final state after a short settle period.
Implement a small accessibility controller in TypeScript
A good pattern is to keep a controller object that receives avatar session events and updates DOM nodes or framework state. The controller should be pure enough to test, and side effects should be limited to DOM writes and live region updates.
The exact event names depend on your agent stack and transport, but the shape is consistent: subscribe to the stream, normalize events, then emit stable semantic updates. If you are using React, this controller can drive state setters. If you are using vanilla TypeScript, it can update DOM nodes directly.
One subtle but important point: if the avatar is embedded in an iframe, the accessibility layer inside the iframe has to be self-contained. Do not depend on the parent page to supply ARIA labels or live regions. The parent can still coordinate focus and layout, but the iframe should expose a complete, keyboard-operable experience on its own.
Handle interruption, replay, and error recovery explicitly
Realtime voice interfaces need interruption support. Screen reader users cannot be expected to wait for a long synthesized response just to find the one sentence they need. Provide a clear stop-interrupt action that cancels playback and returns the session to listening. If the model or TTS stack supports barge-in, wire that through; if not, at least stop local playback and announce that the response was interrupted.
Replay is also useful, but it should replay the last finalized assistant turn, not the entire transcript. In practice, users want “say that again” more than “read me everything.”
For errors, differentiate between recoverable transport issues and hard failures:
Recoverable: reconnecting, temporary upstream latency, TTS retry.
Hard: invalid credentials, session expired, unsupported browser.
Announce recoverable issues politely and keep the control surface available. For hard failures, move focus to the error message and provide a path to restart or contact support. This is where assertive live regions are appropriate.
How this maps to a real avatar session
In a hosted avatar workflow, you usually create or manage a session on the backend, then stream events to the browser client. The accessibility layer sits entirely in the browser and subscribes to the session’s semantic events, not the video pixels.
If you are creating sessions with the REST API, keep the API key on the server. The browser should receive only a session identifier or a short-lived, scoped artifact. A simple request from a trusted backend might look like this:
The resulting session object can be used to connect your frontend avatar UI, which then feeds the accessibility controller. The important design choice is that the semantic layer stays in your app, where you control announcements, focus, and transcript persistence. That keeps you from depending on video playback state alone, which is not accessible enough for screen reader users.
If you are integrating through the LiveKit agent path, the same rule applies. The video face is just one output of the agent; the accessibility layer should subscribe to the agent’s transcript and state events independently. The plugin itself gives the voice agent a synchronized face, but you still need your app to render a textual and keyboard-accessible representation of the conversation. For examples, start with the quickstarts in the GitHub organization and the integration notes in the documentation.
Testing and rollout
Test with real screen readers, not just browser accessibility inspectors. Inspectors will tell you whether roles and names are valid; they will not tell you whether your live announcements are tolerable under actual streaming conditions.
A practical test plan:
Connect, disconnect, and reconnect the session.
Trigger rapid partial transcript updates and confirm only finalized content is announced.
Interrupt a long response and verify the state resets cleanly.
Force an auth or network error and confirm focus lands on the error.
Run the flow with NVDA, VoiceOver, or JAWS and listen for duplicate announcements.
Also verify that the experience still works when the avatar is embedded in a constrained layout or iframe. Small viewport changes often expose accidental focus traps or controls that are visually present but not keyboard reachable.
Conclusion
A real-time avatar accessibility layer is mostly about discipline: normalize noisy stream events, expose stable semantics, and keep the user in control when the agent is speaking. If you do that well, the avatar becomes an enhancement rather than a barrier.
The implementation pattern is straightforward in TypeScript: build a small state machine, drive polite and assertive live regions intentionally, and treat final transcripts as durable content. Then wire that layer to your agent or avatar session, whether you create sessions through the REST API, host the experience in an iframe, or attach a face to a LiveKit voice agent.
If you want to adapt this to your stack, start with the public docs at docs.protoface.com and the relevant quickstarts in the repository linked from the homepage. The core accessibility ideas here are transport-agnostic; you can apply them to any realtime avatar pipeline that emits meaningful session events.