Introduction

For users with low vision or motor impairments, a conventional chat UI is often the wrong interaction model. Text-heavy interfaces assume accurate pointer control, fast typing, and strong visual scanning. A voice-first interface removes a lot of that friction, but a plain audio agent is still missing a useful affordance: state. Users need to know when the system is listening, thinking, or speaking, and they need feedback that works even when they cannot rely on visual precision.

This is where a realtime AI avatar becomes more than decoration. A synchronized talking face gives the user a stable target for turn-taking, clear speaking cues, and a more natural sense that the agent is present. In this post, we’ll build the architecture for a voice-driven avatar experience in Protoface terms: capture user speech, stream it to a voice agent, render a lip-synced avatar, and keep the interaction accessible for users who can’t depend on mouse-heavy or visually dense controls.

By the end, you should know how to structure the frontend, how the agent/avatar timing works, where WebRTC fits, and where the accessibility trade-offs are in a real implementation.

Designing for accessibility first

Accessible voice experiences are not just “speech-to-text with a button.” For low-vision and motor-impaired users, the critical design goals are:

• Large, predictable controls with keyboard focus and no drag gestures.

• Minimal state ambiguity so the user can tell if the agent is listening, processing, or speaking.

• Low interaction cost for starting, pausing, repeating, and ending a session.

• Graceful interruption so users can barge in without waiting for a long response to finish.

In practice, this means your avatar should not be the only cue. Pair it with explicit audio prompts and a few robust keyboard actions. If you can make the UI work with a single large “talk” control plus shortcuts like Space to speak and Esc to stop, you are already much closer to usable than a chat widget with tiny icons.

One useful pattern is to treat the avatar as a synchronized status surface, not the control surface itself. The control surface stays simple and highly accessible; the avatar reflects agent state. That separation keeps your UI resilient when the video render fails, when bandwidth is constrained, or when users choose reduced-motion settings.

How the realtime path works

A voice-driven avatar pipeline has three realtime loops running in parallel:

1. Audio capture and transcription: user speech is captured in the browser or client app and streamed to the voice stack.

2. Agent inference and turn management: the agent decides when the user has finished, generates a response, and emits audio.

3. Avatar rendering: the avatar receives timing cues from the spoken audio so lip motion stays synchronized.

In a browser, this usually lands on WebRTC or another low-latency streaming transport because you want sub-second turn-taking, not request/response latency. The important detail is that the avatar is not synthesized from text after the fact; it tracks the live audio signal or a closely coupled timing stream. If the agent starts speaking before the frontend knows it is speaking, the user experience feels broken. If the avatar lags behind the audio by even a noticeable fraction of a second, the lip sync looks off and the accessibility value drops because the visual cue no longer matches the auditory one.

That’s why you should define a few explicit states in your app:

• idle — avatar visible, not speaking.

• listening — user can talk; show a clear recording indicator.

• thinking — input ended; agent is generating a response.

• speaking — audio is streaming and the avatar is animated.

If you are using a screen reader, expose these states through ARIA live regions or equivalent announcements. The visual avatar should help, but it should never be the only source of truth.

Building the SvelteKit frontend

SvelteKit is a good fit here because it lets you keep the interaction logic close to the UI while still isolating browser-only APIs. The main implementation detail is that audio capture and realtime transport must run client-side. Anything that touches the microphone, WebRTC, or a live media element should be behind an onMount boundary or a {#if browser} guard.

A simple client component generally needs:

• a button to start and stop a session,

• status text for assistive tech,

• a video element or avatar container,

• keyboard handlers for Space, Enter, and Esc,

• cleanup logic so tracks and sessions end reliably.

Here is a minimal shape for the UI state. The exact transport code will vary based on your voice stack, but the state machine is the part worth getting right:

type Mode = 'idle' | 'listening' | 'thinking' | 'speaking';

}

type Mode = 'idle' | 'listening' | 'thinking' | 'speaking';

}

type Mode = 'idle' | 'listening' | 'thinking' | 'speaking';

}

For accessibility, avoid hiding the only controls behind hover actions or tiny icons. If the app supports wake words or continuous listening, still provide explicit stop controls. Users with motor impairments need a deterministic exit path when speech recognition goes wrong or background noise triggers unwanted input.

A second point that often gets missed is focus management. When the session starts, do not steal focus to arbitrary elements unless there is a clear reason. Keep the start/stop button focusable, announce state changes, and ensure the avatar never becomes a keyboard trap.

Connecting the agent and the avatar

The core engineering problem is not “show a face.” It is “keep the face consistent with the agent’s turn state and media timing.” If your agent is already producing streamed audio, the avatar should subscribe to the same session boundary or media event stream. That way, when the agent begins speaking, the avatar starts animating at the same moment, and when the audio stops, the mouth closes promptly.

In a LiveKit-based voice agent, a common integration point is a plugin that attaches an avatar to the agent process. The agent keeps doing what it already does for ASR, LLM, and TTS; the plugin handles the avatar sidecar. The nice part is that you do not need to build a separate rendering pipeline in your SvelteKit app. You can let the agent own the realtime media flow and keep the frontend focused on interaction and accessibility.

For example, the integration shape looks like this:

from livekit.agents import JobContext

await avatar.start(ctx)

from livekit.agents import JobContext

await avatar.start(ctx)

from livekit.agents import JobContext

await avatar.start(ctx)

That is intentionally schematic. The important part is the model: the voice agent and avatar are coupled at runtime, not via a separate post-processing step. If you want a concrete starting point, the plugin’s repo and examples are more useful than a generic code snippet: GitHub repository and the Pipecat integration guide at this reference are the quickest way to see how the media events are wired.

Two implementation gotchas come up frequently:

• Latency budget: if transcription, inference, or synthesis gets slow, the avatar becomes a lagging indicator. Keep turn detection tight and avoid unnecessary hops between services.

• Interruptibility: if the user starts speaking while the agent is talking, your stack should support barge-in. For accessibility, the ability to interrupt is often more important than finishing a long answer cleanly.

Using the REST API for session control

If you are not embedding the avatar directly in a voice agent process, the REST API is the cleanest way to create and manage avatars and realtime sessions from your backend. This is particularly useful if you want SvelteKit to handle UI and your server to handle session orchestration, auth, and audit logging.

Auth uses API keys in a standard bearer header. The exact request schema for avatars and sessions is documented, but the interaction pattern is straightforward:

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

In your SvelteKit app, you would normally call your own backend endpoint rather than the API directly from the browser. That keeps keys out of client code and lets you apply your own access control. The API is a better fit than client-side orchestration when you need to provision sessions server-side, rotate credentials, or attach app-specific metadata.

If you prefer Python for backend glue, the SDK gives you a lighter-weight path for automation and session management. Typical usage is only a few lines; again, exact method names and payload fields are in the docs:

from protoface import Client

)

from protoface import Client

)

from protoface import Client

)

Practical accessibility details that matter

A good accessible avatar experience is mostly a collection of small decisions:

• Keep text alternatives close: announce what the agent just said, or provide a transcript panel that updates live.

• Support reduced motion: if the user prefers it, dampen transitions and avoid extra animation beyond lip sync.

• Use high-contrast status colors: do not rely on subtle shades to distinguish listening from thinking.

• Provide repeat and pause controls: these matter more than fancy avatar customization.

• Design for session recovery: if media permission is denied or the connection drops, explain the failure and offer a retry path.

Also be careful with autoplay. If your app starts speaking immediately, make sure the browser gesture requirements are satisfied and that the session can still start from a keyboard-only flow. For users with motor impairments, any hidden prerequisite interaction becomes a real barrier.

Where Protoface fits

For this kind of app, the most practical path is to keep SvelteKit as the accessible shell and let the avatar/media layer be handled by the realtime platform. Protoface gives you that layer through the LiveKit plugin, the REST API, and the dashboard for session and usage management. The plugin is especially useful if your app already has a LiveKit voice agent, because it lets you add a synchronized video face without rewriting your agent architecture.

If you are starting from scratch, the quickest path is usually to read the docs, then pick the integration surface that matches your stack: browser-facing iframe embeds if you want no backend exposure, or the LiveKit/Python path if you already have an agent service. The public docs at docs.protoface.com cover the exact session and avatar fields, and the quickstarts linked from the project README are a good way to get from theory to a running demo without guessing at API shape.

Conclusion

The main takeaway is that an accessible voice avatar is an interaction system, not a visual effect. For low-vision and motor-impaired users, the right architecture combines keyboard-friendly controls, explicit state feedback, realtime audio, and a lip-synced avatar that reflects the agent’s turn state accurately.

In SvelteKit, keep the frontend focused on accessibility and session control. Put mic access and media transport behind client-only code, expose clear status states, and make interruption reliable. Then attach the avatar to the same realtime path as the voice agent so the visual cue stays in sync with the audio.

If you want to implement this with less infrastructure work, start with the relevant Protoface integration docs and choose the surface that matches your stack. The docs at docs.protoface.com are the right place to verify exact request shapes, session settings, and quickstart code before wiring it into production.