Introduction

Multimodal accessibility agents are harder than they look because they are not just “voice bots with a UI.” A usable system has to keep speech, video, captions, and keyboard input aligned under real network conditions, while still behaving like an ordinary application: stateful, debuggable, rate-limited, and secure. If you’re building for support, education, or task assistance, the hard part is usually not the model call; it’s the system design around streaming media and interaction timing.

This post walks through a practical architecture for those agents. By the end, you should be able to design a realtime avatar experience that supports live speech, synchronized facial video, captions for accessibility, and keyboard-driven control without turning your app into an unmaintainable pile of media callbacks.

Model the problem as one realtime session with multiple input and output streams

The cleanest mental model is a single session that carries four independent but coordinated channels:

• Audio in: user speech or other recorded audio.

• Audio out: synthesized or model-generated speech.

• Video out: a talking face or avatar lip-synced to the output speech.

• Text and control: captions, transcript, and keyboard events that can interrupt, steer, or request clarification.

Don’t treat video as a separate feature that happens “after” the agent speaks. The avatar needs to be coupled to the same conversational turn state as the audio so that mouth motion, gaze, and expression changes stay aligned with the word stream. In practice, the agent runtime should emit a single notion of turn boundaries, partial hypotheses, interruption events, and final responses. Everything else subscribes to that state.

That separation matters for accessibility. Captions should not be generated from the video pipeline; they should come from the agent transcript or ASR output so they can be rendered independently, indexed, and copied. Likewise, keyboard shortcuts should not be bolted onto the video player. They are control signals for the session: mute, pause, repeat last turn, skip, or switch mode.

Use a transport that can carry low-latency media and control events

For voice and avatar experiences, WebRTC-style transport is usually the right choice because it handles real-time audio/video with jitter buffering, congestion control, and peer-friendly latency characteristics. The important thing is not the protocol label but the properties:

• sub-second round trip for conversational interaction,

• separate media tracks for audio and video,

• a signaling path for session metadata and control messages,

• support for reconnect and state recovery.

Your application should assume that media can arrive out of order or be temporarily delayed. The agent runtime should therefore maintain explicit session state, including:

1. current turn id,

2. active speaker state,

3. caption transcript buffer,

4. interruption/ barge-in state,

5. last acknowledged keyboard command.

That state is what keeps the UI honest. If the user interrupts the agent mid-sentence, the transcript should stop advancing, the avatar should stop animating the old utterance, and captions should close the current segment rather than continue to “hallucinate” a completed sentence.

Captions are a first-class output, not an afterthought

Accessibility fails quickly if captions are treated as a postprocessing layer. For a realtime agent, you want two caption modes:

• Partial captions from streaming ASR or streamed model output, useful for live comprehension.

• Final captions for the completed turn, useful for review, logging, and compliance.

The implementation detail that matters is segmentation. Captions should be chunked by semantic or timing boundaries, not by arbitrary token counts. If the caption renderer receives a new partial segment every few hundred milliseconds, it can update in place without flicker. If it receives a new DOM node for every token, you’ll get noisy rendering and poor screen-reader behavior.

A good pattern is to emit caption events with stable identifiers:

{
}

{
}

{
}

The UI can then update a single caption line until the segment is final, at which point it can lock the text and begin the next segment. This keeps the visual presentation stable and makes it easy to map keyboard navigation to caption history.

Keyboard control is part of the interaction contract

For users who cannot or do not want to rely on audio, keyboard support should be complete enough to operate the session without touching the mouse. That means more than tab order. It means defining an explicit control surface.

At minimum, support actions such as:

• start and stop listening,

• mute/unmute output,

• repeat the last assistant turn,

• open/close captions,

• pause avatar motion if the user needs a static presentation,

• submit text input as an alternative to speech.

Technically, keyboard control is just another event stream. The important part is priority. A user keypress should preempt low-priority animation and can preempt agent speech if it maps to interruption. If the user hits “skip,” the session should stop speaking immediately and mark the current turn aborted. If they hit “repeat,” replay the last final assistant turn, not the current partial hypothesis.

Two implementation gotchas show up often:

1. Focus management: if your captions, transcript, and controls are in the same page, make sure focus is never trapped in the avatar container. Preserve a logical tab sequence and expose accessible labels for every control.

2. State drift: keyboard commands should go through the same session state machine as voice commands. If you mutate UI state locally without informing the agent session, the avatar and captions will drift out of sync.

Keep the agent orchestration deterministic enough to debug

Realtime systems become painful when every component “helps” by making hidden decisions. Instead, centralize orchestration in a small state machine and keep the branches explicit. A practical session loop looks like this:

1. User audio or text arrives.

2. ASR or text normalization produces a partial transcript.

3. The agent generates a response incrementally.

4. Speech synthesis or audio streaming begins as soon as a stable prefix exists.

5. The avatar renders that same output stream with synchronized lip motion.

6. Captions follow the same turn and segment ids.

7. Keyboard events can interrupt, replay, or redirect the session.

That loop gives you a clean place to attach logging. Log turn boundaries, timestamps, interruptions, and final transcript text. For accessibility work, those logs are invaluable: they let you verify that a shortcut actually interrupted speech, that caption latency stayed within target, and that the avatar never rendered content after a turn was canceled.

Also budget for failure modes:

• network jitter causing temporary audio/video desync,

• captions lagging behind speech by a few hundred milliseconds,

• reconnects that preserve session identity but lose in-flight partials,

• browser autoplay policies that block audio until user interaction.

Your UI should degrade gracefully when any one channel drops. If video fails, keep audio and captions working. If audio is blocked, let text and captions continue. If the avatar stalls, keep the conversation operational.

Where Protoface fits

This is exactly the kind of session layer that Protoface is meant to sit inside. In practice, developers usually plug it into one of two surfaces: a LiveKit voice agent via the livekit-plugins-protoface plugin, or a direct session workflow through the REST API or Python SDK when they need to create and manage avatars programmatically. The useful part is that the avatar is tied to the same realtime session as the agent, so the talking face stays synchronized with the generated speech instead of being bolted on after the fact.

For a LiveKit-based voice agent, the integration is intentionally small. The shape is roughly:

# Illustrative example; exact configuration fields are in the docs

# Illustrative example; exact configuration fields are in the docs

# Illustrative example; exact configuration fields are in the docs

If you prefer direct API control, the REST surface is straightforward: create an avatar, start a session, and attach the session to your runtime. Authentication is via API key, so keep it server-side.

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"avt_123"}'

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"avt_123"}'

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"avt_123"}'

Exact field names, lifecycle details, and available quality tiers are documented in the docs. If you’re starting from a fresh voice-agent stack, the quickstarts linked from the project README are useful for seeing the wiring in a real app.

Deployment and security details that matter in production

Accessibility systems often end up embedded in customer-facing environments, so the deployment shape matters as much as the runtime design. A few rules keep things sane:

• Never expose API keys in the browser unless the surface is explicitly designed for it on the backend side.

• Bind session creation to your server when the client should not be allowed to mint arbitrary avatars or sessions.

• Set explicit rate limits for session duration, requests per IP, and per-user concurrency.

• Use origin allowlists for embedded experiences so the control surface only works on trusted sites.

For websites that need a fast integration path, a customer-managed iframe embed is the cleanest option because the browser never sees a secret. That is especially relevant when you’re adding an accessible assistant to a public page: the security model should be simple enough that frontend teams can deploy it without inventing a custom auth scheme.

Conclusion

If you treat voice, video, captions, and keyboard input as one realtime session with multiple synchronized streams, the architecture gets much simpler. The key decisions are to centralize turn state, keep captions independent from video rendering, give keyboard events real precedence, and design for partial failure instead of assuming perfect media delivery.

From there, pick the integration surface that matches your stack: LiveKit plugin for an existing voice agent, REST or Python SDK for programmatic session management, or an iframe embed when you need a secure browser-only deployment. If you want implementation details, examples, and exact request fields, start with docs.protoface.com.