Introduction

If you want an avatar that feels responsive in a live conversation, you’re really building three systems at once: low-latency audio transport, state synchronization, and video rendering that stays aligned with speech. In practice that means your browser UI needs a stable realtime channel for events, a media channel for audio/video, and a rendering pipeline that can keep lip motion, head movement, and transcripts in sync without introducing visible lag.

This post walks through a practical architecture for a realtime accessibility avatar in Remix using WebRTC for media and WebSocket for control. By the end, you should be able to structure a Remix app that:

• creates or joins a session from the browser,

• streams audio over WebRTC,

• uses WebSocket messages for session state and UI updates,

• handles reconnection and timing issues, and

• keeps the avatar usable for accessibility, not just visually interesting.

Architecture: separate media from control

The first thing to get right is the split between the media plane and the control plane.

WebRTC is the right tool for audio/video because it gives you low-latency, jitter-buffered, peer-to-peer media transport with codecs and congestion control built in. For an avatar, that’s where the microphone audio goes in and the synthesized video face comes back out. If you’re using a voice agent behind the avatar, WebRTC is also usually where you attach the agent’s audio stream.

WebSocket is the right tool for non-media events: session creation, readiness, transcript chunks, speaking state, errors, metadata, and UI coordination. Don’t try to move avatar video frames over a WebSocket unless you have a very specific reason; you’ll just rebuild a worse media stack.

In a Remix app, the server typically does two things:

1. mints or proxies session credentials, and

2. returns a lightweight bootstrap payload the browser can use to connect.

The browser then establishes the WebRTC peer connection and a WebSocket connection independently. That separation matters because you can lose one without necessarily losing the other. For example, the avatar media can continue while the websocket reconnects and resynchronizes UI state.

What the browser actually needs to do

A reasonable client flow looks like this:

1. User enters the page and requests an avatar session.

2. Remix loader/action returns session metadata and whatever token or ephemeral credentials your backend issues.

3. The browser opens a WebSocket to receive state updates.

4. The browser creates a WebRTC RTCPeerConnection, captures microphone audio, and negotiates an SDP offer/answer exchange.

5. When the remote audio/video tracks arrive, the UI attaches them to an <audio> or <video> element and updates accessibility state from WebSocket events.

The important part is to keep the avatar “speaking” state driven by events, not by guessing based on local playback. A reliable UI should reflect server-confirmed state: connected, listening, thinking, speaking, interrupted, errored, and disconnected.

Remix server route: create session metadata

In Remix, put the credential-minting logic on the server. The browser should never see a long-lived API key. If you need to create a realtime session from your app, your server can call the API and return only the minimum the client needs.

import { json } from "@remix-run/node";

}

import { json } from "@remix-run/node";

}

import { json } from "@remix-run/node";

}

That example is intentionally schematic. The exact request and response fields depend on the session type and avatar configuration, so use the docs as the source of truth. The point is the pattern: backend creates, browser connects.

WebSocket state machine: keep the UI honest

For accessibility, the UI should not only show a talking face; it should expose what the assistant is doing in text and state transitions. A screen-reader user needs the same conversational cues that a sighted user gets from animation.

A simple message taxonomy is enough for most apps:

• session.ready — avatar is connected and can receive input

• speech.start / speech.end — agent began or finished speaking

• transcript.partial / transcript.final — live captions

• error — recoverable or fatal transport/app error

The exact names don’t matter as much as the discipline: treat WebSocket messages as source-of-truth state transitions, and make reconnection idempotent. If the socket drops, reattach and resubscribe; don’t tear down the whole peer connection unless the media path actually failed.

Also pay attention to timing. If your agent streams audio and captions, a transcript event may arrive before the corresponding audio frame is audible, or after it. For accessibility, you generally want captions to appear as soon as they are stable enough to be useful, but not so early that you create visible churn. Partial transcripts can be announced visually; final transcripts can be sent to the live region for assistive tech.

WebRTC details that matter in practice

Three implementation details tend to cause the most pain:

1. Autoplay restrictions. Browsers often block audio playback until there’s a user gesture. If the avatar speaks automatically on page load, you may need an explicit “Start” button or a muted preview flow. Don’t build a design that only works on your dev machine.

2. ICE and NAT traversal. WebRTC is robust, but only if you let it do its job. Make sure your connection setup handles STUN/TURN as required by your environment. If you’re testing on a restrictive network, a peer connection that works locally may still fail in production.

3. Track lifecycles. Remote tracks can end, restart, or be replaced. Bind UI state to the peer connection and track events, not just to initial negotiation success.

For a realtime avatar, it is usually better to send one mono microphone track and receive one synthesized media track than to overcomplicate the media graph. Keep the browser-side media logic small and observable. If you need screen sharing, camera input, or multiple participants, add those deliberately rather than mixing them into the avatar connection path.

Accessibility considerations beyond the avatar itself

An “accessibility avatar” should do more than talk. It should help the conversation remain understandable for users who rely on assistive technology, lower bandwidth, or reduced visual attention.

That means:

• Provide live captions via WebSocket and render them as text, not as baked-in video overlays.

• Expose role and state in ARIA-friendly elements, not only in animation.

• Avoid motion dependence; the user should be able to follow the interaction if the video is hidden or paused.

• Handle interruptions cleanly, so a user can barge in or stop the agent without waiting for the current audio stream to drain.

In Remix, this usually means pairing the video element with an accessible transcript panel and a small status region that announces key changes. Keep the transcript incremental. Keep the status short. And make sure the avatar never becomes the only way the user understands the conversation.

Where Protoface fits

This is the part where Protoface becomes useful: it gives you a developer-facing avatar layer you can drop into a realtime voice stack without building the lip-sync and session orchestration yourself. If your app is already using a voice agent backend, the LiveKit Agents plugin is the most direct integration path; if you’re orchestrating sessions from your own server, the REST API and Python SDK let you manage avatars and realtime sessions programmatically. For browser-only embeds, the customer-managed iframe option keeps API keys out of the client entirely. The right surface depends on where your session logic already lives.

If you’re using the LiveKit path, the plugin is a small amount of glue rather than a new architecture. A typical setup is a voice agent that emits audio and a Protoface avatar that renders the synchronized face:

# illustrative only; see the plugin docs and examples for exact setup

)

# illustrative only; see the plugin docs and examples for exact setup

)

# illustrative only; see the plugin docs and examples for exact setup

)

For server-side session creation, the REST API looks like standard bearer-token authentication:

curl https://api.protoface.com/sessions \
-d '{"voice":"en-US","instructions":"Be concise."}'

curl https://api.protoface.com/sessions \
-d '{"voice":"en-US","instructions":"Be concise."}'

curl https://api.protoface.com/sessions \
-d '{"voice":"en-US","instructions":"Be concise."}'

And if you want to inspect the supported flows, the examples and setup notes in the docs are the fastest way to map your app onto the right integration model.

Putting it together in Remix

The cleanest Remix implementation is usually:

1. a route action that creates or fetches a short-lived session bootstrap payload,

2. a client component that opens WebSocket and WebRTC connections on mount,

3. an event reducer that updates avatar status, captions, and error UI, and

4. a teardown path that closes peer connections and sockets on navigation or logout.

Keep the session bootstrap server-side, keep media client-side, and make the UI state machine explicit. That separation will save you from the usual issues: leaked API keys, half-dead peer connections, confusing reconnect behavior, and avatars that keep animating after the underlying session has already failed.

Conclusion

If you build the avatar as “WebRTC for media, WebSocket for control, Remix for bootstrap,” you get a system that is relatively straightforward to reason about and much easier to debug in production. The key implementation choices are not exotic: protect credentials on the server, treat transport state as first-class UI state, and design for accessibility from the start with captions and clear status updates.

For implementation details, session fields, and integration examples, start with the docs at docs.protoface.com. If you prefer working from code, the quickstarts linked from the project repo are a good way to see the moving parts end to end.