Building a Realtime Avatar for Accessibility in Remix with FastAPI, STT, and TTS
Build an accessible realtime avatar in Remix and FastAPI with streaming STT, TTS, session orchestration, and lip-sync.
Introduction
If you’re building an accessibility-focused assistant, the hard part is usually not speech recognition or text generation in isolation. It’s the orchestration: capturing audio reliably, turning it into text with low enough latency to feel conversational, synthesizing a response fast enough to avoid awkward gaps, and presenting that response in a way that users can follow.
This post shows a practical way to wire together Remix on the frontend, FastAPI on the backend, streaming speech-to-text (STT), and text-to-speech (TTS) so you can ship a realtime avatar that supports spoken interaction and is usable for accessibility-sensitive flows. By the end, you should understand the architecture, the latency trade-offs, and where a realtime avatar fits in without turning your app into a science project.
Start with the interaction model, not the avatar
For accessibility, the avatar is not the product; it is the presentation layer for a conversational system. The core loop looks like this:
The browser captures microphone input.
Audio streams to your backend or agent runtime.
STT produces partial and final transcripts.
Your agent logic decides what to say next.
TTS streams audio back to the browser.
The avatar lip-syncs to that audio while the user sees a speaking face.
That last step matters because visual feedback reduces uncertainty for many users. If the system is speaking, the user should know who is speaking, whether it is still listening, and whether the system is processing. A realtime avatar can carry those cues, but only if the surrounding audio pipeline is responsive.
Architecture in Remix and FastAPI
A clean split is:
Remix: UI, recording controls, transcript display, accessibility state, and the avatar surface.
FastAPI: session creation, auth, agent orchestration, and integration with STT/TTS providers.
Realtime transport: WebRTC or a managed streaming layer for audio and video.
Why split it this way? Remix is good at rendering state and handling user interactions, but speech pipelines need long-lived connections, backpressure handling, and a place to keep API keys off the client. FastAPI gives you a straightforward async runtime for that backend work.
A minimal backend endpoint might create a realtime session and return only the fields the browser needs:
On the Remix side, you typically request that session, then initialize the client-side transport. Keep the browser responsible for presentation, not secrets.
STT: optimize for partials, not just finals
For accessibility, partial transcripts are often more important than final accuracy. A user benefits from seeing the system “hear” them while they are still speaking, especially if the UI also exposes input state such as listening, thinking, and speaking.
Two practical points matter here:
Endpointing: you need a good balance between waiting for the user to finish and responding promptly. Too aggressive, and you interrupt mid-sentence. Too conservative, and the app feels sluggish.
Incremental updates: show interim text in the UI, but distinguish it from final text. Don’t present partials as committed output.
With FastAPI, you can keep the transcription path asynchronous and push transcript events back to Remix via WebSocket or your realtime transport. The UI should treat those events as a stream of state transitions rather than discrete form submissions.
TTS: stream audio as soon as you have enough text
TTS quality matters, but latency dominates perceived quality in conversational systems. If you wait for a full paragraph before synthesizing, the interaction will feel detached. For an accessibility assistant, users generally prefer prompt, slightly segmented speech over perfect long-form cadence.
The standard pattern is:
Generate text incrementally from your agent.
Chunk it into natural phrases.
Start TTS on the first stable chunk.
Stream the resulting audio to the client as it becomes available.
That lets the avatar start moving its mouth quickly, which is critical for maintaining conversational turn-taking. It also gives you a place to insert nonverbal cues like “thinking” or “listening” states when the model is still generating.
One implementation detail developers often miss: if your STT is fast but your TTS is slow, the user will experience a visual delay even if your transcript appears instantly. The fix is not usually a bigger model; it is better pipelining and smaller audio chunks.
Client-side state in Remix
In the UI, don’t model this as a single “chat response” event. Use explicit state so assistive behavior is predictable:
idle: no active conversation
listening: microphone active, input being captured
processing: STT finalizing and/or agent reasoning
speaking: TTS audio playing and avatar animating
error: permission denied, reconnect, or provider failure
That makes it easier to build an interface that works with screen readers and keyboard navigation. The avatar should be decorative from an accessibility tree perspective unless it is conveying essential state. Actual status text should be rendered separately and announced carefully.
Here is a simple client-side fetch from Remix to your FastAPI session endpoint:
Where the avatar fits: use a managed realtime surface
This is the part that is usually more work than it looks. A talking face is not just a video element; it is a synchronized stream that needs to stay aligned with the speech audio. If you build that yourself, you end up handling session lifecycle, auth, transport reliability, and lip-sync coordination.
That is where Protoface fits well. In practice, you use one of the developer surfaces depending on your stack: the REST API for creating and managing avatars and sessions, or a plugin if you are already running a voice agent framework. For a LiveKit-based agent, the plugin path is straightforward: drop the avatar into the agent pipeline so the voice agent gains a synchronized talking face without exposing secrets in the browser.
If you prefer a lower-level integration, the REST API is useful for server-side session creation:
The key architectural point is that the browser should not need your long-lived API key. Create ephemeral session credentials on the backend, hand those to the client, and keep the authoritative control plane server-side. If you are working in Python, the SDK is the same idea with less manual request plumbing; see the Python SDK repo and the docs for the exact methods and payloads.
Accessibility and operational gotchas
A few things are worth getting right early:
Don’t rely on the avatar alone for comprehension. Provide text transcripts and clear status labels. Some users will disable motion or audio.
Respect reduced-motion settings. If the avatar is purely decorative in your flow, consider a low-motion fallback.
Handle microphone permissions explicitly. Permission failures should route to a keyboard-accessible fallback.
Stream defensively. Network hiccups happen; design reconnection and resumption paths for both audio and video.
Separate user-facing state from provider state. A TTS timeout should not leave the UI stuck in “listening.”
Also be deliberate about rate limits and session lifetime. For accessibility-focused apps, short-lived sessions with clear re-entry points are easier to reason about than one giant always-on connection.
Conclusion
If you treat the avatar as the visible edge of a realtime voice pipeline, the implementation becomes much more manageable. Remix can own the user experience, FastAPI can own session orchestration and provider integration, and STT/TTS can stay streaming and incremental so the interaction feels responsive. The avatar layer then becomes a synchronization problem, not a custom video product.
For exact API shapes, session fields, and integration details, start with the documentation. If you want to see the developer surfaces in context, the plugin and SDK repos are the fastest way to move from architecture to a working prototype.