Flask Guide: Streaming Lip-Synced AI Avatars for Screen Reader Users
Build accessible streaming lip-synced AI avatars in Flask: sync avatar motion with TTS, handle latency, and support screen readers.
Introduction
Screen reader users can consume a lot of UI text efficiently, but most “AI avatar” experiences still assume a sighted user watching a video widget. That leaves an obvious gap: if your product has a voice agent, the avatar should be more than decoration. It should stay synchronized with the agent’s speech, expose enough state to avoid confusing animations, and degrade cleanly when the network or browser can’t keep up.
This post walks through the mechanics of streaming lip-synced avatars for voice-first applications, with a focus on accessibility. By the end, you should understand the moving parts of a realtime avatar pipeline, the latency and synchronization trade-offs, and how to integrate one into a voice agent without making the experience worse for users of assistive tech.
What “streaming lip-sync” actually means
A realtime avatar is usually not a single video file. It is a session-based media stream that updates continuously as the agent speaks. The important constraint is synchronization: the face animation has to track the audio timeline closely enough that mouth motion, pauses, emphasis, and turn boundaries feel plausible.
In practice, that means you are coordinating at least four subsystems:
ASR for detecting user speech and turn boundaries.
LLM / agent logic for generating the response.
TTS for producing the audio track the user actually hears.
Avatar rendering / lip-sync for turning the speech signal into a visual stream.
The avatar stream is usually driven by the same text or audio that drives TTS. If the avatar runs ahead of the audio, the user sees mouth motion with no sound. If it lags behind, it looks detached. The goal is to keep all three outputs — audio, animation, and conversational state — inside a latency budget that feels “live.” For most voice agents, that budget is tighter than developers expect: once you add ASR, model inference, TTS, and media transport, every extra hop matters.
Accessibility first: don’t let the face become the interface
If you are building for screen reader users, the avatar should never be the only way to understand the interaction. That sounds obvious, but it is easy to violate accidentally when the visual layer becomes the default “agent presence.”
Design the experience so the avatar is supplementary. The accessible channel is the conversation itself: spoken audio, clear transcript access, and explicit status signals. A screen reader user needs to know:
when the agent is listening, thinking, or speaking;
when the turn has changed;
whether the session is reconnecting or degraded;
how to stop, repeat, or interrupt output.
That means the avatar component should be treated like a synchronized media surface, not like the source of truth. Use semantic UI for status, captions, and controls. If you embed an avatar in the page, keep it in a region that is easy to skip, and avoid autoplaying visual changes that don’t correspond to actual conversational state.
From an implementation standpoint, the easiest way to avoid accessibility regressions is to separate concerns:
The agent produces text/audio and conversation state.
The avatar consumes the speech stream and renders motion.
The accessible UI announces state changes independently.
That separation also helps when the avatar is unavailable. A screen reader user should still be able to finish the task.
Latency, buffering, and turn-taking
Most “bad avatar” bugs are actually timing bugs. The visual layer exposes every mismatch between speech timing and conversation state. A few things are worth watching:
1. Start speaking only when the audio pipeline is ready. If the avatar begins animating on text generation before TTS has started streaming audio, the mismatch is noticeable. Prefer speech-driven synchronization over text-driven animation when possible.
2. Keep buffering small but nonzero. Zero-buffer streaming sounds ideal, but it increases jitter. A small buffer smooths packet variation and helps the mouth motion stay stable. The trick is to keep this buffer short enough that the user still perceives the response as realtime.
3. Handle interruptions explicitly. In voice agents, users interrupt. When they do, you need to stop or fade out the current speech and reset the avatar state immediately. If the avatar keeps “talking” after the agent has yielded the floor, it feels broken.
4. Do not over-animate idle states. Idle motion matters, but excessive blinking, head movement, or random gestures become distracting for screen reader users who are relying on stable auditory feedback. Keep idle animation subtle and state-driven.
5. Reconnection must be visible and audible. WebRTC sessions can reconnect. If the media stream drops, the UI should expose that state clearly rather than leaving a frozen face on screen.
A minimal integration pattern
If you already have a voice agent, the cleanest integration is to attach the avatar at the media layer and let the existing agent stack keep ownership of conversation logic. For Python-based agents, the LiveKit plugin path is straightforward because the avatar becomes another synchronized media participant in the session.
Conceptually, your agent still does the normal work: receive user audio, stream ASR, generate the answer, and stream TTS. The avatar plugin subscribes to the speech output and emits the visual stream in lockstep.
The important implementation detail is not the exact constructor shape; it is that the avatar should be downstream of the speech generator, not a separate timer-based animation loop. That keeps the visual output tied to the actual audio stream. If your stack is not LiveKit-based, the same principle still applies: the avatar should consume the same speech events or audio frames that the user hears.
For a concrete starting point, the plugin repository and examples are the fastest way to understand the data flow: GitHub and the public docs at docs.protoface.com.
How Protoface fits in
Protoface gives developers a managed realtime avatar layer so you do not have to build the media orchestration, session management, or rendering plumbing from scratch. The relevant surface here is the LiveKit Agents plugin, which drops a synchronized talking face into an existing voice agent, and the broader session API for creating and managing avatar sessions when you need direct control.
In practical terms, this is useful when you want to:
give a voice agent a face without rebuilding your media pipeline;
keep the avatar synchronized with streamed speech rather than post-processing a recorded clip;
avoid exposing backend credentials to the browser if you embed the experience in a web app.
If you want to prototype quickly, the plugin route is the shortest path. If you need full lifecycle control — avatar creation, session management, usage visibility — the REST API and Python SDK let you wire it into your existing backend. For browser-only experiences, customer-managed iframe embeds are the safer option because the API key stays off the client entirely.
Here is a representative REST call to create a session. Field names vary by endpoint, so treat this as illustrative and check the docs for the exact payload:
And a simple Python SDK pattern for backend orchestration:
The broader point is that the avatar should be managed like any other realtime dependency: created server-side, scoped to a session, and tied to the lifecycle of the conversation. That keeps your auth story sane and makes it easier to enforce rate limits, time limits, and usage visibility.
Practical gotchas for screen reader-friendly deployments
A few mistakes come up repeatedly when teams ship voice agents with avatars:
Do not hide critical state in the video. “The avatar looks paused” is not a useful status signal. Expose listening, speaking, thinking, reconnecting, and error states in text.
Do not rely on autoplay alone. Browsers can block or gate autoplay in ways that complicate first interaction. Design for an explicit user action to start the session.
Do not assume the avatar is cheap. Realtime video has bandwidth and compute cost. If you bill by quality tier, choose the lowest tier that still meets your UX requirement. Higher visual fidelity is not automatically better for accessibility.
Do not ship a decorative avatar with no fallback. If the media stream fails, the conversation still needs to work. A transcript-only or audio-only fallback is acceptable; a dead video box is not.
Do not overfit to sighted QA. Test the interaction with a screen reader and a keyboard. Make sure the user can start, interrupt, repeat, and exit without chasing hidden controls.
Conclusion
Streaming lip-synced avatars are useful when they reinforce a voice experience, but they only work well if they stay tightly synchronized with the actual speech pipeline and never become the only way to understand the agent. For screen reader users, the accessible path is the conversation state, audio, and transcript; the avatar is a supplemental media layer.
If you are adding this to an existing voice agent, start with a realtime integration that keeps the avatar downstream of TTS and turn-state changes, then add explicit accessible status and fallback behavior. If you want implementation details, examples, or the current API shapes, start with the docs and the relevant quickstart repositories on GitHub.