How to Add Real-Time Captions, Lip-Sync, and Screen Reader Support to an AI Avatar
Build AI avatars with synced captions, lip-sync, and screen reader support using shared timelines, DOM text, and WebRTC handling.
Introduction
Realtime avatars are easy to demo and surprisingly hard to ship well. The hard part is not rendering a talking face; it is keeping speech, mouth motion, captions, and accessibility signals aligned under real network and model latency. If the transcript lags the audio by even a second, captions become unusable. If the lip motion drifts, the avatar feels broken. If the avatar has no screen-reader path, the experience is visually polished but functionally inaccessible.
This post breaks down the engineering pieces you need to add to an AI avatar so it can support three things at once: low-latency speech, synchronized lip motion, and usable captions/screen reader output. By the end, you should have a practical mental model for how these systems fit together, what to watch for in a realtime pipeline, and where a service like Protoface fits if you want to avoid building the avatar layer from scratch.
1. Treat speech, visuals, and text as separate realtime streams
The first design choice is conceptual: do not think of “the avatar” as one blob of output. It is usually three coordinated streams:
Audio: the TTS or agent voice, ideally delivered in chunks as soon as they are generated.
Video: the face animation or rendered frames that need to follow the speech envelope closely enough to look believable.
Text: the transcript or caption stream, which can be exposed to the browser, assistive technology, or both.
Those streams are related, but they do not have identical latency requirements. Video can tolerate a small amount of buffer if it preserves smoothness. Captions should appear quickly, even if they are updated incrementally. Audio is the source of truth for what the user hears, and everything else should be synchronized against it.
In practice, the agent pipeline usually looks like this:
The user speaks or types.
The agent generates text response chunks.
TTS emits audio chunks from the response.
The avatar renderer drives lip motion from the same speech timing.
Caption events are emitted from the same transcript stream, not from a separate “best effort” OCR or audio transcription pass.
The key point is that captions and lip-sync should be derived from the same underlying utterance metadata whenever possible. If you generate captions from a different service than the voice, you introduce another timing domain and another failure mode.
2. Build for alignment, not just low latency
Developers often optimize for “fast” and then discover that fast but unsynchronized output is worse than slightly slower output that stays aligned. In a realtime avatar, the important metric is perceptual coherence.
Use a shared timeline
Even if your stack includes separate services for LLM, TTS, and video generation, you want a single playback clock. This can be as simple as a monotonically increasing session timestamp attached to each emitted event. Every caption chunk, audio chunk, and animation cue should be labeled relative to that timeline.
This lets the browser or client do two important things:
Buffer slightly when necessary to keep mouth motion and audio in phase.
Render captions immediately while still associating them with the correct utterance.
Handle partials explicitly
Captions are most useful when they start early and update incrementally. That means you need to support partial transcript events, not only finalized ones. Partial text should be visually distinct from final text so users can tell what may still change.
A practical pattern is:
partial: transient text, often italicized or lightly styled
final: committed text, stable in the caption history
For screen readers, partial updates should be rate-limited. If you push every token as a live region announcement, you will create a firehose. A better pattern is to announce finalized sentence fragments or clause-sized chunks.
Keep lip-sync deterministic enough
Perfect viseme-level lip-sync is expensive and often unnecessary for conversational agents. What matters is that mouth opening, closure, and broad articulation cues track the audio closely enough that the viewer does not notice drift. In many systems, the avatar renderer uses speech timing metadata rather than phoneme-perfect animation.
Common failure modes:
Audio leads video: the face appears late and the agent feels sluggish.
Video leads audio: the face “speaks” before the sound arrives.
Buffer underruns: jitter causes visible resets in the mouth shape.
The mitigation is usually to make the avatar consumer of the audio timeline, not an independent predictor. If the renderer knows when a phonetic segment begins and ends, it can interpolate smoothly even when packets arrive unevenly.
3. Make captions and screen reader support first-class outputs
Accessibility is not an overlay you add after the avatar works. For a conversational avatar, captions and screen reader support are part of the product surface.
Captions in the DOM
If the avatar lives in a web app, render captions as real text in the DOM rather than burned into the video. That gives you:
Selectable and searchable text
Native browser accessibility tree support
Control over styling, contrast, and responsive layout
Keep the caption container visually separate from the video element. The avatar can be a canvas or video stream; the captions should be semantic HTML.
Use aria-live="polite" for finalized or low-frequency updates. If you use assertive too aggressively, you will interrupt the user and any concurrent assistive announcements. In most conversational apps, “polite” is the safer default.
Screen reader strategy
There are two common approaches:
Announce finalized utterances: best for short responses and low-noise interaction.
Expose a transcript log: best when users need to review the conversation history.
For most agents, I recommend both. Use a live region for the current utterance, and keep a scrollback log of finalized lines. This avoids losing context when the conversation gets long.
One subtle but important point: do not put the actual video frame in the accessibility tree unless it has meaningful alternate text. The face itself is usually decorative from an accessibility standpoint; the spoken content is what matters. If the avatar conveys state that is not spoken, then the state must also be exposed textually.
4. WebRTC and browser delivery: manage jitter, buffering, and recovery
If your avatar is delivered over WebRTC or a similar realtime media path, the browser is responsible for a lot of the user experience. Network jitter is normal. Small drift is normal. Disconnects happen. Your implementation should assume this.
Practical rules for the client
Buffer a little, not a lot: enough to smooth packet variation, not enough to make interaction feel delayed.
Reconcile on utterance boundaries: if the stream stutters, resync at the next sentence or segment rather than trying to “fix” every frame.
Make reconnection idempotent: when the session resumes, do not duplicate caption lines or replay already-announced text.
For avatars, recovery matters more than perfect continuity. If the network drops for a few seconds and the session comes back cleanly, users will forgive it. If the mouth keeps moving with stale audio or captions repeat, the experience feels unreliable.
This is also where rate limiting and session management become relevant. Customer-facing avatar products tend to be abused in the same ways as any other realtime endpoint: automated retries, tab storms, and long-lived idle sessions. A session model with explicit duration limits and per-IP controls is a good default, especially when an avatar is exposed in the browser.
Protoface in a LiveKit voice agent
When you already have a LiveKit voice agent, the fastest path is usually to drop in a video face as another synchronized media surface. The LiveKit plugin from the Protoface ecosystem does exactly that: it lets your agent keep its existing speech pipeline while adding a talking avatar that tracks the voice session. The integration is useful because it keeps the avatar bound to the same realtime conversation rather than treating it as a separate UI component.
A minimal setup looks like this at a high level:
If you are building outside LiveKit, the REST API and Python SDK are the cleaner fit. The REST API is the right surface when you need to create sessions from a backend, manage avatars centrally, or integrate with your own auth. The Python SDK is better when you want the avatar/session lifecycle in application code without hand-rolling HTTP requests. The exact request schema is documented in the public docs, but the pattern is standard: authenticate with an API key, create or reference an avatar, then start a realtime session and stream the output into your client.
For readers evaluating the integration surface, the public documentation at docs.protoface.com is the place to check the exact fields, session lifecycle, and examples. The important architectural idea is the same regardless of the API shape: keep captions, audio, and avatar animation attached to one session timeline.
Gotchas that usually show up late
There are a few issues that tend to appear only after you have a working demo:
Token streaming without utterance boundaries: the UI looks alive, but captions become a noisy stream instead of meaningful sentences.
Different clocks for audio and captions: text appears “early” or “late” relative to the voice.
Over-eager screen reader announcements: the experience becomes unusable for assistive tech.
Network retries that replay state: users hear duplicated speech or see duplicate subtitles.
Invisible failures: the avatar is still running, but captions or accessibility metadata stop updating.
The fix for most of these is to treat accessibility and sync as observable state, not side effects. Log utterance IDs, segment timestamps, finalization status, and client acknowledgements. If captions disappear, you should be able to tell whether the issue was in generation, transport, or rendering.
Conclusion
A good AI avatar is not just a video face glued onto an agent. It is a coordinated realtime system where voice, animation, captions, and accessibility all move on the same conversation timeline. If you design for shared timing, partial transcript handling, DOM-based captions, and sane recovery behavior, the experience stays usable under real-world latency instead of just working in a demo.
If you want to inspect the API shapes or wire this into an existing agent, start with the public docs and the quickstart examples in the Protoface ecosystem. The shortest path is usually: integrate the avatar surface into your current voice stack, expose captions as semantic text, and verify synchronization under actual network conditions before you ship.