Introduction

Building a conversational video agent for hearing-impaired users is not mainly a UI problem; it is a synchronization problem. If your system can transcribe speech accurately, render responses fast enough to feel conversational, and keep the visual avatar aligned with the agent’s actual turn-taking, you can make a face-to-face experience that is much easier to follow than audio alone. In practice, that means coordinating speech recognition, text generation, playback timing, and a live video surface in a way that does not drift under network jitter or model latency.

This post walks through the architecture I would use in React and TypeScript: how to structure the agent loop, how to keep the transcript usable as the primary modality, and how to render a talking avatar without turning your frontend into a stream-processing project. I’ll also show where a realtime avatar API fits cleanly when you want the agent’s face to stay synchronized with the conversation.

Design the experience around text first, avatar second

For hearing-impaired users, the avatar is not the information source; the transcript is. The video face should reinforce turn-taking, emotion, and attention, but the actual content needs to be available as readable text with stable timing.

A good mental model is:

• Transcript is authoritative. Every user utterance and agent response should be persisted as text, not inferred from audio playback.

• Avatar is a synchronized presentation layer. It should reflect who is speaking and when, but not drive state.

• Latency budget is user-visible. A half-second delay in transcript rendering is usually tolerable; a face that keeps moving after the agent has stopped speaking is not.

That leads to a UI pattern that looks more like live captions plus a speaker indicator than a classic chat widget. In React, I would keep a transcript timeline in state, render messages incrementally, and treat avatar activity as a separate realtime stream.

Core realtime loop in React and TypeScript

The implementation usually breaks into four stages:

1. Capture microphone audio in the browser or receive audio from a voice stack.

2. Stream audio to ASR and produce partial and final transcripts.

3. Generate the agent response, ideally as text first so captions can appear immediately.

4. Play the response through a synthetic voice and keep the avatar lip-synced to that playback.

For a hearing-impaired audience, step 2 and step 3 matter more than step 4. You want partial transcripts quickly, and you want the final response text to appear even if the audio layer is disabled or unavailable.

A practical React state model

Keep the UI state explicit. Don’t let the avatar component own conversation state, and don’t bury transcript events inside ad hoc refs. A simple reducer works well:

type Role = "user" | "assistant";

}

type Role = "user" | "assistant";

}

type Role = "user" | "assistant";

}

The important detail is that partial transcripts are first-class. Hearing-impaired users benefit from seeing the agent “think” in real time only if the partial state is visibly distinct from the final caption. Otherwise, you create churn and make the conversation harder to scan.

Don’t confuse lip-sync with accessibility

It is tempting to treat a talking avatar as the accessibility feature. It isn’t. Lip-sync makes the agent feel embodied, which can help with attention and turn-taking, but it does not substitute for readable text, clear contrast, keyboard navigation, or stable layout.

When I build this kind of interface, I usually do three things:

• Render the transcript in a fixed column with predictable line wrapping.

• Use the avatar as a secondary panel, not a floating overlay that obscures captions.

• Expose state changes such as “listening,” “processing,” and “speaking” through text labels, not only animation.

That separation matters because avatar playback and text rendering fail differently. If the video stream hiccups, the captions should keep going. If ASR slows down, the avatar should not keep animating as if it still has something to say.

Integrating a realtime avatar surface

At the point where the agent has a generated response, you need a video face that stays synchronized with speech timing. That is where a developer-facing avatar service is useful: instead of hand-rolling viseme timing or stitching together your own video pipeline, you hand the response to a service that returns a realtime session or embed, and you consume a live avatar stream in the browser.

For example, if you are using a managed session flow, the browser only needs to render the video surface and listen for caption events. The server can create the session, attach the agent instructions, and return the session metadata your frontend needs. The exact fields depend on the API shape in the docs, but the pattern is consistent: create session, receive session URL or token, mount it in the client, then keep conversation state separate from presentation state.

For direct API usage, this is the kind of interaction you would expect:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

If you want to drive the same flow from Python, the SDK keeps the orchestration on the backend where it belongs:

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

Use the REST API or Python SDK when you need server-side control, durable session creation, or integration with your own agent orchestration. Keep browser code focused on rendering, not secret management.

React rendering details that matter in production

A few frontend details are easy to miss until you ship:

• Autoplay policy: browsers may block audio playback until a user gesture. Plan for an explicit “Start conversation” action.

• Layout stability: reserve space for the avatar and caption regions so the page does not reflow when the stream initializes.

• Reconnection behavior: if the avatar stream drops, keep the transcript visible and retry in the background.

• Accessibility semantics: captions should be in the DOM as readable text, not only drawn into a canvas.

For the avatar component itself, I prefer a thin wrapper around the session surface rather than a bespoke video player. In React, that means a component that accepts a session URL or embed source, handles lifecycle cleanup, and emits status events back into your reducer.

Where the plugin approach fits

If your assistant already runs on LiveKit, the lowest-friction route is to add a video face at the agent layer rather than rebuilding the media pipeline in the browser. The LiveKit plugin pipecat-protoface and the related examples in the plugin repository show the pattern: the voice agent produces speech, and the avatar is attached as a synchronized rendering surface. That keeps timing aligned with the same agent event loop that already governs turn-taking.

This is useful when the backend already owns the conversation loop. Your React app can then subscribe to transcripts and state updates, while the avatar stays synchronized through the agent stack instead of through a separate frontend timing heuristic.

Operational concerns: rate limits, privacy, and failure modes

For a product aimed at hearing-impaired users, reliability and privacy are not optional extras. If you expose an avatar via an iframe embed, keep the browser free of API keys and restrict the embed origin. If you use server-side API access, treat keys like any other production secret and rotate them regularly. In either case, make sure your UI degrades gracefully when the avatar is unavailable.

In practice, I would implement three fallback paths:

• Primary: transcript + synchronized avatar.

• Secondary: transcript only, no avatar stream.

• Tertiary: static agent state with a reconnect prompt and preserved conversation history.

This is especially important because media streams fail in non-obvious ways: a WebRTC connection can be nominally “connected” while the rendered track stalls, and an ASR service can continue delivering partial hypotheses after the user has already stopped speaking. The UI should reflect the actual state of each subsystem, not assume success.

Conclusion

The key design choice for a conversational video agent aimed at hearing-impaired users is to make text the source of truth and the avatar a synchronized layer on top. In React and TypeScript, that means explicit transcript state, separate speaker state, stable layout, and a media surface that can fail independently without breaking the conversation.

If you are building the backend yourself, use the REST API or Python SDK to create sessions and attach your agent logic; if you already have a LiveKit voice agent, the plugin route is a clean way to add a face without rebuilding your stack. The docs at docs.protoface.com cover the exact request shapes, session fields, and integration details.

For a practical next step, wire up a transcript-first React UI, then connect a realtime avatar session behind it and verify that captions remain readable even when the video layer is delayed or disconnected. That test tells you whether the experience is actually usable, not just visually polished.