How to Integrate ElevenLabs TTS with a React Realtime Avatar App
Learn how to integrate ElevenLabs TTS into a React realtime avatar app with backend streaming, lip sync, and session sync.
Introduction
If you already have a React app that renders a realtime avatar, adding ElevenLabs for text-to-speech is mostly a systems problem: you need to generate audio with low latency, stream or hand off that audio to the avatar pipeline, and keep lip sync aligned with what the user hears. The “hard part” is not producing speech; it’s managing timing, buffering, and state so the face and the voice stay synchronized under network jitter and backend latency.
In practice, the architecture usually looks like this: a React client captures user input, your backend turns text into audio through ElevenLabs, and the avatar layer consumes that audio or the downstream conversation events. By the end of this post, you should be able to wire up the TTS side cleanly, choose the right integration pattern for your app, and avoid the common mistakes that break realtime feel: blocking the UI, waiting for full audio files, or letting the avatar and audio drift apart.
What “realtime” means in this setup
For a conversational avatar, “realtime” does not mean every subsystem is hard real-time. It means the end-to-end interaction feels immediate enough that the user perceives the avatar as responding live. In a voice-driven app, there are three timing domains that matter:
Text generation: your agent decides what to say.
Audio synthesis: ElevenLabs turns that text into speech, ideally with streaming delivery or short chunk latency.
Avatar rendering: the video face plays mouth movements and expression in sync with the spoken audio.
The main mistake developers make is treating TTS as an offline step. If you wait for the full audio file before doing anything, the user sees a dead face for too long. A better design streams or incrementally prepares speech, then begins playback as soon as the first usable audio arrives. If your avatar layer is consuming the same audio clock as the player, lip sync stays much tighter.
Recommended integration pattern in a React app
In a React frontend, keep the browser focused on UI and media playback, not on secret management or speech synthesis. Put the ElevenLabs API call behind your backend, then return either a signed media URL, a stream, or the metadata your avatar/session layer needs.
A typical flow is:
User submits a prompt in React.
Your backend sends text to ElevenLabs TTS.
Your backend either streams audio to the client or stores it briefly and returns a playable URL.
The client starts playback and feeds the same utterance boundary to the avatar layer.
If your avatar system can consume the synthesized audio directly, that is usually preferable. You avoid clock drift caused by separate audio playback and animation timers. If not, make sure the audio element is the source of truth and the avatar is driven off the same playback lifecycle events.
Backend-first TTS call from React
Do not call ElevenLabs directly from the browser with a long-lived secret. Use a backend endpoint that accepts text, forwards it to TTS, and returns audio data or a short-lived reference. In React, this keeps the client simple:
That snippet is intentionally generic. The exact ElevenLabs request shape depends on the voice, model, and streaming mode you choose. The important part is the contract: the frontend receives something it can play immediately, and the avatar reacts to the same utterance.
Keeping lip sync aligned with speech
Once you introduce an avatar, synchronization is the real implementation detail. There are two common approaches:
Audio-driven sync: the avatar animates based on the audio playback signal. This is simpler and usually good enough for TTS.
Phoneme- or event-driven sync: the synthesis layer emits timing information that can be mapped to visemes or mouth cues. This is more precise, but more work.
For most React apps, audio-driven sync is the practical choice. It is resilient and easy to reason about: once playback starts, the face moves with the audio. The main thing you must avoid is starting the avatar animation before audio is actually flowing, or restarting audio without resetting the animation state.
Pay attention to these edge cases:
Partial failure: audio starts but the avatar session fails, or vice versa. Handle each independently and surface a clear error state.
Buffering: if the browser waits on media, show a “connecting” state rather than animating a talking face with no sound.
Queued utterances: if the user triggers multiple responses, serialize them. Overlapping TTS responses usually produce bad sync and confusing UI.
Cancellation: if the conversation context changes, stop the current utterance cleanly so the next one can start without stale animation.
Minimal backend example for ElevenLabs audio generation
On the server side, the exact implementation depends on your stack. The key is to keep the API key server-only and return audio in a format the client can consume. Here is a simple Python example of the shape you want:
If you prefer streaming, return chunks as they arrive rather than waiting for the full file. That reduces perceived latency and gives the avatar a faster start time. Just make sure your playback path can handle streamed media correctly; otherwise you trade one latency problem for a buffering problem.
Where Protoface fits
This is the point where Protoface is useful: instead of stitching your own face renderer, sync logic, and session orchestration together from scratch, you can attach a realtime avatar surface to the same voice workflow you already have. For developers using LiveKit-based voice agents, the ElevenLabs agents quickstart is a good reference for the overall pattern, and the docs at docs.protoface.com cover the exact session and avatar APIs.
For example, if your app already has a backend voice agent that produces spoken responses, the integration point is often just “take the agent’s audio output and bind it to an avatar session.” That keeps your React client thin: it renders UI, opens the media connection, and displays the avatar stream. The avatar stays synchronized with the generated speech because they are part of the same session lifecycle instead of separate browser timers.
If you are using a LiveKit agent stack, the Protoface plugin is the most direct path. The plugin makes the avatar appear as part of the agent pipeline, so the agent can speak and display a synchronized face without you manually managing a second video subsystem. If you are building your own backend orchestration instead, the REST API is the right surface for creating and managing avatars and realtime sessions.
Practical React implementation notes
A few implementation details matter more than the API choice:
Keep the session state machine explicit: idle, synthesizing, buffering, playing, interrupted, failed.
Make playback idempotent: if the user double-clicks a send button, deduplicate or queue the request.
Clean up object URLs: if you create blob URLs for audio, revoke them after playback.
Use a single source of truth: if the audio element is playing, the avatar should reflect that state; do not animate independently.
Hide backend latency: optimistic UI helps, but only if the avatar state matches reality. Showing “thinking” is better than showing a talking face before audio is ready.
If you are already using a voice agent framework, the fastest path is usually to keep the agent logic where it is and add avatar synchronization at the session boundary. That avoids duplicating text generation or speech orchestration in the browser.
Example REST flow for session creation
If you want to create and manage avatar sessions from your backend, the REST API is the natural fit. The exact fields depend on the session model in the docs, but the pattern looks like this:
Use this kind of call server-side only. Your React app should receive a session token or a short-lived embed/session reference, not a long-lived API key. That keeps the browser attack surface small and lets you rotate credentials without touching frontend code.
Common trade-offs
There is no single best approach for every app:
Direct client playback is simple, but harder to secure and harder to keep synchronized at scale.
Backend-generated audio files are easy to reason about, but add latency unless you stream them.
Streaming synthesis improves responsiveness, but makes error handling and cleanup more important.
Embedded avatar sessions reduce frontend complexity, but constrain how much you customize the media pipeline.
For most production React apps, I would start with a backend-mediated TTS flow, keep the browser state machine explicit, and only optimize further once you have measured where the latency actually lives.
Conclusion
Integrating ElevenLabs TTS into a React realtime avatar app is mostly about getting the media contract right: generate speech server-side, start playback as early as possible, and keep the avatar session aligned with the audio lifecycle. If you preserve that synchronization boundary, the rest of the UI is straightforward.
If you want to skip a lot of the avatar/session plumbing, start with the Protoface docs and the relevant quickstart, then map your existing voice agent or React playback flow onto that session model. The result is a cleaner architecture: React handles interaction, your backend handles TTS and session control, and the avatar stays synchronized with the voice instead of drifting behind it.