Introduction

When you add a realtime AI avatar to a voice application, the hard part is not “making it talk.” The hard part is making it feel stable under real network conditions: low enough latency to track speech, clean enough audio to avoid feedback, and synchronized enough video to avoid the uncanny mismatch where the face trails the voice by half a second.

This post is about the practical audio side of that problem in React: how to wire microphone capture, noise suppression, and echo cancellation into a browser-based agent experience, and how to think about the trade-offs that determine whether the avatar feels like a conversation partner or a demo. By the end, you should have a mental model for the audio pipeline, know where browser processing helps and where it hurts, and understand how a realtime avatar service fits into the stack. If you want to compare implementation options, the Protoface docs are a good reference point for the API shapes and integration surfaces discussed here.

Start with the audio path, not the avatar

In most browser avatar apps, the video face is downstream of speech. If your audio is noisy, clipped, delayed, or echoing the assistant’s own output back into the microphone, the avatar will inherit those problems. So the first design decision is the audio path:

1. Capture microphone audio from the user.

2. Apply browser-side constraints for noise suppression and echo cancellation when available.

3. Send the cleaned audio to your voice agent or transcription pipeline.

4. Play the assistant audio through speakers or headphones.

5. Ensure the assistant’s playback does not leak back into the mic path.

In React, this usually means requesting media with explicit audio constraints rather than relying on defaults. Modern browsers expose built-in processing through getUserMedia(), and for many applications it is the simplest and lowest-latency approach.

const stream = await navigator.mediaDevices.getUserMedia({
});

const stream = await navigator.mediaDevices.getUserMedia({
});

const stream = await navigator.mediaDevices.getUserMedia({
});

Those flags are not magic. They delegate to the browser’s implementation, which varies by platform. Still, they are the right baseline for a browser avatar app because they solve the common case without adding another DSP stage in your app.

Noise suppression and echo cancellation: what they actually do

These features are often lumped together, but they solve different problems.

• Noise suppression tries to reduce steady or non-speech background noise: fan hum, HVAC, keyboard clicks, distant traffic, and so on.

• Echo cancellation tries to remove the assistant audio that the browser microphone picks up from local speakers.

For a conversational avatar, echo cancellation is usually the more important of the two. If your assistant speaks and the browser hears its own output, the model can transcribe itself, trigger premature turn-taking, or get trapped in a feedback loop. Noise suppression improves ASR stability, but over-aggressive suppression can also distort consonants and make the voice sound “watery,” especially with low-quality microphones.

The main trade-off is that browser processing is opinionated. If you need maximum fidelity for a professional microphone setup, or if you plan to do your own DSP server-side, you may prefer to capture raw audio and control the pipeline yourself. But for a React app embedded in a normal user environment, browser-provided AEC and noise suppression are usually worth enabling.

Implementing the capture layer in React

A clean implementation keeps microphone acquisition isolated from UI state. You want a hook or service that owns the media stream, exposes the current status, and tears everything down reliably when the component unmounts or the user leaves the session.

import { useEffect, useState } from "react";

}

import { useEffect, useState } from "react";

}

import { useEffect, useState } from "react";

}

A few practical notes:

• Ask for mic access only when you actually need it. Browsers are increasingly strict about permissions, and users are better served by explicit, user-initiated capture.

• Stop tracks when the call ends. Leaving the track open is a common source of stale device indicators and background capture bugs.

• Expect constraint behavior to vary. Some devices and browsers ignore or partially implement the audio processing flags.

If you are streaming audio into a realtime agent, it is better to keep the frontend simple and let the backend handle turn detection, transcription, and response timing. The frontend should capture clean audio, play remote audio reliably, and avoid introducing jitter.

Keep the avatar and audio in sync

Once the microphone path is sane, the next problem is timing. A realtime avatar is not just a video element; it is a synchronization problem. The avatar face must animate in lockstep with the assistant’s audio stream, and the browser must avoid unnecessary buffering that makes the face feel detached from the voice.

For a WebRTC-style pipeline, the key is to minimize the number of transformations between the agent output and the media element. Every extra hop — transcode, upload, transform, rebuffer — adds latency and makes lip sync harder. In practice:

• Prefer a realtime media transport over polling or long-lived HTTP fetches.

• Keep the assistant audio path stable and low-jitter.

• Do not run the avatar video through a separate asynchronous update loop if the service already gives you synchronized audio/video.

On the React side, that means treating the avatar surface like a media session, not a regular component. Mount it when the call starts, subscribe to session lifecycle events, and unmount on disconnect. If you try to re-render or recreate media objects on every state change, you will introduce visible glitches.

Don’t let the assistant hear itself

Echo cancellation only works if the browser has a coherent reference signal for what it played locally. That is why speaker selection and audio routing matter. If the assistant audio and the user microphone share the same device, cancellation is usually fine. If you route audio through virtual devices, output mixers, or browser tabs, the quality of cancellation becomes less predictable.

There are a few common failure modes:

1. Speaker playback at high volume causes residual echo even with cancellation enabled.

2. Multiple audio outputs make the echo path harder for the browser to model.

3. Concurrent media elements can create overlapping playback that the AEC layer was not tuned for.

For production apps, headphones are still the most reliable answer. For consumer deployments, you should assume some users will use speakers and design for graceful degradation: detect repeated self-interruption, shorten barge-in windows, and avoid immediate re-triggering of the same utterance if the user is obviously hearing the model playback.

Also remember that echo cancellation is only about the local acoustic loop. It does not solve server-side feedback if your backend mistakenly feeds assistant audio back into the input stream. That class of bug usually shows up as duplicate transcripts, self-referential prompts, or a model that appears to “interrupt itself.”

Where Protoface fits

Once the audio path is stable, the avatar layer becomes straightforward: your app sends or connects to the agent’s voice stream, and the avatar service handles synchronized talking-face video. For LiveKit-based agents, the most direct integration is the livekit-plugins-protoface package, which drops a talking face into an existing voice agent flow. The plugin repo and examples are a useful starting point if you already have a LiveKit stack and want the avatar to inherit the agent’s audio timing rather than building a custom video pipeline from scratch.

If you are using Python to create or manage avatars and sessions programmatically, the Python SDK gives you the same general workflow from backend code. The exact request/response fields are documented in the public docs, but the shape is simple: create or reference an avatar, start a realtime session, then connect your agent or media pipeline to that session.

from protoface import ProtofaceClient

)

from protoface import ProtofaceClient

)

from protoface import ProtofaceClient

)

For direct HTTP integration, the REST API is equally straightforward. Keep API keys on the server, never in the browser, and use the API to create sessions or manage avatars from your backend:

curl https://api.protoface.com/v1/sessions \
}'

curl https://api.protoface.com/v1/sessions \
}'

curl https://api.protoface.com/v1/sessions \
}'

If you want the browser-only deployment model for a website widget, the iframe embed route avoids exposing any API key in client code. That is a different integration pattern, but the same audio principles apply: clean capture, stable playback, and a lifecycle that avoids tearing down the media session unnecessarily.

Debugging checklist for real users, not lab machines

The first version of a realtime avatar demo works on a developer laptop and then falls apart in the wild. The usual culprits are mundane:

• Users deny mic permissions or switch devices mid-call.

• Mobile browsers apply different audio processing than desktop browsers.

• Playback starts before the user gesture that unlocks media on a given platform.

• Echo cancellation behaves differently on speakers, headphones, and virtual audio devices.

• State churn in React reinitializes streams and breaks the session.

Instrument the session from the start. Log when the mic is acquired, when audio tracks start and stop, when the assistant begins playback, and when the avatar session transitions. If your backend exposes session metadata, store enough to correlate user complaints with actual media events. You will save yourself a lot of guesswork.

Conclusion

Building a good realtime AI avatar in React is mostly an exercise in audio hygiene. Enable browser noise suppression and echo cancellation where they help, keep your capture and playback paths stable, and treat the avatar as synchronized media rather than a decorative component. The rest is integration: get audio into your agent reliably, keep latency low, and avoid self-feedback.

If you want to implement this with a real avatar backend, start with the public docs at docs.protoface.com, then choose the integration surface that matches your stack: the LiveKit plugin for voice agents, the Python SDK for backend orchestration, or the REST API for direct session management. The important part is to keep the browser side boring and predictable; that is what makes the avatar feel realtime.