Introduction

If you are building a talking avatar for a voice agent, the hard part is not “playing video while audio is present.” The hard part is keeping mouth motion aligned with phonemes under real network and compute constraints: audio arrives in chunks, inference runs asynchronously, WebRTC jitter exists, and the browser’s rendering loop is not synchronized to the model’s output cadence.

This post focuses on the engineering side of frame-accurate lip-sync in a React app: how to think about timing, what actually causes drift, and how to structure a client so the avatar feels stable instead of “mostly aligned.” By the end, you should be able to reason about the pipeline, spot the common failure modes, and wire a talking face into a React UI without fighting playback glitches every time latency changes.

What “frame-accurate” actually means

For avatars, “lip-sync” is often used loosely. In practice, you want three things:

• Temporal alignment: mouth shapes line up with the audio that the user hears.

• Frame stability: the rendered face advances smoothly at the display’s refresh cadence, without random jumps or duplicated frames.

• Latency tolerance: the system stays believable even when audio starts late, the network jitters, or the model produces output in bursts.

The important distinction is that lip-sync is not just a visual effect; it is a synchronization problem. Audio is the source of truth for timing, but the browser is responsible for rendering, and those two clocks are never perfectly identical. In React, you also have a third timing domain: component lifecycle and state updates.

The practical goal is not “every mouth frame corresponds exactly to every audio sample.” The goal is to map audio time to animation time consistently, then keep that mapping stable across buffering, tab throttling, and rerenders.

Build around a single timeline

The cleanest implementation is to treat the audio playback clock as canonical and derive avatar frames from it. That means:

1. Start audio and avatar output from the same session boundary.

2. Use timestamps, not arrival order, to place animation frames.

3. Render the current mouth pose based on playback position, not wall clock time.

This matters because audio transport is usually packetized. You might receive a burst of predicted visemes, a sequence of frames, or a video stream whose decoded frame times are only known once buffered. If you simply animate on “frame received,” mouth motion will drift whenever the network slows down or the decoder catches up.

In a browser client, the implementation usually looks like this:

• Maintain a session clock in milliseconds.

• When an audio segment starts, store its start timestamp relative to that clock.

• Associate each viseme or avatar frame with a time offset.

• On each animation tick, compute the active frame from the current playback time.

React should not own the timing loop. It can own state and view composition, but the animation scheduler should live outside normal rerender paths, typically in a small controller object or a hook with refs. If you push every timing update through React state, you will create unnecessary rerenders and add jitter exactly where you do not want it.

Dealing with drift, buffering, and dropped frames

There are a few failure modes you will run into quickly:

Audio/video drift

Even when generated together, audio and visual output can diverge if one path buffers more than the other. The fix is to measure offset continuously and make small corrections rather than large jumps. If the avatar lags a bit behind the audio, advance the displayed pose slightly faster for a short interval; if it runs ahead, hold a frame briefly. Avoid hard resets unless the offset becomes visibly wrong.

Jitter from decoding and scheduling

WebRTC or streaming video may decode frames unevenly. Browsers can also delay callbacks under load. For mouth motion, prefer a representation that tolerates irregular delivery: visemes, timed keyframes, or short pose segments. Then interpolate between adjacent states on the client. That gives you smoother motion even if the transport is bursty.

State update overhead in React

React is not a real-time animation engine. If you model every mouth frame as component state, rerender frequency can become part of the problem. Instead, keep the latest timing data in a ref, and let a requestAnimationFrame loop update the visible frame. React can still render the avatar container, connection status, and controls; it just should not be on the critical path for per-frame animation.

A minimal pattern looks like this:

import { useEffect, useRef } from "react";

}

import { useEffect, useRef } from "react";

}

import { useEffect, useRef } from "react";

}

This is intentionally schematic. The key idea is that the avatar renderer is time-driven, not event-driven. You still need to define how frames are selected and interpolated, but the architecture keeps the control loop stable.

Implementation details that matter in production

When you move beyond a demo, a few details become important:

• Choose the right sampling granularity: too coarse and the mouth looks robotic; too fine and you waste bandwidth and CPU.

• Clamp corrections: if the sync offset changes, adjust gradually so the face does not “snap.”

• Handle silence explicitly: many agents need a neutral face, micro-movements, or blinks during pauses so the avatar does not freeze unnaturally.

• Separate transport from presentation: network code should deliver timestamps and frames; rendering code should turn those into pixels.

One practical rule: do not assume the first audio packet is the first visually meaningful event. Real agents often spend a moment connecting, buffering, or waiting on upstream inference. If you begin rendering too early, you get an out-of-sync “opening mouth” effect before the user even hears speech. Wait for a defined session start signal or a confirmed audio playback start.

Another rule: always test under intentionally bad conditions. Throttle network throughput, introduce packet delay, and background the tab. If your sync loop only works on a localhost happy path, it is not frame-accurate in any meaningful sense.

How Protoface fits into this

Protoface is one way to avoid building the avatar timing stack from scratch. For a React app, the most relevant surface is the customer-managed iframe embed: you drop in an iframe, keep the API key off the browser, and let the embed handle the realtime avatar session, playback, and synchronization details behind the scenes. That is especially useful if you want to add a talking face to a product UI without taking ownership of low-level video timing.

If you are integrating a voice agent server-side, the LiveKit plugin is the other useful path because it attaches a synchronized avatar to an existing agent pipeline. The important architectural benefit is that the avatar becomes part of the same conversational session, rather than a separate video widget that you have to coordinate manually.

For example, if you are creating or managing sessions programmatically, the REST API and Python SDK let you keep the timing and session state on the backend instead of exposing credentials in the browser. The exact request fields depend on the avatar/session model you use, but the shape is straightforward:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

And from Python:

from protoface import Client

)

from protoface import Client

)

from protoface import Client

)

Use the docs for the exact schema and supported options: docs.protoface.com. If you are following the LiveKit path, the plugin and examples live in the GitHub org, and the integration guide for Pipecat is also available for agents built on that stack.

React integration pattern

If you are embedding a realtime avatar into a React app directly, keep the component boundary simple:

1. The component owns layout, controls, and lifecycle.

2. A controller object owns transport state and playback timing.

3. The visual surface is updated imperatively from the controller.

This prevents rerender churn from interfering with frame timing. It also makes it easier to swap implementations later. For example, you can start with a plain video element, then move to a canvas compositor or WebGL-based renderer if you need more control over blending, expression layers, or overlays.

One last thing: if your avatar stream is video-based, remember that “frame-accurate” in the browser is still bounded by the display refresh rate. A 60 Hz screen gives you about 16.7 ms per frame; 120 Hz gives you 8.3 ms. You cannot render arbitrary sub-frame detail, so the real engineering task is keeping the mapping from speech to displayed pose deterministic and low-jitter within those constraints.

Conclusion

Frame-accurate lip-sync is mostly a timing problem: establish one clock, keep transport and presentation separate, and make small corrections when drift appears. In React, keep the animation loop outside normal state-driven rerenders, and treat the browser as a presentation layer, not the source of truth for timing.

If you are building this yourself, test under real network conditions before you call it done. If you want a faster path to a production-ready avatar surface, start with the Protoface docs and the relevant integration surface for your stack. From there, you can decide whether your app should manage sessions through the API, attach an avatar to a LiveKit agent, or embed one directly in the UI.