Introduction

Frame-accurate lip-sync is one of those details that is easy to underestimate until you ship a realtime avatar and the illusion falls apart. If the face starts late, the mouth lags behind phonemes, or the video cadence jitters under network load, users notice immediately. The goal is not just “make the avatar talk”; it is to keep the visual speech signal aligned with the audio stream closely enough that the result feels continuous and responsive.

In this post, I’ll walk through the practical pieces you need to add lip-sync to a React-based realtime avatar UI over WebSocket streaming: how to think about timing, how to handle audio and animation state, where frame accuracy comes from, and how to avoid common sync bugs. I’ll also show where Protoface fits if you want to move faster with a production-oriented avatar backend.

What “frame-accurate” actually means in a realtime avatar

In a standard browser video element, “sync” usually means audio and video are played from the same encoded asset with shared timestamps. Realtime avatars are different. You often receive:

• audio chunks or a live audio stream,

• animation state or viseme timing metadata, and sometimes

• video frames or a generated frame stream.

Those streams are not guaranteed to arrive together. WebSockets give you a bidirectional transport, but they do not provide intrinsic media synchronization. So the client has to maintain a local timeline and map each incoming event onto that timeline.

For lip-sync, the useful unit is usually not the frame itself but the animation keypoint: a phoneme, viseme, or mouth-shape transition with a timestamp relative to the audio start. If you can associate each mouth movement with an audio clock, then rendering each frame becomes deterministic: at playback time t, choose the mouth pose that corresponds to the current audio position.

That is the difference between “pretty good” and “stable under jitter.”

Designing the client timeline in React

The core implementation pattern is simple:

1. When a speech turn starts, capture a playback origin on the client.

2. Buffer incoming audio and timing metadata.

3. Render mouth states from the current audio position, not from message arrival time.

4. Use requestAnimationFrame for visual updates, but treat it as a render tick, not the source of truth.

A common mistake is to use the arrival timestamp of a WebSocket message to decide which mouth shape to show. That couples your animation to network jitter. Instead, derive a shared speech clock. In practice, that clock is often:

• audio output time for a live audio element, or

• client-side elapsed time from the moment the agent began speaking.

When you receive a speech event, you should preserve the server’s relative offsets. If the server says “viseme X begins at 180 ms,” store that against the utterance. Then, while the utterance is active, compute:

speechOffsetMs = nowMs - utteranceStartMs - playbackDelayMs

speechOffsetMs = nowMs - utteranceStartMs - playbackDelayMs

speechOffsetMs = nowMs - utteranceStartMs - playbackDelayMs

and find the latest viseme at or before that offset.

A minimal React pattern for mouth-shape playback

The code below is intentionally simplified. It assumes the backend sends a start event, then a sequence of viseme markers with offsets relative to that start. It does not depend on a specific avatar library; the point is the timing model.

type Viseme = { atMs: number; shape: string };

}

type Viseme = { atMs: number; shape: string };

}

type Viseme = { atMs: number; shape: string };

}

There are two important caveats here:

• Do not reset state on every message. Treat the utterance as a timeline, not a collection of independent events.

• Do not assume exact arrival order. If your backend can send out-of-order markers, buffer and sort them by atMs.

For actual production use, you will also want a small jitter buffer so that a marker arriving a few milliseconds late does not cause the mouth to “pop” backward. A 50–150 ms buffer is usually enough to absorb network variance without making the avatar feel sluggish.

WebSocket streaming: what to send and what to avoid

If your client and backend communicate over WebSocket, keep the protocol explicit. You want separate messages for speech lifecycle events and for animation payloads. A typical flow looks like this:

{ "type": "speech_start", "utteranceId": "u_123", "playbackDelayMs": 80 }
{ "type": "viseme", "utteranceId": "u_123", "atMs": 240, "shape": "FV" }

{ "type": "speech_start", "utteranceId": "u_123", "playbackDelayMs": 80 }
{ "type": "viseme", "utteranceId": "u_123", "atMs": 240, "shape": "FV" }

{ "type": "speech_start", "utteranceId": "u_123", "playbackDelayMs": 80 }
{ "type": "viseme", "utteranceId": "u_123", "atMs": 240, "shape": "FV" }

There are a few things worth avoiding:

• Don’t stream one WebSocket message per rendered frame. That creates unnecessary overhead and makes timing noisier.

• Don’t bake timing into animation names. Keep the shape model separate from the timeline.

• Don’t let the UI “free-run” the mouth. If the avatar is speaking silently while audio stalls, the illusion breaks fast.

If you are sending audio as well, keep audio and timing metadata logically coupled. The most robust pattern is to treat the audio stream as the source of speech start/stop and the viseme stream as a subordinate schedule. That way, if you need to recover after a reconnect, you can replay the timing schedule against the audio clock rather than trying to reconstruct state from the UI.

Animation rendering details that matter in practice

The browser part is usually not the bottleneck; the timing model is. Still, a few rendering details matter if you want the avatar to look stable:

• Interpolate between shapes. Hard switching between mouth poses often looks robotic. Even a short 60–100 ms blend can help.

• Keep render cost low. If you are compositing avatar video in React, avoid unnecessary rerenders. Store the active shape in a ref or a very small state slice.

• Handle silence explicitly. Define a neutral “rest” pose and transition into it when speech ends rather than waiting for a timeout.

• Use a fallback strategy for packet loss. If a viseme marker is missing, hold the nearest plausible shape instead of snapping to neutral.

Also, remember that “lip-sync” is not only about the mouth. Eye blinks, head motion, and small facial movements help cover unavoidable latency. If the mouth is slightly off but the face still feels alive, users forgive minor error much more readily.

Where Protoface fits

If you want to avoid building the avatar backend, timing scheduler, and video generation pipeline yourself, Protoface gives you a developer-facing realtime avatar layer that plugs into the stack you already use. For React applications, the most relevant surfaces are the REST API for creating sessions and the browser-friendly embed model; for voice-agent backends, the LiveKit plugin is the direct path.

For the backend side, session creation is done through the API with normal bearer auth:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

Exact fields depend on the endpoint shape in the docs, but the point is the same: your app asks for a realtime session, then streams or attaches to the session over the supported transport. The service handles the avatar side of the lip-sync pipeline, which is the part that is painful to build correctly from scratch.

If you are using LiveKit voice agents, the plugin path is even more direct. The livekit-plugins-protoface package drops an avatar into the agent so the agent can speak with a synchronized video face without you managing separate lip-sync logic in the app layer. That is useful when your “React app” is really just the frontend for an agent-driven conversation and you want the avatar rendering to stay aligned with the voice pipeline.

For implementation details and current request/response shapes, the docs are the right source of truth: docs.protoface.com. If you want to inspect the plugin example code rather than just the API shape, the relevant repository is also useful: GitHub.

Testing and debugging sync issues

Most lip-sync bugs are timing bugs, not ML bugs. When something looks off, instrument the pipeline before you start changing models or assets.

Useful diagnostics:

• Log the speech start time on both client and server.

• Record viseme offsets and the shape chosen at each render tick.

• Measure transport delay separately from playback delay.

• Compare audio-clock position to rendered mouth state for a few utterances.

If the avatar is consistently late by the same amount, you probably need to adjust initial playback delay. If it jitters, you need buffering or better state management. If it drifts over time, you are probably deriving animation from wall-clock time instead of the media timeline.

One practical trick is to record a short test phrase with strong consonants—something with p, f, t, and m sounds—because those shapes expose timing errors more clearly than vowels do. Then inspect whether the visible lip closures line up with the audible consonants.

Conclusion

To get frame-accurate lip-sync in a React realtime avatar, treat the problem as timeline synchronization, not just animation. Keep a speech clock, buffer incoming markers, render from audio-relative offsets, and make the WebSocket protocol explicit about starts, stops, and visemes. That approach holds up far better than “update the mouth when a message arrives.”

If you want to skip building the avatar pipeline yourself, start with the docs and one of the quickstarts, then decide whether the REST API, Python SDK, or LiveKit integration fits your architecture best. The key is to keep the client simple and the timing model disciplined. From there, the rest is mostly engineering hygiene.

Next step: read the implementation details at docs.protoface.com, then wire up a small end-to-end prototype and measure the delay budget before you polish the UI.