Introduction

If you’re adding a talking avatar to a Next.js app, the hard part is not “showing a video.” It’s keeping the face, voice, and transcript aligned tightly enough that the result feels intentional instead of uncanny. The usual failure modes are familiar: audio arrives before the mouth movement, the mouth keeps moving after the audio stops, or the browser redraws a new frame source every time state changes and introduces visible jitter.

This post walks through the practical pieces of accurate lip-sync in a realtime avatar app: how the media pipeline should be structured, where synchronization actually comes from, what can go wrong in a React/Next.js frontend, and how to keep latency low enough that the avatar appears responsive. By the end, you should be able to reason about the full path from speech generation to rendered video and integrate a realtime avatar without turning your frontend into a timing bug farm.

What “accurate lip-sync” actually means

In practice, lip-sync is not one problem. It’s three separate alignment problems that need to be solved together:

• Phoneme-to-viseme alignment: mapping speech sounds to mouth shapes.

• Audio-video playout alignment: making sure frames and audio are rendered in the same temporal window.

• Interaction latency: reducing the delay between the user speaking and the avatar reacting.

For a realtime avatar, the second and third are usually where the engineering effort goes. If you’re using a model or service that generates the face and the speech together, you still need to deliver and render them with stable timestamps. If the avatar is driven by a voice agent, the mouth animation should be derived from the same underlying audio stream or speech event timeline, not from a separate best-effort guess in the browser.

That implies a basic architectural rule: keep the lip-sync source of truth server-side or in the media pipeline, and keep the browser focused on playback.

Design the Next.js app around a media session, not a component re-render loop

In Next.js, it’s tempting to treat the avatar as “just another component” that receives props every time app state changes. That’s the wrong abstraction for realtime media. A video face needs a stable session and a stable element in the DOM. Re-rendering the component should not tear down the media connection, reset the stream, or replace the underlying media element.

A better mental model looks like this:

1. Initialize a session once.

2. Attach the avatar stream to a persistent <video> element or iframe.

3. Send conversational events, text, or audio into the session.

4. Let the media pipeline own timing and lip-sync.

In a Next.js client component, that usually means:

• creating the session inside an effect that runs once;

• storing mutable session objects in refs, not state;

• avoiding prop-driven remounts of the video element;

• cleaning up explicitly on unmount.

Here is the kind of pattern you want. The exact fields depend on the SDK or API shape, but the structure matters more than the details:

import { useEffect, useRef } from "react";

}

import { useEffect, useRef } from "react";

}

import { useEffect, useRef } from "react";

}

Two gotchas are worth calling out:

• Don’t bind media lifecycle to React state. State changes are for UI, not for maintaining a realtime transport.

• Use <video playsInline> on mobile browsers. Otherwise you can get unwanted fullscreen behavior or autoplay restrictions that break the experience.

Keep latency low enough that the illusion survives

Even perfect viseme mapping won’t save a 1.5-second round trip. Users tolerate some delay, but lip-sync quality degrades quickly when speech generation, network transport, and rendering stack up. For conversational avatars, aim to minimize the time between “user finished speaking” and “avatar starts responding.”

There are a few practical techniques:

• Use streaming, not batch. Stream audio or speech events as soon as they’re available instead of waiting for a complete utterance.

• Keep the browser path thin. Avoid decoding or re-encoding media in the client unless you have a specific reason.

• Preserve session continuity. Rejoining a session should not force the avatar to rebuild its state from scratch.

• Debounce UI, not media. You can update chat bubbles or status indicators less frequently, but don’t let UI throttling affect the media stream.

Another subtle issue is clock drift. If audio and video are produced by different systems, they can slowly diverge even if they start aligned. In a good realtime avatar stack, the media layer carries timing information forward so the renderer can keep the stream coherent. If you’re seeing the mouth “lead” or “lag” consistently, the bug may not be the face model at all; it may be timestamp handling, buffering, or the way the stream is attached in the browser.

How to wire it into a voice agent

If your app already has a voice agent, the cleanest integration is to treat the avatar as another realtime participant in that conversation. The agent generates speech; the avatar session consumes that speech and renders the synchronized face. The important thing is that the avatar is fed from the same response stream the user hears, rather than from a separate text-to-face approximation.

For developers using LiveKit-based voice agents, the usual integration point is a plugin that inserts the avatar into the agent pipeline. The plugin approach is attractive because it keeps the transport and timing inside the agent stack, which is exactly where lip-sync should live. You can inspect the package and examples in the plugin repository; the key idea is to keep the avatar attached to the live voice session rather than spinning up a separate ad hoc media channel: https://github.com/protoface-ai/protoface-quickstart-openai-realtime.

For a typical setup, your agent emits audio and the avatar consumes it as part of the same session lifecycle. A minimal Python-oriented pattern looks like this:

from protoface import Client

session.start()

from protoface import Client

session.start()

from protoface import Client

session.start()

If you’re using a lower-level API instead of an SDK, the pattern is the same: create a session, connect it to your voice pipeline, and keep the session stable while the conversation runs. The REST API is there for provisioning and management; the realtime session is where lip-sync actually happens. See the API and request examples in the docs when you need the exact payload shape: https://docs.protoface.com.

Operational details that matter in production

A few implementation details tend to separate “works on my machine” from a production-grade integration:

• Auth and secrets: keep API keys server-side. In browser apps, never expose them directly.

• Session cleanup: close sessions on route change, tab close, and reconnection failure.

• Observability: log session creation, join latency, disconnect reasons, and media start time. If lip-sync drifts, these are the first numbers you’ll need.

• Browser compatibility: validate autoplay, media permissions, and mobile Safari behavior early.

If you want a quick way to create or inspect avatars and sessions from the command line, the REST API is useful for debugging. A minimal request looks like this:

curl https://api.protoface.com/sessions \
-d '{"avatar_id":"avatar_123"}'

curl https://api.protoface.com/sessions \
-d '{"avatar_id":"avatar_123"}'

curl https://api.protoface.com/sessions \
-d '{"avatar_id":"avatar_123"}'

Keep in mind that the exact request fields and response schema are documented in the API reference. The point here is not the specific endpoint shape; it’s that you should be able to reproduce session creation outside the browser when you need to isolate a media bug.

Protoface in this architecture

Where Protoface fits is the part you do not want to custom-build: the realtime avatar session, the synchronized video face, and the developer-facing plumbing around it. For a Next.js app, the practical win is that you can keep your frontend simple: initialize a session, render the video element or embed, and let the backend/media layer handle the synchronization problem. If you’re integrating with a voice agent, the LiveKit plugin path is the most natural place to start; if you’re provisioning avatars or sessions directly, use the REST API or Python SDK; if you want a browser-only integration, use the customer-managed iframe embed and keep secrets off the client entirely.

That division of responsibility is important. Frontends should orchestrate UI and session lifecycle. Realtime media systems should own timing, lip-sync, and transport.

Conclusion

Accurate lip-sync in a Next.js avatar app is mostly about respecting the media pipeline. Keep the avatar session stable, avoid tying it to React re-renders, stream speech with low latency, and let the avatar system own the synchronization between audio and mouth movement. Once you do that, the browser becomes a thin presentation layer instead of the place where timing gets corrupted.

If you’re implementing this yourself, start by prototyping the session lifecycle in isolation, then wire it into your app with a persistent video element or embed. For concrete integration details, API examples, and SDK usage, the docs are the right next stop: https://docs.protoface.com.