Introduction

The hard part of connecting text-to-speech to a talking avatar is not generating audio; it’s keeping the face synchronized with the audio stream without adding visible latency or drift. If the mouth motion is driven by stale text, buffered audio, or a separately timed animation loop, the result looks “off” immediately. The fix is to treat the avatar, the TTS output, and the transport as one realtime pipeline: the same utterance should produce audio and lip motion from the same timing source.

By the end of this post, you should have a practical mental model for building that pipeline in Next.js, know where lip-sync usually breaks, and understand the simplest integration patterns that keep latency low and playback stable.

Start with the pipeline, not the UI

A Next.js app is usually just the orchestration layer. The actual realtime path looks more like this:

1. User message arrives in your app.

2. Your backend or agent decides on a response.

3. TTS begins streaming audio as soon as the first chunk is available.

4. The avatar component receives the same session/audio timing information and animates the face against that stream.

5. The browser renders the video and plays the audio with minimal buffering.

If you split those responsibilities across unrelated timers or separate requests, lip-sync degrades. The common failure modes are:

• Audio starts late because the browser waits for the full MP3 before playback.

• Mouth motion leads audio because animation is based on text tokens rather than actual phoneme/audio timing.

• Audio and video drift because they are produced or buffered independently.

• Session resets mid-utterance because the UI re-renders and tears down the media element.

The practical rule is simple: once a response starts, keep the media session stable until that utterance finishes. In a Next.js app, that usually means the avatar player lives in a component whose identity does not change on every render, and your transport layer streams rather than batch-delivers.

Use a realtime transport that preserves timing

For a lip-synced avatar, “TTS” usually means streaming synthesis, not a one-shot file download. That distinction matters because a one-shot file forces you to wait for completion before playback, which adds latency and makes the avatar feel detached from the conversation.

What you want instead is one of these patterns:

• Streaming audio to the browser, with the avatar session synchronized to the same stream.

• A voice-agent runtime that emits audio incrementally and exposes stable timestamps or session state.

• A server-managed media session where the browser only joins and renders, rather than trying to coordinate timing itself.

In all three cases, avoid tying animation to raw text length or estimated utterance duration. Real TTS systems can change timing based on punctuation, emphasis, and prosody, so the only reliable source of truth is the audio stream or the media session that wraps it.

Next.js integration: keep the browser thin

In a Next.js app, the safest architecture is to keep API credentials off the client and treat the browser as a consumer of a session already created on the server. That keeps the app easy to reason about and reduces the chance that a page reload will interrupt media orchestration.

A minimal server route can create or fetch a session, then return a session token or embed URL to the client. The exact request fields depend on your backend, but the shape is usually straightforward:

import { NextResponse } from 'next/server';

}

import { NextResponse } from 'next/server';

}

import { NextResponse } from 'next/server';

}

On the client, render the avatar in a stable container and avoid remounting it unnecessarily:

'use client';

}

'use client';

}

'use client';

}

The example above uses an iframe because it cleanly isolates the media session from your app’s render cycle. That is especially useful when the avatar is interactive and stateful. If you instead mount a custom player directly in React, make sure the component does not unmount on every chat state update, or you’ll introduce glitches that look like lip-sync bugs.

A few implementation details matter in practice:

• Autoplay policy: browsers often require a user gesture before playing audio. Plan for a click-to-start interaction if needed.

• Stable keys: do not give the iframe or player a changing React key unless you intend to restart the media session.

• Single source of truth: keep the conversation state on the server or in a dedicated agent runtime, not split across random client state stores.

• Backpressure: if the user interrupts the agent, stop the current utterance cleanly before starting the next one.

When you are using a voice agent, sync at the agent layer

If your application already has a live voice agent, the cleanest integration is to plug the avatar into the agent runtime instead of trying to glue audio and video together manually. That way the agent owns turn-taking, streaming audio, interruptions, and the timing of the response. The avatar then becomes just another synchronized output of the same session.

For LiveKit-based systems, the relevant integration surface is the LiveKit/plugin-style quickstart path and the Python plugin published as livekit-plugins-protoface on PyPI. The exact API is documented in the package and docs, but the idea is simple: attach the avatar to the agent so the agent’s synthesized speech drives the mouth animation automatically.

# illustrative only; refer to the plugin docs for exact names and config fields

)

# illustrative only; refer to the plugin docs for exact names and config fields

)

# illustrative only; refer to the plugin docs for exact names and config fields

)

This style of integration reduces glue code and eliminates the common mismatch where the UI thinks an utterance has started but the audio pipeline has not yet emitted anything. It also fits interruption handling better: when the agent cancels or truncates a response, the avatar session can stop animating on the same boundary rather than finishing a fake mouth movement.

One concrete way Protoface fits here

Protoface is useful when you want the avatar side of the pipeline to be managed as a realtime session instead of hand-rolling lip-sync in the browser. For a Next.js app, the most practical surface is the customer-managed iframe embed: your server creates the session, your client loads the iframe, and the browser never sees an API key. That keeps the integration simple and avoids leaking secrets into frontend code.

If you need server-side control instead, the REST API and Python SDK let you create avatars and sessions from your backend. Here is a minimal request shape to illustrate the flow:

curl https://api.protoface.com/v1/sessions \
}'

curl https://api.protoface.com/v1/sessions \
}'

curl https://api.protoface.com/v1/sessions \
}'

The exact endpoints and fields are documented in the docs, but the integration model stays the same: create the session on the server, pass a session-specific URL or token to the browser, and let the media session own timing.

Debugging lip-sync issues in practice

When lip-sync looks wrong, check these in order:

1. Does the audio stream start immediately? If not, you are probably waiting for a full response rather than streaming.

2. Is the avatar session stable? If the iframe or player remounts, animation resets and desync becomes visible.

3. Are you mixing clocks? Text timestamps, UI timers, and media timestamps are not interchangeable.

4. Are interruptions handled atomically? A new turn should cancel the prior one at the agent layer, not just visually hide the previous utterance.

5. Is buffering too aggressive? Over-buffering can make the avatar feel delayed even if it stays synchronized once playback starts.

If you need a faster path to a working baseline, start from one of the quickstarts in the Protoface repo family and adapt the session handoff to your app. That gets you a known-good media path before you add custom conversation logic.

Conclusion

To connect TTS to a Next.js avatar without breaking lip-sync, keep the pipeline realtime end-to-end: stream audio, preserve a stable media session, and let the avatar animate off the same timing source as the speech. In practice, that means the browser should render a session created elsewhere, not try to invent timing on its own.

For documentation, integration details, and the supported session/SDK shapes, start with docs.protoface.com. If you are building on LiveKit, use the plugin path; if you are embedding on the web, use the managed iframe route. Either way, the architectural goal is the same: one utterance, one synchronized media session, no fragile client-side lip-sync code.