How to Add a Lip-Synced AI NPC to Your Game in Next.js
Build a lip-synced AI NPC in Next.js with realtime voice, avatar sync, server-side sessions, and safe browser embedding.
Introduction
If you want an NPC that can talk, react, and actually feel present in a game UI, the hard part is not rendering a face. The hard part is keeping three systems in sync: the model’s text or speech output, the audio stream that the player hears, and the avatar animation that the player sees. Once those drift apart, the illusion breaks immediately.
In this post, I’ll show the practical shape of a lip-synced AI NPC in a Next.js app: how to structure the frontend, how the realtime session fits into the rendering pipeline, what to watch out for with latency and browser media constraints, and where a service like Protoface fits when you do not want to build the avatar layer yourself. By the end, you should have a clear implementation plan for a web-based game character that speaks in sync with generated audio.
What “lip-synced AI NPC” actually means
A lip-synced NPC is not just an image that animates while audio plays. It is a realtime media pipeline:
The player sends a prompt, voice input, or game event.
Your agent generates a response, often through an LLM plus a TTS engine.
That response is streamed as audio with timing metadata or frame-level alignment.
The avatar renderer uses the same timing to drive mouth shapes, head motion, and gaze.
For a browser game, the key constraint is end-to-end latency. If you wait for a full response before rendering anything, the NPC feels dead. If you stream audio but the face lags by a few hundred milliseconds, the mismatch is obvious. In practice, you want the avatar attached to the same realtime session as the voice agent so audio and video are derived from one source of truth.
In Next.js, that usually means keeping your game UI in the browser, while a backend service or agent runtime handles session creation, authorization, and media coordination. The frontend should not know any secret API key and should only receive short-lived session data or an embeddable URL.
Architecture in a Next.js game
A sane implementation splits responsibilities like this:
Next.js client: game UI, NPC panel, player input, and playback surface.
Agent runtime: conversation logic, tool calls, speech generation, and state.
Avatar layer: the talking face, synchronized to the agent’s audio output.
Session backend: creates and manages sessions, enforces limits, and stores metadata.
The browser should render the avatar as a video element or iframe-backed surface, depending on the integration model. The important detail is that the avatar is not independently “thinking.” It is consuming the same realtime stream as the agent. That keeps lip movement aligned with the utterance timing instead of trying to infer phonemes client-side.
In a Next.js app, you typically use one of two approaches:
Direct media integration: your app joins a realtime session, receives audio/video tracks, and renders them in the page.
Managed embed: your page loads an iframe, and the avatar session runs in an isolated origin you do not control.
For a game prototype, the managed embed is often the shortest path. For a deeply integrated in-game UI, direct media control gives you more room to coordinate animations, subtitles, and gameplay state.
Implement the Next.js side cleanly
Do not create sessions from the browser with long-lived credentials. The browser should ask your server for a session token or embed URL, and the server should call the avatar/session API using its own secret key.
A simple Next.js route can proxy the server-side call. The exact request fields depend on your avatar/session configuration, but the pattern looks like this:
Then the client uses the returned session data to mount the avatar surface. If your integration is iframe-based, the browser only needs the embed URL and any allowed parameters. If it is media-based, you would attach the remote video track to a video element or canvas-backed renderer.
Two practical notes for Next.js:
Use the app router or route handlers for secrets. Do not put the API key in a client component or environment variable exposed to the browser.
Keep game state separate from avatar state. The NPC can be speaking while the game world continues ticking; do not block gameplay on avatar lifecycle events unless you really mean to.
Realtime sync: latency, buffering, and mouth shapes
Lip sync only feels right when the avatar follows the audio timing, not the text timing. That sounds obvious, but it is where most integrations get sloppy. Text-based animation usually ends up over-articulated and late. A realtime avatar pipeline should instead use speech-aligned video generation or facial animation driven by the voice stream itself.
For developer sanity, think in terms of timing budgets:
Network round trip: session setup, signaling, and track negotiation.
Model latency: time for the agent to produce the response.
TTS and rendering latency: time to convert output into audio and synchronized face motion.
Browser playback latency: buffering and media pipeline startup.
Your goal is not zero latency. Your goal is consistent latency. A stable 700 ms delay often feels better than a jittery 300 ms delay. If your game already has diegetic pauses, radio chatter, or NPC dialogue boxes, you can mask the delay with UI affordances like a subtle “thinking” state, but you should still preserve stream continuity once speech starts.
For game UX, also decide what happens when the player interrupts the NPC. Realtime systems often need barge-in behavior: stop the current utterance, cancel the remaining audio, and switch the avatar back to an idle listening state. If you do not implement cancellation, the character will keep talking over the player, which feels broken fast.
One practical way to add the avatar layer
If you are using LiveKit for the voice agent, the simplest path is to drop the avatar into that existing agent pipeline. Protoface provides a LiveKit Agents plugin that attaches a synchronized talking face to the agent, so the avatar tracks the same realtime speech stream instead of being wired up as a separate subsystem. The plugin is published on PyPI as livekit-plugins-protoface, and the examples in the plugin repository are the right place to start if your backend already speaks LiveKit. See the plugin repo for the current integration patterns and sample code: plugin examples.
A minimal agent-side sketch looks like this:
The point of the plugin is not novelty; it is to avoid writing your own media sync layer. If your game backend already uses a voice agent runtime, this is usually less work than trying to synthesize a talking face from scratch.
If you are not on LiveKit, you can still use the REST API directly from your backend or use the Python SDK for session management. The SDK is useful when you want to create avatars, start sessions, or inspect usage from a script or server process instead of hand-rolling HTTP calls. For Python-centric stacks, the SDK repo is the place to look: Python SDK. For HTTP-only workflows, the API is straightforward: authenticate with a bearer token, create the session server-side, and hand only the minimum necessary data to the browser.
Browser embedding without exposing secrets
If your game UI only needs an avatar panel, an iframe can be the cleanest option. It keeps the media session isolated, avoids browser credential exposure, and makes the frontend integration trivial. That matters if you are shipping on a tight deadline or if you want to let designers tweak voice and instructions without touching application code.
The main security controls to keep in mind are:
Parent-origin allowlisting: only your game origin should be allowed to embed the session.
Per-embed instructions: each NPC can have its own behavior and voice settings.
Rate limits: duration and per-IP limits protect you from abuse.
That model is a good fit for a web game where the avatar is part of the HUD or dialogue box rather than a fully custom 3D character. It also means you can iterate on NPC personalities quickly in the dashboard and in-browser playground without redeploying the entire game.
Operational details that matter in production
Once the prototype works, the usual problems show up in production: session cleanup, billing, and debugging. Track session lifecycle explicitly in your app so that abandoned tabs or rage-quit players do not keep consuming resources. Surface avatar state in your telemetry the same way you would surface game server state: connected, speaking, idle, interrupted, errored.
Also make sure you have a fallback when media negotiation fails. If the avatar cannot connect, your NPC should degrade gracefully to text dialogue instead of blocking the rest of the game. That is especially important for embedded browser experiences where autoplay rules, microphone permissions, and corporate proxies can all interfere.
Finally, keep in mind that quality tiers affect cost. If you only need a lightweight helper NPC in a menu screen, you may not need your highest-quality avatar setting. If the character is central to the experience, spend the budget there and optimize elsewhere.
Conclusion
The shortest path to a convincing lip-synced AI NPC in Next.js is to treat the avatar as part of the realtime voice pipeline, not as a separate animation problem. Keep session creation server-side, stream media rather than waiting for full responses, and choose an integration surface that matches how much control you need in the browser.
If you want to go deeper, start with the docs at docs.protoface.com, then pick the path that matches your stack: the LiveKit plugin if your agent already lives there, the Python SDK if you want server-side control, or an iframe embed if you want the simplest browser integration. The quickstart repos linked from the project README are a good next stop once you have the core architecture clear.