How to Embed a Realtime Conversational NPC Avatar in a Web Game with Next.js
Next.js guide to embedding a realtime NPC avatar in a web game with synced voice, media sessions, and interrupt handling.
Introduction
If you want a non-player character to feel responsive in a web game, audio alone is usually not enough. Players need visible cues that line up with speech: mouth motion, timing, gaze, and a frame rate that doesn’t fight the interaction loop. The implementation problem is not “how do I render a video?”; it is “how do I keep a conversational avatar synchronized with a realtime voice agent while the game continues to run at browser-frame pace?”
This post walks through the practical shape of that integration in Next.js. By the end, you should know how to:
keep the NPC’s conversation state in a browser game without blocking the render loop,
route mic or game events into a realtime voice agent,
display a lip-synced avatar surface alongside the game canvas, and
avoid the common mistakes around auth, latency, and media synchronization.
What “realtime conversational NPC” actually means
For a game, “realtime” usually means two different clocks are involved:
Game clock: the browser render loop, input events, entity updates, and UI state.
Conversation clock: audio capture, speech-to-text, LLM inference, text-to-speech, and avatar animation/video transport.
You do not want to couple these tightly. The game should keep running even if the model takes 400 ms longer than usual. The NPC interaction should be treated as a side channel that emits events back into the game: start listening, partial transcript, final intent, speaking, idle, interrupted, and so on.
The simplest stable architecture is:
The browser game captures player input and sends voice/media to a backend or managed session.
A voice agent processes speech, produces a response, and exposes realtime turn state.
An avatar surface receives synchronized speech/video output and is rendered next to the game or overlaid in the HUD.
If you skip the separation and try to make the avatar “part of the canvas,” you usually make latency and layout worse. Treat it like a realtime media component, not a sprite.
Embed the avatar outside the render loop
In a Next.js game UI, the avatar is typically a fixed overlay, a chat panel sibling, or a modal dialogue layer. That keeps layout predictable and avoids reflowing your actual gameplay canvas every time the NPC speaks. The key is to render the avatar as a separate DOM surface with its own media lifecycle.
A practical pattern is:
game canvas handles world state and interactions,
React state tracks conversation status, and
an iframe or video element handles the avatar media stream.
For a conversational NPC, you usually want the game to trigger a session when the player enters a zone, clicks the NPC, or opens a dialogue UI. After that, the avatar should stay mounted until the interaction ends.
One thing to avoid: recreating the iframe or media element on every React render. That resets media state, causes visible flicker, and can interrupt the voice turn. In Next.js, keep the session identifier in state, but memoize the embed surface or isolate it in a client component with a stable `key`.
Pass interaction context, not just audio
An NPC in a game rarely exists in a vacuum. The voice agent usually needs some scene context: the player’s current quest, nearby objects, the NPC’s role, and maybe a few user-specific flags. Feed that into the session at creation time or as metadata associated with the conversation.
Keep the payload small and deterministic. You do not need to ship your entire game state to the avatar layer; you need enough context to make the NPC’s next turn coherent. A good pattern is to send:
NPC identifier
scene or quest identifier
player name or handle
brief instruction set for the NPC’s personality and scope
That lets the model stay grounded without becoming a second game engine. When the player moves to a different area, end the current conversation session and start a new one with new context rather than mutating the old one indefinitely.
Next.js integration pattern
In a Next.js app, keep the media/session logic in a client component. The server can prepare the game page and any authenticated session bootstrap data, but the browser owns the realtime interaction.
At a high level, your component flow looks like this:
The exact AvatarSurface implementation depends on the surface you use, but the architectural constraint is the same: the session is created once, then reused until the dialogue ends.
If you are streaming player speech from the browser, do not push raw microphone audio through React state. Use the Web Audio API or the browser media stack directly, and treat the agent connection as an imperative resource. React should reflect state, not carry the media path.
Where the voice agent meets the avatar
The hardest part is usually syncing speech output to facial animation. A usable integration needs the avatar to be driven by the same turn that generates audio, not by a separate timer. Otherwise you get lip motion lagging behind audio, or a face that keeps “talking” after the response finished.
That is why the agent and avatar surfaces should share the same realtime session semantics: when the agent starts speaking, the avatar begins; when the audio stream stops, the face settles; when the user interrupts, both sides stop promptly. In WebRTC-based setups, the transport already gives you low-latency media and session state, but you still need to avoid adding extra buffering layers in your own app.
There are a few implementation gotchas worth calling out:
Don’t synthesize speaking state from text timing. Use actual stream state or turn events.
Don’t remount the avatar on every prop change. Use stable keys and isolated components.
Handle interruption explicitly. Game dialogue is interruptible; players expect to cut NPCs off.
Preserve origin and session boundaries. If you mix multiple players or tabs into one media session, behavior gets confusing quickly.
Using Protoface for the avatar side
On the avatar layer, Protoface gives you a few useful options depending on where your integration lives. For a web game, the cleanest path is often a customer-managed iframe embed: the browser gets an interactive avatar without exposing any API key, and you can control per-embed instructions, voice, parent-origin allowlists, and rate limits. That makes it a good fit when you want the game frontend to stay thin and avoid building a custom media backend.
If you prefer to create sessions server-side, you can do that through the REST API or the Python SDK, then pass the resulting session reference into your game flow. The API is authenticated with your secret key; keep that work off the browser.
Example session creation via curl:
And a minimal Python SDK shape:
The field names above are illustrative; check the docs for the exact request shape and session lifecycle details. If you want to wire this into a broader voice-agent stack, the LiveKit plugin is another path: add the avatar to the agent so the conversation and face stay synchronized in the same media pipeline. The plugin and examples are in the relevant repo on GitHub, and the integration guide in the docs is the right place to confirm the current setup.
For API details and current quickstarts, see the docs. If you want examples beyond a bare session call, the GitHub organization has the integration code and starter repos that are more useful than a polished marketing demo.
Practical trade-offs for game developers
There is no single “best” avatar integration. The right choice depends on how much infrastructure you want to own.
If you want the least backend work: use an iframe embed and keep the game client focused on gameplay.
If you already run a voice agent backend: create sessions programmatically and attach the avatar to your existing turn flow.
If you need tight agent/media control: integrate at the agent layer so interruption, turn-taking, and video state share one lifecycle.
Also pay attention to quality-tier cost. Realtime video avatars are not free to run, and the visual fidelity you choose should match the interaction’s importance. An NPC in a town hub may need less fidelity than a critical story character.
Conclusion
The implementation pattern is straightforward once you separate responsibilities: the game loop handles gameplay, the voice agent handles conversation, and the avatar surface handles synchronized visual output. In Next.js, that usually means a client-side media component, a stable session lifecycle, and careful handling of interruption and remounts.
If you are building a conversational NPC, start with a narrow interaction: one character, one scene, one session lifecycle. Get the media timing right before you add branching logic or a large cast. Then use the docs and quickstarts to wire the avatar into the path that best matches your stack. The result is a web game character that can actually hold a conversation instead of just playing a canned animation.