Integrating a Streaming AI Avatar into a Unity Game for NPC Dialogue
Integrate a streaming AI avatar into Unity NPC dialogue with backend sessions, WebRTC-style media, and low-latency voice sync.
Introduction
Adding a streaming AI avatar to a Unity NPC is mostly an integration problem, not a rendering problem. The hard part is coordinating three realtime systems at once: your game loop, the voice agent, and the avatar video stream. If you get the boundaries right, the NPC can listen, speak, and show a synchronized face without blocking gameplay or turning your client into a networking mess.
In this post, I’ll show the architecture I’d use for a Unity NPC that talks to players with low latency, how to keep the avatar synchronized with agent speech, and where the realtime session should live. By the end, you should be able to wire Unity to a voice backend, stream the avatar into the UI, and avoid the common traps around latency, authority, and API-key handling.
Start with the right mental model
For a game NPC, the avatar is not the AI. The avatar is the presentation layer for an agent that already has a voice pipeline and conversation logic. That distinction matters because Unity should generally not orchestrate language model calls directly unless you have a very specific reason. Instead, treat the game client as a thin realtime consumer:
Unity captures player input, usually microphone audio or push-to-talk events.
A backend voice agent handles speech-to-text, dialog state, and response generation.
The avatar stream renders the agent’s face, lip sync, and video state.
This separation keeps your client deterministic enough for gameplay while allowing the agent side to evolve independently. It also gives you a clean failure mode: if the avatar stream drops, the NPC can still exist as audio-only or text-only, rather than taking the whole interaction down.
Use a session boundary, not a polling loop
Realtime avatars work best when the client joins a session that persists for the duration of a conversation. You do not want to poll for a new image every frame or request one-off clips for each utterance. The right pattern is:
Create a session when the NPC conversation starts.
Attach the player’s audio or text events to the agent.
Receive a streaming video face that tracks the agent’s speech.
Tear the session down when the conversation ends or times out.
That gives you backpressure, session-level configuration, and a single place to apply rate limits or conversation-specific instructions. It also maps cleanly onto WebRTC-style realtime systems, where the client subscribes to a media session instead of repeatedly fetching assets over HTTP.
Unity integration pattern: render the avatar as a remote video surface
In Unity, the avatar typically lands in one of two places: a UI panel for dialog scenes, or a world-space billboard / mesh texture for in-game NPCs. Either way, the implementation should look like a normal video receiver with a stable texture source.
At a high level:
Join the realtime session from Unity.
Subscribe to the avatar video track.
Map incoming frames onto a Unity texture or video surface.
Drive subtitle, lip-sync, or dialog UI from the same session state.
If you are already using WebRTC in Unity, the avatar stream should feel familiar: negotiate, subscribe, render. The important detail is to keep the avatar transport isolated from gameplay networking. Your game server should not be relaying avatar frames unless you have a strong reason; let the avatar service handle media distribution and let the game server handle game state.
Practical client-side concerns
A few issues come up immediately in production:
Latency budget: Avatar responsiveness is dominated by speech pipeline latency, not Unity rendering. Aim to keep the path from player utterance to first agent response as short as possible.
State ownership: Decide whether the game server or the voice backend owns conversation state. Mixing both usually creates hard-to-debug drift.
Cancellation: If the player interrupts the NPC, the backend should support barge-in or turn cancellation so the avatar does not keep talking over new input.
Fallbacks: If video fails, preserve the dialogue. Text subtitles or audio-only output are much better than dropping the interaction entirely.
In practice, the avatar stream should be treated like any other realtime media source: reconnectable, observable, and disposable. Log session IDs, track join/leave events, and surface transport failures in your game telemetry so you can distinguish network issues from agent issues.
Backend flow: keep secrets off the client
For a Unity game, the cleanest setup is to create and manage avatar sessions on the backend and hand the client only the minimum session credentials needed to join. Do not embed long-lived API keys in the game build. Players can inspect client binaries, intercept requests, and extract anything shipped to the device.
A backend flow usually looks like this:
Your game server authenticates the player.
The server requests or prepares a realtime avatar session.
The server returns a short-lived session token or join info to Unity.
Unity connects to the session and starts rendering the avatar.
That also gives you a natural place to enforce per-player limits, conversation policies, and NPC-specific behavior. For example, you can attach different instructions depending on whether the player is talking to a merchant, quest giver, or tutorial guide.
Example: creating a session with the REST API
Below is the kind of server-side call you would make to provision an avatar session. Exact request fields depend on the API shape in the docs, but the pattern is straightforward: authenticate with an API key, create the resource, then pass the relevant session data to the client.
That is the right pattern even if your actual deployment uses a different endpoint shape: create on the server, join from the client, and never expose the API key in Unity.
Where Protoface fits without adding unnecessary plumbing
This is exactly the sort of workload Protoface is meant for: a developer-facing avatar service that can sit behind your voice agent and supply the synchronized talking face. For a Unity NPC, that means you focus on gameplay integration and session management rather than building your own avatar video stack.
If you are using a Python backend, the Python SDK is the easiest place to create or manage sessions before handing Unity a joinable session. If you are already running a voice agent stack, the LiveKit plugin can be the shortest path when your agent lives in LiveKit Agents and you want the avatar attached to that same conversation flow. The docs at docs.protoface.com cover the API and integration details you will need.
Example: backend orchestration in Python
This is a minimal shape for server-side orchestration. It creates the session on the backend, then returns data that the Unity client can use to join. Replace the placeholders with the actual SDK calls from the Python package docs.
Even if your exact SDK method names differ, keep the same separation: provisioning on the backend, media consumption in Unity, and no secret material in the client.
If you are using LiveKit Agents
If your NPC speech stack is already built on LiveKit Agents, the integration path is even cleaner: drop the avatar into the agent rather than teaching Unity how to talk to the LLM, TTS, and lip-sync pipeline separately. The LiveKit plugin published on PyPI is designed for that. Your agent continues to manage turn-taking and speech, while the plugin adds the synchronized talking face as another realtime output.
That matters because a game NPC often needs a single conversational brain with multiple outputs: audio, video face, and possibly text captions. When those are tied together at the agent layer, you avoid the classic bug where the audio says one thing, the subtitle says another, and the avatar is two seconds behind both.
If you are using LiveKit already, the plugin repo examples are the right place to start: https://github.com/protoface-ai/protoface-plugin-pipecat. If your stack is Pipecat-based instead, the Pipecat service guide is the relevant reference: https://docs.pipecat.ai/api-reference/server/services/video/protoface.
NPC-specific trade-offs
Game NPCs are a little different from customer-support avatars or web demos. Players expect them to be reactive, but they also expect continuity and character consistency. That leads to a few useful design choices:
Short response windows: NPCs should often speak in shorter turns than support agents. Players are usually in motion.
Interruptible speech: Let the player cut the NPC off. Games feel much better when the agent respects player control.
Context scoping: Keep per-NPC memory bounded. A merchant should not inherit the entire history of a boss fight.
Visual fallback: If the avatar stream is unavailable, continue the conversation with audio or captions rather than stalling gameplay.
These are product decisions as much as technical ones. The engineering goal is to make the avatar feel like part of the game, not a browser widget embedded in a scene.
Conclusion
The clean way to integrate a streaming AI avatar into Unity is to treat it as a session-based media consumer: provision the conversation on the backend, keep secrets off the client, and render the avatar as a remote realtime video surface. That architecture scales from a single NPC prototype to a game with many conversational characters without forcing Unity to own the entire voice stack.
If you want to implement this yourself, start with the docs, wire up a minimal session lifecycle, and test the end-to-end latency before you optimize anything else. The quickest path is usually to build one NPC with a single backend session and get the media/rendering loop stable first. From there, you can expand into interrupt handling, per-character prompts, and richer gameplay state.
For integration details, session management, and API shape, go to docs.protoface.com.