How to Build a Realtime AI Game NPC Avatar in Unity with WebRTC
Build a realtime AI NPC avatar in Unity with WebRTC: low-latency audio/video, lip sync, session state, and reconnect handling.
Introduction
If you want a game NPC to feel present, a static portrait is not enough. Players notice when speech, mouth motion, timing, and turn-taking are disconnected. The usual failure mode is simple: the model speaks, the UI animates some generic idle loop, and the character never quite lands as a real conversational entity.
This post walks through the practical path to building a realtime AI NPC avatar in Unity using WebRTC. By the end, you should understand the architecture, how the media pipeline fits together, what to synchronize, and where to draw the line between “good enough for a prototype” and “stable enough for production.”
I’ll also show where Protoface fits when you need a realtime talking face for a voice agent or game NPC, without pushing avatar rendering and lip-sync into your own backend.
What “realtime AI NPC avatar” actually means
For a game NPC, the hard part is not just generating text. It’s the end-to-end loop:
the player speaks or types a message,
your agent produces a response,
audio is streamed back with low latency,
the avatar face stays synchronized to that audio,
the NPC can be interrupted, react, and continue naturally.
WebRTC is a good fit because it gives you low-latency media transport, jitter buffering, and a model that already matches “live conversation” better than file-based playback or polling an HTTP endpoint. Unity is the right client if the NPC lives inside a 3D scene, because you can render the avatar as a textured quad, a UI element, or a world-space object, then drive game state from the same conversation session.
The key point: don’t think of this as “send text to an API and display video.” Think of it as a realtime session with a bidirectional media channel. The avatar is one participant in that session.
Architecture: separate conversation, media, and game state
A clean implementation usually has three layers:
Conversation layer — your LLM, tool calls, memory, and policy.
Media layer — WebRTC audio/video transport and avatar rendering.
Game layer — Unity scene logic, quest state, triggers, and animations.
Keep these decoupled. The conversation layer should not know about Unity objects. The game layer should not care how lip sync is generated. The media layer should only care about transport, frame delivery, and session state.
That separation matters when you add features like:
interruptions while the NPC is speaking,
branching dialogue based on player location or inventory,
multiple NPCs sharing the same conversation service,
fallback modes when bandwidth drops or the avatar disconnects.
WebRTC in Unity: the practical setup
Unity does not natively solve every WebRTC detail for you. In practice, you need a plugin or package that can establish a peer connection, handle signaling, and expose the remote video track as a texture or render target. The exact package choice varies, but the mechanics are the same:
create a peer connection,
exchange signaling metadata with your service,
attach remote audio and video tracks,
render the received video into the NPC material or UI,
keep the connection alive and reconnect cleanly on failure.
For a game NPC, the remote video should usually be treated as an animated facial layer, not as a full-screen video call. That means you’ll often map the stream to:
a UI panel for a dialogue portrait,
a masked face region on a 3D mesh, or
a stylized “hologram” surface in the world.
Two implementation details matter more than people expect:
First, latency budget. If your turn-taking feels slow, the NPC feels fake. WebRTC keeps transport delay low, but your model inference, tool execution, and any server-side avatar generation still need to fit the interaction budget.
Second, interruption handling. A good NPC can stop mid-sentence when the player interrupts. That means your Unity client should treat audio/video as streamable, cancelable media, not as a single monolithic response.
Synchronizing voice, lips, and animation
Realism depends on synchronization more than resolution. A sharp 720p avatar with bad timing looks worse than a simple avatar that matches speech correctly. You want three things aligned:
Audio onset — when speech begins, the mouth should start moving immediately.
Phoneme timing — mouth shapes should track the rhythm of speech closely enough to feel natural.
Expression timing — blinks, head motion, and emotion cues should not drift away from the spoken content.
In a Unity NPC, the easiest mistake is to drive one animation state machine from text generation events and a different one from audio playback events. That creates drift. Instead, use the audio stream as the source of truth for lip sync, and layer higher-level expressions on top of that.
If you are using a stylized character rather than a photoreal face, keep the animation system honest. The less the model resembles a human face, the more forgiving the viewer is about subtle mismatch; but the more exaggerated the style, the more important it is that speech cadence and motion agree.
Session management and game integration
For games, session lifecycle is as important as media transport. A realtime NPC should have explicit states such as:
idle — available, listening, not speaking,
thinking — request in flight,
speaking — audio and video streaming,
interrupted — stopped mid-response,
disconnected — transport lost or session expired.
Drive those states from your conversation service and transport callbacks, not from arbitrary Unity timers. That makes it easier to show subtitles, trigger quest events, or blend the NPC into a combat scene without guessing whether the agent is still speaking.
When I see teams struggle here, it’s usually because they treat the avatar as an isolated widget. It works in a demo, then breaks when the NPC has to persist across scenes, reconnect after a match load, or resume a dialogue tree after a timeout. The fix is to keep the session identifier and dialogue state outside the scene graph, then reconnect the media surface to that session when the scene is recreated.
Where Protoface fits
If you do not want to build the avatar-media side yourself, Protoface gives you a realtime avatar API designed for this exact “voice agent with a face” problem. The most relevant surface for a Unity game is the realtime session layer exposed through the REST API at docs.protoface.com, plus the Python SDK and quickstarts when you want to prototype the backend flow before wiring Unity to it.
A typical backend flow is:
create an avatar or reuse an existing one,
open a realtime session,
hand the session connection details to your game client,
connect Unity over WebRTC,
stream audio/video while your agent talks.
For example, you might create a session from your server with the REST API, keeping your API key off the client entirely:
The exact request fields depend on the session shape you choose, so treat that as illustrative rather than copy-paste complete. The important part is the boundary: your game client connects to a short-lived session, not to your private API key.
If you are building your agent in Python first, the SDK is the fastest way to validate the conversation loop before you touch Unity. Then, once the backend behavior is stable, you can connect the game client to the same session model and focus on rendering and transport.
Implementation gotchas that matter in production
A few issues come up repeatedly:
Do not expose long-lived credentials in the client. Use a backend to mint session data.
Handle reconnects explicitly. WebRTC will recover from some failures, but not all of them.
Budget for bandwidth variability. Avatars need graceful degradation, not just one “best” stream format.
Keep NPC behavior deterministic enough for gameplay. Free-form generation is fine, but quest-critical logic should not depend on a vague model response.
Test with real latency. Localhost demos hide the cost of network hops, transcoding, and model latency.
If you want a clean separation of concerns, let the game own authority over state, and let the avatar own presentation. The agent can suggest actions, but the game decides what is allowed.
Conclusion
Building a realtime AI NPC avatar in Unity is mostly an exercise in systems design: low-latency media transport, clean session management, and careful synchronization between speech and animation. WebRTC gives you the transport model you need. Unity gives you the rendering and scene integration. The rest is keeping your conversation layer, media layer, and game logic separated enough that the whole thing remains debuggable.
If you want a quicker path to a synchronized talking face for a voice agent or NPC, start with the docs, prototype the backend session flow, and then wire that into Unity once the agent behavior is stable. The practical next step is to read the integration notes at docs.protoface.com and build a small end-to-end demo before you commit to a larger character pipeline.