Quickstart: Adding a Talking AI NPC to Unreal Engine with LiveKit
Quickstart for adding a talking AI NPC in Unreal Engine with LiveKit: stream voice, sync a video avatar, manage sessions, and render the face in-game.
Introduction
If you already have an Unreal Engine NPC that can move, animate, and speak, the last piece is usually the hardest: making the face feel alive while the character is talking. The UI-level solution is trivial; the real problem is synchronizing speech, lip motion, and the video or texture that represents the face so the result stays responsive under realtime constraints.
This post shows the practical path to adding a talking AI NPC to Unreal Engine using LiveKit for realtime audio transport and a video face layer for synchronized avatar rendering. By the end, you should understand where the audio pipeline ends, where the avatar pipeline begins, how to wire them together, and what to watch for when you’re shipping something that needs to feel interactive rather than pre-rendered.
The architecture: separate the voice loop from the face loop
For an NPC with a talking face, treat the system as two coupled realtime streams:
Voice path: player audio goes into your agent, the agent reasons and speaks back, and audio is delivered over LiveKit.
Face path: the avatar receives the agent’s speech signal and produces a synchronized talking video face, which you render in Unreal as a texture, media surface, or web view depending on your integration.
The important detail is that the face should not be driven by hand-authored mouth shapes or frame-by-frame animation if you want conversational timing. The avatar layer needs to follow the speech stream closely enough that phoneme timing, pauses, and turn-taking feel aligned. In practice, that means your agent orchestration should emit speech as a stream, not as a fully buffered blob, and the avatar renderer should be subscribed to the same turn state as the audio output.
Latency matters more than raw quality here. A face that is slightly lower resolution but reacts in under a few hundred milliseconds will feel much more convincing than a high-fidelity face that lags behind the speaker.
Unreal Engine integration: what actually changes in your project
In Unreal, the common pattern is to keep your NPC logic and your avatar rendering isolated:
Your gameplay code decides when the NPC should listen, speak, or idle.
A network layer connects Unreal to the realtime agent session.
The avatar video is rendered into an in-game surface used by the character mesh, UI, or a dedicated billboard.
If you are already using LiveKit for voice, the face should be attached to the same session lifecycle. That means when the agent joins a room, the avatar session starts with it; when the agent leaves, the avatar tears down cleanly; and when you interrupt the agent, both audio and face stop together.
That lifecycle alignment is what prevents the classic failure modes: speaking while the face is frozen, a face continuing to mouth words after the audio has stopped, or a new utterance starting before the previous visual turn has cleared.
Practical setup: start with the agent, then add the face
For a first pass, keep the Unreal side simple and use your existing LiveKit voice agent flow. Once the agent is producing speech reliably, attach Protoface at the agent layer rather than trying to make Unreal synthesize the avatar directly. That keeps the gameplay client thin and lets the backend own the realtime avatar session.
In Python, the shape usually looks like this: create or fetch an avatar, open a session, and hand that session to the component that is already producing agent speech. The exact request and response fields depend on the SDK version, so treat the following as a structural example rather than copy-paste production code.
Once the session exists, your LiveKit agent can stream speech into the avatar pipeline. The point is not the Python call itself; it is that the avatar is managed as a realtime session with the same operational model as the rest of your voice agent stack.
Using the LiveKit plugin to give the agent a face
If your NPC is already built on a LiveKit agent, the cleanest integration path is the LiveKit plugin published as livekit-plugins-protoface on PyPI. The plugin’s job is straightforward: it drops a Protoface avatar into the agent so the agent gains a synchronized talking video face without you having to invent a custom bridge between speech events and avatar rendering.
In code, this generally means adding the plugin to your agent setup and configuring the avatar/session parameters in the same place you define the agent’s voice behavior. The details depend on your agent framework and how you structure room joins, but the conceptual boundary is consistent: LiveKit handles room transport; the plugin handles face synchronization.
That division of responsibility is useful because it keeps the engine-specific code out of your gameplay loop. Unreal should care about when the NPC is speaking and what surface is displayed. The plugin should care about how the avatar session maps to the agent’s output.
Rendering the avatar in Unreal without overengineering it
There are several ways to bring the face into Unreal, and the right one depends on how much control you need over the render path:
Browser-like surface: easiest when the avatar is delivered as a web/video surface and you can place it in a widget or texture target.
Video texture pipeline: useful if you want the face embedded on an NPC mesh or an in-world display surface.
Dedicated overlay/UI: works well for dialogue scenes where the face is shown in a conversation panel rather than as a fully integrated character head.
Whichever route you choose, keep the networking side decoupled from the render side. The avatar session can reconnect or refresh independently of the Unreal scene as long as your code treats the session ID and connection state as source of truth.
One operational gotcha: don’t tie avatar startup to level load if the agent may need to appear mid-session. It is better to have a session manager that can instantiate, attach, detach, and reattach the avatar from gameplay events. That makes NPCs easier to stream in and out, and it avoids leaks when the player hops between scenes.
Where the avatar platform fits
This is where a managed avatar platform helps. Instead of building your own lip-sync timing, session management, and avatar delivery layer, you can create and manage avatars and sessions through the REST API or the Python SDK, then connect that session to your LiveKit agent. The API is authenticated with standard bearer API keys, and the dashboard gives you a place to inspect sessions, avatars, keys, and usage while you iterate. For the mechanics and field names, the docs are the source of truth: docs.protoface.com.
For example, creating a session via REST is the kind of flow you want when your Unreal backend is already orchestrating game state:
The exact endpoint shape and payload fields may vary, but the pattern stays the same: your game backend owns authorization and session creation; the client only consumes the resulting realtime media.
Gotchas that matter in production
A few issues show up repeatedly when teams move from a demo to a playable NPC:
Turn timing: if the agent’s speech is generated in large chunks, the face will feel delayed. Stream the turn if possible.
Interruptions: make sure “barge-in” stops both audio and facial motion. Otherwise the NPC will keep mouthing stale words.
Connection churn: treat reconnects as normal. Recreate or rebind the avatar session rather than assuming a single long-lived connection.
Quality tier: choose the lowest tier that still holds up in the camera distance you expect. In a game, the right answer is often scene-dependent.
State sync: the NPC’s dialogue state should be owned by one controller. If Unreal, the agent, and the avatar all independently decide when the NPC is “speaking,” you will get drift.
Also remember that the face is part of the interaction budget. If your game already spends 200 ms on game logic and 200 ms on voice transport, adding another 500 ms for avatar startup will be noticeable. Pre-warm the session if the NPC is likely to speak soon.
Conclusion
The implementation is not complicated once you separate the concerns: LiveKit handles realtime voice transport, the avatar layer handles synchronized facial output, and Unreal consumes the result as a renderable surface tied to gameplay state. If you keep the session lifecycle aligned across those pieces, the NPC feels responsive instead of bolted on.
For the detailed API shapes, SDK usage, and current integration notes, start with the docs at docs.protoface.com, and use the LiveKit plugin repository if you are wiring the face into an existing LiveKit agent. For Unreal-specific work, the same basic principle applies: keep the game client thin, keep the session state explicit, and let the realtime avatar layer do the part it is designed for.