How to Integrate a Realtime Talking Avatar into Unreal Engine with WebRTC and LiveKit
Integrate a realtime talking avatar in Unreal Engine with WebRTC, LiveKit, and low-latency voice agent media sync.
Introduction
If you are building a voice agent in Unreal Engine, the missing piece is usually the face. Audio alone works for a prototype, but the moment you want something that feels present in a 3D scene, you need synchronized video, low latency transport, and a clean way to keep speech and facial motion aligned. In practice, that means combining WebRTC for realtime media, a voice agent for speech and turn-taking, and an avatar service that can generate a talking face fast enough to stay believable.
This post walks through the architecture and integration path I would use in Unreal Engine. By the end, you should understand how to:
connect Unreal to a LiveKit media session over WebRTC,
attach a realtime talking avatar to a voice agent,
keep lip sync and audio timing stable under network jitter, and
choose the right integration surface depending on whether you are building a game NPC, support bot, or live interactive experience.
The examples below are intentionally compact. They are meant to show the shape of the integration, not to replace the docs.
How the realtime pipeline fits together
The clean mental model is: Unreal is the client, LiveKit is the realtime media plane, and the avatar service provides the visual stream that corresponds to the agent's speech. WebRTC handles the low-latency transport for audio and video. The agent listens to the user, generates a response, and the avatar renders that response as a talking face stream.
That split matters because the avatar itself is not a local animation clip. It is a realtime media producer. Your application is responsible for session setup, authentication, and media plumbing; the avatar service is responsible for generating the synchronized face frames from the agent's output.
In a typical Unreal setup, you have two tracks to manage:
Inbound audio from the user into the agent.
Outbound media from the agent/avatar back into Unreal as a remote video track, and often audio as well.
The integration challenge is not “how do I show a video?” It is “how do I keep the agent’s turn-taking, speech, and facial motion aligned while handling reconnects, packet loss, and varying device performance?”
WebRTC and LiveKit in Unreal Engine
Unreal is not usually the place where you want to hand-roll transport. WebRTC gives you the right primitives: peer connections, ICE negotiation, jitter buffering, and adaptive behavior under changing network conditions. LiveKit layers the room/session abstraction on top, which is helpful when your agent is just one participant in a larger interactive environment.
On the Unreal side, the practical implementation is usually:
join a room using a short-lived access token,
subscribe to the agent's published video track,
render that track onto a UI widget, plane, or character surface, and
optionally route audio to an in-world speaker or voice component.
If you are already using a LiveKit Unreal integration, the avatar is just another remote participant from the engine's perspective. That is a good thing: keep the media stack generic and let the avatar service own the face generation logic.
Session lifecycle: auth, joining, and teardown
Before media flows, you need a session. The important design choice is to keep API keys off the client. For anything Unreal ships to players or end users, your backend should mint temporary credentials or session tokens and hand those to the game client.
At a minimum, the flow looks like this:
Your backend authenticates with the avatar API.
Your backend creates or configures an avatar session.
Your backend issues a short-lived LiveKit token for the Unreal client.
Unreal joins the room and subscribes to the avatar tracks.
When the interaction ends, the backend closes or expires the session.
That separation is important for security and for operational control. If you later want per-user limits, custom instructions, or per-session voice settings, you want those controls enforced server-side, not embedded in the client build.
From there, your backend would pass the Unreal client whatever connection material it needs for the room. The exact payloads vary by setup, but the separation of responsibilities does not.
Attaching the avatar to a LiveKit voice agent
The fastest path to a credible talking avatar is to let the voice agent own the conversation and use a plugin to attach the avatar as the visual output of that agent. In practice, that means you do not have to stitch speech, visemes, and frame timing together yourself. The agent generates audio; the plugin turns that agent output into synchronized video.
If you are already running your agent on LiveKit, the plugin route is the lowest-friction integration. The Python package is published as livekit-plugins-protoface, and the GitHub repo has the corresponding examples and implementation details.
Conceptually, the plugin takes care of the “talking face” part while your agent handles the language model, speech synthesis, and turn-taking. That reduces the amount of custom glue you need in Unreal: the engine only needs to render the remote media track it receives from LiveKit.
If you are not using the LiveKit agent plugin, you can still use the same underlying session model from your backend and subscribe to the resulting media track in Unreal. The plugin just removes a lot of the tedious synchronisation work on the agent side.
Practical rendering choices in Unreal
Once the avatar track is arriving in Unreal, you have a few implementation options. The right choice depends on how “diegetic” the face should feel in the scene.
UI overlay: simplest to ship, good for support agents or conversational HUDs.
In-world screen or monitor: useful when the avatar lives on a device in the environment.
Character surface or billboard: suitable if you want the avatar to feel physically present.
The key technical constraint is texture update cost. A remote video feed rendered into a material or UI texture can become expensive if you are not careful about resolution, frame rate, and copy paths. Keep the avatar stream at the quality tier you actually need. For many in-game uses, a slightly lower video resolution is perfectly acceptable if the motion stays smooth and latency stays low.
Also pay attention to audio-video sync. WebRTC will generally keep the streams aligned, but your rendering path can reintroduce drift if you buffer video excessively or if your UI tick rate is inconsistent. Avoid custom buffering unless you have a concrete reason; let the media stack do the timing work.
Operational gotchas: latency, jitter, and failure modes
Realtime avatars are forgiving visually, but only up to a point. A few things usually cause trouble:
Long agent think time: users notice dead air before they notice modest video compression artifacts.
Overbuffering: adding too much local latency to smooth playback makes the avatar feel disconnected from the speech.
Unbounded reconnect behavior: if the network drops, you want a deliberate reconnect strategy, not a stale face frozen on screen.
Client-side secrets: never ship API keys in Unreal or expose them in browser code.
For debugging, I recommend treating the avatar session like any other media service: log join/leave events, media subscription state, and session IDs. When a report says “the face is behind the voice,” you want to know whether the problem is packet loss, rendering delay, agent latency, or session misconfiguration.
Where Protoface fits
This is where Protoface is genuinely useful: it gives you the avatar/session layer without forcing you to build your own realtime face pipeline. For Unreal + LiveKit specifically, the simplest pattern is usually to let your voice agent run in LiveKit, attach a Protoface avatar through the LiveKit plugin, and have Unreal subscribe to the resulting media track. If you prefer to create or manage sessions from your own backend, the REST API is available at docs.protoface.com, and the Python SDK is useful for automating avatar/session setup in test environments or orchestration code. If you want a reference implementation for the agent side, start with the plugin repo and the quickstarts linked from the project README.
Conclusion
The core integration pattern is straightforward once you separate concerns: Unreal handles rendering and game logic, LiveKit handles realtime transport, and the avatar service handles synchronized facial video for the agent. Keep API keys server-side, keep buffering minimal, and treat the avatar as a realtime media participant rather than a static asset.
If you are building this for the first time, start with a single voice agent in LiveKit, wire in the avatar on the backend, and render the remote track in a simple Unreal UI surface before you move to in-world placement. That gives you a working end-to-end path quickly and exposes timing issues early.
For implementation details, session schemas, and the current supported surfaces, the best next step is the documentation at docs.protoface.com.