Introduction

Embedding a realtime AI avatar into Unreal Engine is mostly a systems problem, not a “render a face” problem. You need a low-latency media path, a clean separation between speech generation and avatar rendering, and a session model that can survive network jitter without making the avatar feel detached from the conversation.

This post walks through a practical architecture for doing that with LiveKit as the transport layer and a realtime avatar service as the video face source. By the end, you should be able to reason about the full path from user audio to agent response to synchronized talking video, and understand where to put the Unreal-side glue code.

Start with the media topology

For a conversational avatar, think in terms of three concurrent streams:

• Inbound audio from the user to the voice agent.

• Agent audio generated from the LLM/TTS pipeline.

• Avatar video driven by the same utterances, lip-synced to the agent audio.

In a typical LiveKit-based deployment, the user joins a room, the agent joins the same room as a participant, and the avatar video is attached to that agent’s response lifecycle. The important constraint is that audio and video must be generated from the same conversational turn, not from separate and independently timed requests. If they drift, the face will look “late” even if the media is technically smooth.

Unreal Engine should not be responsible for generating the avatar. It should treat the avatar as a realtime remote video source, render it on a material, and keep its input/output path as simple as possible. That keeps the engine focused on presentation, scene logic, and gameplay integration instead of media orchestration.

Use a single session boundary for conversation state

The cleanest design is to create one backend session per conversation. That session owns:

• the voice agent identity in LiveKit,

• the avatar instance or avatar session,

• the mapping from transcript turns to media output,

• and any usage/accounting metadata you care about.

This matters because the avatar should react to the same conversational state as the agent. If the model interrupts itself, retries a response, or changes the utterance mid-stream, the video pipeline needs to follow that same turn boundary. Don’t key avatar playback off “some text arrived”; key it off the finalized response segment or whatever turn event your agent stack exposes.

A useful mental model is:

1. User audio arrives in LiveKit.

2. The agent produces text and/or audio for the reply.

3. The avatar service turns that response into a synchronized talking face.

4. Unreal receives the video stream and renders it as the character face.

That separation keeps the system debuggable. If the face is wrong, you can inspect the avatar/session layer independently of the Unreal rendering layer.

Unreal integration: keep the engine side thin

In Unreal, the avatar should be treated like any other remote media source: you connect to the stream, decode frames, and hand them to a texture pipeline. The implementation details depend on your plugin stack, but the design goal is the same:

• avoid blocking the game thread on network or decode work,

• buffer only a small amount of video to hide jitter,

• and expose explicit state for connected, degraded, reconnecting, and disconnected.

There are a few practical concerns that show up quickly in production:

• Lip sync is only as good as your audio timing. If the agent audio has variable buffering, the avatar should follow that same timing source rather than an independent clock.

• Frame drops are preferable to latency buildup. In a conversational UI, stale video is worse than occasional frame loss.

• Resolution matters less than motion consistency. A stable 720p face with correct timing usually feels better than a sharper stream with uneven cadence.

• Disconnect handling needs to be explicit. When the session ends, clear the texture and transition cleanly; don’t let the last frame hang forever unless that is a deliberate UX choice.

If you are building the Unreal-side bridge in C++, model it as a state machine around the stream lifecycle. Typical states are: idle, connecting, streaming, recovering, and ended. That makes retries, room rejoin behavior, and graceful teardown much easier to reason about.

Provision the avatar and session from your backend

Do not create realtime avatar sessions from the client app if you can avoid it. Session creation is a backend concern because it usually needs credentials, policy checks, and lifecycle bookkeeping. The frontend or Unreal client should receive only the minimum connection data required to join.

A common backend flow is:

1. Generate or select the avatar for the request.

2. Create a realtime session.

3. Return the session connection details to the client or agent process.

4. Start the LiveKit conversation and attach the avatar stream.

Here is a minimal Python example for the control plane. Exact field names may differ; check the docs for the current schema.

import os

print(session.join_url)

import os

print(session.join_url)

import os

print(session.join_url)

For debugging or automation, the REST API is also straightforward to call directly:

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

The exact request shape depends on the endpoint you use, but the operating principle is the same: create the avatar/session on the server, then hand a short-lived result to the client or agent runtime.

Where LiveKit fits in the pipeline

LiveKit is a good fit here because it gives you the transport primitives you need for realtime voice: room membership, participant tracks, and an ecosystem for voice agents. Your voice agent can live in the same room as the user, while the avatar stream is synchronized to the agent’s response flow. That means the agent can hear the user, think, speak, and animate its face without you stitching together separate RTC systems.

For developers already using LiveKit Agents, the integration path is especially simple: drop the avatar into the agent process so the agent’s spoken turns are mirrored by a synchronized talking face. The plugin approach is valuable because it keeps the avatar logic close to the voice logic, which is where timing decisions are already being made.

If you want a concrete starting point, the LiveKit plugin is published as pipecat-protoface, and the related example repository is useful when you want to see how the pieces are wired together in practice. For Pipecat-based stacks, the integration guide is the right reference.

The design trade-off is that tighter coupling improves sync but reduces flexibility. If your agent runtime, TTS provider, and avatar service all need to agree on turn boundaries, keep that orchestration in one backend service rather than spreading it across the client, the Unreal app, and a serverless function chain.

Operational concerns you should plan for

Realtime avatars are sensitive to the same issues that affect any low-latency media system, but they show up as user-visible “uncanniness” instead of obvious errors.

Network jitter: buffer enough to smooth short spikes, but not enough to make the face feel detached from the conversation. In practice, small controlled latency is better than oscillating latency.

Rate limits and session scope: if you expose avatar creation directly to end users, you need policy controls. Customer-facing embeds typically need parent-origin allowlists, per-embed instructions, and hard limits on duration and abuse. Even for Unreal, if the client can initiate sessions, make sure your backend enforces quotas.

Observability: log session IDs, room IDs, avatar IDs, and agent turn IDs together. That makes it possible to answer the question “why did the face lag behind the voice on this turn?” without guessing.

Fallback behavior: decide what the user sees when the stream fails. A static portrait, a neutral idle animation, or a retry state are all better than a black rectangle with no explanation.

Content separation: keep policy/identity decisions in the backend. Unreal should render and react, not authorize.

How Protoface fits this architecture

This is the kind of pipeline Protoface is built for: create the avatar and session server-side, connect your LiveKit voice agent, and let the avatar track the agent’s turn timing instead of inventing a parallel media path. If you are using Python for orchestration, the SDK is the quickest way to provision sessions and wire them into your agent flow; if you are already in a LiveKit/voice-agent stack, the plugin route keeps the integration small and local to the agent process. The public docs at docs.protoface.com are the place to confirm the current API shapes and integration details.

Conclusion

The core pattern is simple: treat the avatar as a realtime media participant, not a UI widget. Keep session creation on the backend, tie the avatar to the same conversational turn state as the voice agent, and make Unreal responsible only for joining, rendering, and handling stream lifecycle cleanly.

If you are implementing this for a production app, start with a backend session flow, then connect a LiveKit agent, then verify lip sync and reconnect behavior in Unreal before you polish the visuals. Once the media path is stable, the rest is mostly UX work.

For implementation details, examples, and current API contracts, start with the docs at docs.protoface.com.