Introduction

If you want a talking AI avatar inside Unreal Engine, the hard part is not “show video on a mesh.” The hard part is doing it without stalling the game thread while audio arrives, frames decode, lip sync updates, and the UI stays responsive. In practice that means treating the avatar as a realtime stream, not a static media asset: receive frames asynchronously, move them across threads safely, and hand the game thread only the minimal state it needs to render the current frame.

This post walks through a sane architecture for that pipeline. By the end, you should be able to reason about how to ingest a realtime avatar stream, avoid blocking Unreal’s game thread, and wire the data path so video, audio, and animation stay synchronized. I’ll also show where Protoface fits when you need a managed avatar backend rather than rolling your own streaming service.

What “blocking the game thread” usually means

Unreal’s game thread is responsible for gameplay logic, actor updates, and a lot of UI and Blueprint execution. If you do any of the following on that thread, you will eventually see frame hitches:

• Waiting on network I/O for a new video frame or audio chunk.

• Decoding video synchronously in response to an event.

• Uploading textures one frame at a time without batching or buffering.

• Doing per-frame JSON parsing or websocket handling directly inside Tick.

The fix is not exotic. Keep network, decode, and buffering work off the game thread. The game thread should only consume already-prepared frame data and update render resources in a predictable, bounded way.

Use a producer/consumer pipeline, not a pull-on-Tick loop

The cleanest shape is:

1. A background networking task receives avatar session events and media chunks.

2. A decoder or frame assembler turns those chunks into complete video frames.

3. A lock-light queue hands decoded frames to the render/game side.

4. The game thread swaps the newest available frame into a texture or media surface.

This is a classic producer/consumer setup. The important part is that the consumer never waits. If no frame is ready, render the previous one. If frames arrive faster than you can display them, drop stale frames instead of building latency.

For a talking avatar, latency matters more than perfect frame retention. You usually want the newest intelligible face, not an exact archive of every packet that came in.

Buffering strategy: keep it small and deterministic

Two buffers are usually enough:

• Network buffer for partial chunks and metadata.

• Frame buffer for complete decoded frames ready for upload.

Use bounded queues. Unbounded queues hide bugs until they become memory pressure or delayed speech. If the render side falls behind, overwrite old frames or drop intermediate ones. For avatars, a small amount of frame loss is better than building a 500 ms backlog.

Also keep audio and video aligned by timestamp, not by arrival order. Network jitter will reorder arrival times; your render logic should use the session clock or frame timestamps to decide what is current. If your avatar service emits lip-sync driven video, the lips should be treated as part of the media stream, not as a separate animation system unless you have a very specific reason to split them.

Updating Unreal textures without stalls

Once you have a decoded frame, the remaining job is to get pixel data onto the GPU without doing expensive work in the wrong place. The safe pattern is to cache the frame in CPU memory, then schedule a texture update from the game thread or render thread using Unreal’s async rendering path.

At a high level:

• Allocate a dynamic texture or render target sized for the avatar video.

• Keep a persistent CPU-side buffer for the latest frame.

• On the game thread, copy only the newest frame pointer or byte span, not the network packet data.

• Enqueue the actual GPU update so render submission stays non-blocking.

Here is a deliberately simplified example of the handoff shape in C++:

// Pseudocode: frame data is produced on a worker thread.

}

// Pseudocode: frame data is produced on a worker thread.

}

// Pseudocode: frame data is produced on a worker thread.

}

The exact Unreal API call you use for texture upload depends on whether you are targeting a UTexture2D, a render target, or a custom media pipeline. The architectural point stays the same: the game thread should schedule work, not perform it synchronously.

Don’t let audio and lip sync drift

A talking face is only believable if the mouth, voice, and frame cadence stay close enough together. There are three common failure modes:

• Video leads audio: the mouth starts moving before the sound arrives.

• Audio leads video: the voice is heard before the face reacts.

• Backpressure buildup: buffering adds so much latency that the avatar feels delayed even though it is technically synchronized.

The practical solution is to choose one clock as authoritative. In a realtime avatar system, that is usually the session timing from the service or the audio playback timeline in the client. Then:

• Buffer enough to smooth short network jitter.

• Drop stale video frames rather than trying to catch up linearly.

• Keep audio playback non-blocking, ideally with its own small ring buffer.

• Use timestamps to match frame display to the currently audible speech segment.

If you are already running a voice agent in Unreal, the avatar stream should be attached to the agent lifecycle, not driven as a separate “play video” feature. When the agent speaks, the avatar speaks. When the session pauses, both pause. That avoids edge cases where the face continues animating after the audio has stopped.

A realistic integration shape in Unreal

A good implementation usually looks like a custom component or subsystem with three responsibilities:

1. Open and maintain the avatar session connection.

2. Receive frame/audio events on worker threads.

3. Expose the latest rendered avatar state to materials, widgets, or mesh components.

Keep the component interface small. For example, your game code should probably only care about methods like Connect(), Disconnect(), and SetVoiceInput(), plus maybe an event for connection loss. Everything else can live behind the component boundary.

One useful rule: never allocate per frame on the game thread if you can avoid it. Reuse frame buffers, texture memory, and event objects. A talking avatar is a sustained realtime workload; garbage-like allocation patterns will show up as frame spikes long before they show up as memory issues.

Where Protoface fits

If your goal is to stream a synchronized talking face into Unreal, the avatar backend does not have to be your problem. Protoface provides the realtime avatar session layer, and its developer surfaces are designed around integration rather than manual media plumbing. For Unreal-specific work, the useful pattern is to keep your game client focused on transport and rendering, while the service handles avatar generation, lip-synced video, and session management.

If you are building this around a voice agent, one practical path is to start from your agent stack and attach the avatar through the LiveKit integration. Protoface publishes a LiveKit Agents plugin on PyPI; the corresponding examples in the quickstart repo are a good reference for session lifecycle and avatar attachment patterns. If you are instead driving sessions directly, the REST API and Python SDK let you create and manage avatars programmatically. Exact request fields and response shapes are in the docs, but the basic REST pattern looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

If you prefer Python, the SDK gives you the same shape programmatically:

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

From Unreal’s point of view, the session ID and stream endpoint are the important outputs. Everything else stays on the service side. The docs at docs.protoface.com cover the integration details and the supported quickstarts.

Common mistakes to avoid

• Doing websocket reads in Tick instead of a worker thread.

• Blocking on frame decode before returning control to the engine.

• Holding a mutex around texture upload or render submission.

• Rendering every received frame instead of the newest usable frame.

• Separating audio and video control paths so they drift under load.

If your avatar gets choppy under stress, the first thing to inspect is backlog, not raw bandwidth. Most “laggy avatar” bugs are really “we let stale frames pile up” bugs.

Conclusion

To stream a talking AI avatar into Unreal without blocking the game thread, treat the avatar as a realtime media pipeline: receive data on worker threads, decode off-thread, buffer lightly, and update render resources with only the latest frame. Keep synchronization timestamp-based, prefer dropping stale frames over accumulating latency, and make sure audio and video share a common session clock.

If you want a managed avatar backend instead of building the session layer yourself, start with the docs and a quickstart from the Protoface GitHub organization. The engineering goal stays the same either way: keep Unreal responsive, keep the media path asynchronous, and let the game thread do only the work it absolutely has to do.