Introduction

Designing a multi-user realtime avatar system is mostly a systems problem, not a rendering problem. The hard parts are concurrency, session isolation, media fan-out, latency control, and making sure each participant sees the right avatar state at the right time. Unreal Engine is a reasonable choice when you need a high-fidelity front end, but once you move beyond a single local demo, you have to treat the avatar as a networked media service with a rendering client attached.

This post walks through a practical architecture for building that system in Unreal Engine. By the end, you should have a clear model for how to separate avatar state from per-user sessions, how to keep lip sync and speech aligned under load, and how to scale to many concurrent viewers without turning the game thread into a bottleneck.

Start with the right concurrency model

The biggest mistake is to think of “one avatar” as “one Unreal actor.” For a single-player prototype, that works. For multi-user realtime delivery, it becomes a coupling point for everything: audio input, speech synthesis, animation, network transport, and per-user permissions. Instead, split the system into three layers:

• Avatar definition: static metadata, visual assets, voice characteristics, and behavior prompts.

• Session: a live conversational instance with its own audio stream, generated speech, lip-sync timeline, and state machine.

• Presentation: the Unreal client or clients that subscribe to a session and render it.

This separation matters because concurrency is not just “how many users are online.” It is “how many live sessions exist at once,” “how many viewers subscribe to a session,” and “how much media each session generates per second.” You want to scale each of those independently.

A good internal mental model is:

• One avatar definition can back many concurrent sessions.

• One session can have one primary speaker and many passive viewers.

• Each viewer should consume the same synchronized timeline, not independently regenerate it.

That last point is critical. If every client re-runs speech timing or viseme generation locally, drift accumulates and the face will not stay in sync. The server should be authoritative for the media timeline, and clients should render from that timeline with small, bounded interpolation.

Media pipeline: keep audio, speech, and facial motion on one clock

Realtime avatars are usually driven by a chain like this:

1. User audio enters the agent pipeline.

2. Speech recognition, tool use, or LLM reasoning produces text or a response plan.

3. TTS generates speech audio.

4. Lip-sync or viseme generation maps phonemes to facial motion.

5. The client renders the avatar with a small amount of buffer for jitter tolerance.

The main engineering requirement is to keep all of those stages referenced to the same timeline. That means you should timestamp generated chunks, maintain sequence numbers, and treat playback as a stream rather than a series of independent clips. For multi-user systems, this lets you avoid per-viewer divergence.

On Unreal, the practical approach is to ingest the avatar as a media source or a video texture with synchronized audio. Let Unreal do what it is good at: compositing, scene layout, post-processing, and camera work. Do not push speech scheduling or lip-sync inference into Blueprint logic on every client unless you have a very specific reason. Keep the heavy media work off the game thread.

Under load, there are a few common failure modes:

• Audio underflow: the client cannot buffer enough audio, so playback stutters and the face drifts.

• Timeline skew: video and audio arrive with different jitter patterns and are rendered independently.

• Head-of-line blocking: one slow avatar session monopolizes CPU or encoder resources for other sessions.

• Fan-out amplification: every viewer causes a full duplicate encode or transform path.

To avoid these, buffer just enough to absorb network jitter, but not enough to make conversation feel sluggish. A few hundred milliseconds is often the right order of magnitude, depending on the transport and the quality tier you choose. Keep the session authoritative, and keep the client dumb enough that it can recover cleanly after reconnects.

Unreal Engine architecture for high concurrency

In Unreal, treat each remote avatar stream as a lightweight presentation object rather than a fully simulated character. That means:

• Use a manager subsystem to track live avatar sessions and subscriptions.

• Represent each session with an object that owns connection state, buffering, and render targets.

• Keep per-user UI, camera framing, and input handling outside the avatar transport path.

If you have many simultaneous viewers, the main scaling question is whether viewers are only consuming the stream or also sending input back. Read-only viewers can often be handled with simple subscription semantics. Interactive users need identity, permissions, and session-scoped event routing.

A practical pattern is to use a central session registry on the server, then let Unreal subscribe to only the sessions it needs. The server decides who may attach, which instructions apply, and what media quality is appropriate. This keeps the client code focused on rendering and avoids a combinatorial explosion of avatar-specific logic in the game layer.

For performance, push any expensive video or audio handling into dedicated worker threads or external services. In Unreal, your game thread should mostly react to new frames, update materials, and handle local interaction. If you are decoding media on the client, isolate that work so a slow decode path does not stall simulation.

State, permissions, and per-user isolation

Multi-user systems fail when a session boundary is ambiguous. If two users connect to the same avatar, you need deterministic rules for what is shared and what is private:

• Shared: the avatar identity, visual assets, and canonical speech output.

• Session-specific: conversation context, user-specific instructions, and access rights.

• User-specific: UI state, authentication, and any private data surfaced by the agent.

This is especially important if the avatar can act as a customer-support or sales agent. One user’s conversation should not bleed into another user’s session, even if they are viewing the same visual avatar. The easiest way to preserve that isolation is to assign each live interaction a session ID and treat that ID as the unit of routing, logging, and billing.

When you add presence, occupancy, or queuing, keep those concerns separate from the media session itself. A queue can decide when a user gets a turn; the media session should only care that a valid participant has been attached.

How Protoface fits without taking over your stack

This is the layer where Protoface is useful: it gives you the avatar/session side of the problem so you can focus on Unreal integration and product logic instead of building media orchestration from scratch. If your Unreal app is part of a voice agent or conversational experience, you can create and manage avatars and sessions through the REST API or the Python SDK, then connect the resulting media stream to your renderer. The API is authenticated with bearer API keys, and the docs show the exact request shapes.

A minimal REST flow looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

And a Python control path is similarly straightforward:

from protoface import Client

)

from protoface import Client

)

from protoface import Client

)

If you are already using LiveKit for realtime voice, the LiveKit plugin path is the cleanest integration point. It drops the avatar into the agent pipeline so the conversational model gains a synchronized talking face without you having to rebuild the media plumbing yourself. For teams standardizing on Pipecat, there is also a dedicated integration guide and package on PyPI, but the architectural idea is the same: keep the agent logic, transport, and avatar rendering loosely coupled.

Operational concerns: load, quality tiers, and failure handling

At high concurrency, operational detail matters more than the initial architecture. A few things to plan for early:

• Backpressure: if session creation spikes, queue or reject cleanly rather than letting Unreal or the media layer thrash.

• Reconnect behavior: clients should be able to reattach to an existing session without restarting the conversation.

• Quality controls: higher quality should cost more CPU, bandwidth, or latency; choose tiers intentionally based on the user experience you need.

• Observability: log session IDs, frame drops, audio underruns, and attach/detach events so you can debug concurrency issues.

One useful rule: if a failure happens, degrade presentation before you degrade conversation state. A brief video glitch is better than losing the interaction entirely.

Also, keep the Unreal client resilient to partial data. It should be able to handle missing frames, delayed audio, and mid-session reconnects. Your render code should prefer “show the last good frame and advance when new synchronized data arrives” over trying to guess the next state locally.

Conclusion

A multi-user realtime avatar system is easiest to build when you treat it like a distributed media service with a rendering client, not like a character prefab. Separate avatar definition from live session state, keep audio and facial motion on one authoritative timeline, and make Unreal responsible for rendering rather than orchestration.

If you want a managed avatar/session layer while keeping control of your own app architecture, start with the docs at docs.protoface.com and the quickstarts in the GitHub examples. The details will vary by transport and agent stack, but the core design stays the same: authoritative sessions, bounded client buffers, and strict isolation between users.