Introduction

If you are building a healthcare intake flow, the hard part is rarely “can the model answer questions?” It is usually orchestration: capture speech reliably, keep the conversation state aligned with the agent, make the experience feel human enough to reduce friction, and do all of that without turning your app into a pile of glue code. Adding a talking avatar helps here because patients respond better when the system feels like one coherent conversational surface instead of a disembodied voice prompt.

This post shows how to integrate a realtime video face into an Unreal Engine experience that is driven by a LiveKit voice agent. By the end, you should understand the moving pieces: how the voice stream and avatar video stay synchronized, where Unreal Engine fits, what to watch out for in a realtime intake workflow, and how to wire Protoface into the path without exposing secrets in the client.

Architecture: separate the agent, the avatar, and the renderer

The cleanest mental model is to treat the voice agent and the avatar as separate realtime services that share one conversational session.

• LiveKit agent: handles audio I/O, turn-taking, and whatever LLM or tool workflow powers the intake conversation.

• Avatar service: produces lip-synced talking video from the agent’s speech or transcript output.

• Unreal Engine client: renders the avatar video and optionally handles the surrounding UI, camera placement, and environmental context.

In practice, the avatar stream is usually just another realtime media source. The important thing is that it is time-aligned with the spoken audio. If the patient hears “What medications are you taking?” and the avatar mouth starts moving a beat later, the whole interaction feels off. Synchronization matters more than photorealism in this setting.

For healthcare intake, you also want low operational complexity. Do not push API keys into the Unreal client. Keep sensitive session creation on the server, and let the client join with short-lived credentials or a session token derived from your backend.

Getting the voice agent to emit a synchronized avatar

If you already have a LiveKit voice agent, the quickest path is to drop the avatar plugin into the agent process. The plugin attaches a Protoface avatar to the agent session so the agent can speak with a synchronized video face, rather than forcing you to run a separate video pipeline.

The exact setup depends on your agent framework, but the shape is typically the same: initialize the avatar service with your server-side credentials, create or attach to a session, and stream the agent’s speech through it. Keep the live media path on the server; only the rendered output goes to the Unreal client.

# illustrative only; exact fields and setup live in the docs

# illustrative only; exact fields and setup live in the docs

# illustrative only; exact fields and setup live in the docs

The useful bit here is not the exact method names; it is the separation of concerns. Your agent should keep handling conversation state, redaction, tool calls, and clinical routing. The avatar layer should only care about making the spoken response visually believable.

Wiring Unreal Engine into the realtime path

Unreal Engine is a good fit when the avatar is part of a larger interactive scene: a virtual front desk, a clinic kiosk, a triage room, or a branded patient onboarding experience. You typically embed or consume the avatar video stream in a widget, plane, or media texture, then place that output inside the 3D scene like any other realtime visual.

From an engine perspective, the integration points you need are straightforward:

1. Start the intake session on your backend.

2. Return a short-lived session token or join parameters to the Unreal client.

3. Connect the client to the media/session transport.

4. Render the avatar video into the scene and route the microphone input back to the agent if the user is speaking.

There are a few practical gotchas:

• Latency budget: once round-trip speech latency gets too high, patients will interrupt more and trust less. Keep inference, transport, and rendering paths short.

• Aspect ratio and compositing: avatar video often looks better if you treat it as a dedicated surface rather than trying to chroma-key it into a generic UI layer.

• Audio focus: avoid feedback loops when the agent and the avatar are both active in a space with speakers and a microphone.

• Session teardown: healthcare workflows often have strict idle and retention rules. Make sure sessions end deterministically when the intake is complete or abandoned.

If you are rendering the avatar inside a UI overlay, keep the scene stable. Fast camera moves and aggressive post-processing can make a talking face feel uncanny even when lip sync is correct. For intake, consistency is usually more important than cinematic presentation.

Session lifecycle and security for healthcare workflows

In a patient-facing application, session management is not just a convenience issue; it is part of your security boundary. The browser or game client should never hold long-lived API credentials. Instead, have your backend do the privileged work: create an avatar session, apply any per-session instructions, and mint the limited data the client needs to join.

That pattern also makes it easier to apply auditability and policy. For example, you can:

• create one intake session per patient encounter,

• bind the session to a specific clinic location or tenant,

• expire the session after a fixed duration,

• log which avatar configuration and instructions were used.

A minimal server-side creation flow via the REST API looks like this:

curl -X POST https://api.protoface.com/...
}'

curl -X POST https://api.protoface.com/...
}'

curl -X POST https://api.protoface.com/...
}'

The actual request body and session fields are documented in the docs; treat the snippet above as the shape of the interaction, not a copy-paste contract. The key point is that session creation belongs on your trusted backend, not in Unreal.

Why this works well for intake specifically

Healthcare intake is structured, but not fully scripted. You usually have a fixed set of fields to collect, conditional follow-ups, and a need to hand off if the patient is confused, distressed, or outside the expected flow. A realtime avatar helps because it preserves the conversational rhythm of a human interaction while still being driven by deterministic backend logic.

That means your agent can:

• ask a question,

• wait for a complete answer,

• extract structured data,

• confirm ambiguity,

• and escalate when confidence is low.

The avatar is not the decision-maker; it is the presentation layer for the agent. That distinction matters when you need to swap providers, tune model behavior, or change your clinical workflow without rewriting the Unreal client.

One practical design choice: keep the conversational logic in a separate service from the rendering client. Unreal should know how to display the session and maybe handle local UX, but it should not own the intake policy. That keeps state transitions testable and makes it easier to reproduce bugs in conversation timing or turn-taking.

Using the Python SDK for backend orchestration

If you prefer to manage avatars and sessions from application code rather than raw HTTP, the Python SDK is the most direct option. It is a good fit for a clinic backend that already creates patient sessions, initializes LiveKit agents, or issues temporary join tokens.

from protoface import Client

)

from protoface import Client

)

from protoface import Client

)

Again, treat this as illustrative. The important engineering pattern is to keep avatar/session creation inside your trusted control plane, then pass only the minimum necessary connection data to Unreal. That gives you a clear place to enforce authorization, per-tenant quotas, and retention policy.

How Protoface fits without adding browser or client risk

The most relevant surface for this use case is the LiveKit plugin, because it lets an existing voice agent gain a synchronized talking face without rebuilding the media stack. If your intake bot already speaks through LiveKit, this is the least disruptive integration point: the agent keeps working as-is, and the avatar becomes an attached visual representation of the same realtime session. The docs and quickstarts are the right place to verify the exact agent hooks and session fields before wiring it into production.

If you are evaluating the integration path, start with the plugin repository and the platform docs:

• GitHub organization for the implementation surface and examples.

• Documentation for API contracts, session lifecycle, and setup details.

For teams already using Python on the backend, the SDK is useful for creating sessions during patient check-in, while Unreal consumes only the realtime media output. That is the boundary you want: privileged orchestration in your backend, realtime rendering in the client, and no API key leakage.

Conclusion

Integrating a realtime avatar into an Unreal Engine intake experience is mostly an exercise in clean separation: let the voice agent do the conversation, let the avatar service handle synchronized video, and let Unreal focus on presentation and local interaction. For healthcare specifically, keep session creation and policy on the server, use short-lived client access, and be strict about lifecycle cleanup.

If you want to implement this yourself, start by reading the docs, then wire up the agent-side plugin or backend SDK flow in a small prototype before you touch the Unreal scene. Once the media path is stable, the rest is just UX work: placement, timing, and making sure the conversation feels calm and reliable.

For setup details and the latest API shape, see docs.protoface.com.