Building a Realtime AI Healthcare Intake Avatar in Unreal Engine
Build a realtime AI healthcare intake avatar in Unreal Engine with voice sync, session state, and structured patient data capture.
Introduction
If you are building a healthcare intake flow, the hard part is rarely “can the model answer questions?” It is the integration problem: how do you collect structured patient information, keep the experience conversational, and present a face that feels responsive without adding a pile of bespoke video infrastructure?
This post shows how to build a realtime AI intake avatar in Unreal Engine that can listen, speak, and stay visually synchronized with the agent’s response stream. By the end, you should understand the architecture, the realtime constraints that matter, and how to connect a voice agent to a talking avatar without turning your game engine into a media server.
What “realtime avatar” actually means in this setup
There are three independent loops involved:
Audio input/output: microphone audio is streamed to your agent, and synthesized speech comes back as audio.
Agent orchestration: the LLM or dialogue system decides what to ask next, when to interrupt, and when to escalate.
Visual sync: mouth motion, gaze, and facial animation track the speech stream closely enough that the avatar looks like it is speaking the returned audio.
The important detail is that “lip sync” is not a postprocess on a saved video. In realtime systems, the avatar is driven from the live speech stream, so the face must be updated with low latency and tolerate mid-utterance changes. If your agent gets interrupted by the user, the avatar has to stop cleanly and transition without obvious lag.
Designing the intake flow for healthcare
For intake, the avatar is not there to be a general-purpose chatbot. It is there to keep the patient engaged while gathering a specific schema: identity, reason for visit, symptoms, meds, allergies, prior conditions, insurance details, and consent prompts. You want the conversation to feel natural, but the backend should still emit structured data that can be validated and stored.
A practical pattern is:
The voice agent asks one question at a time.
Responses are normalized into a small intake state machine.
The agent confirms ambiguous fields before advancing.
Anything clinically sensitive or uncertain is routed to a human.
This keeps the realtime experience simple and reduces failure modes. Do not try to make the avatar “understand everything” in one turn. For healthcare, narrowly scoped prompts and explicit field collection are safer and easier to debug.
Unreal Engine integration: keep media separate from gameplay
In Unreal, the biggest mistake is coupling avatar rendering to the rest of the game loop too tightly. Treat the avatar as a media surface with a small state interface:
connect / disconnect
start listening / stop listening
play speech / interrupt
receive state updates for speaking, idle, and error
Your UI can live in UMG or a 3D scene, but the transport and session logic should be isolated in a service layer. That makes it easier to handle reconnects and to swap backends later.
At a minimum, you need to handle:
Session lifecycle: create a session, attach the avatar, and clean it up when the interaction ends.
Latency budgeting: keep round trips low enough that the patient does not perceive dead air.
Interruption handling: if the user speaks, stop the current agent response and visually reflect the interruption immediately.
State recovery: if the network drops, re-establish the session without losing the intake transcript.
In practice, Unreal is consuming a realtime stream and painting the avatar, while your agent backend handles speech and dialogue. The engine should not be responsible for NLU, scheduling, or consent logic.
Minimal backend shape for the intake agent
Even if the avatar is the visible surface, your backend should own the business logic. A straightforward shape is:
an agent process that receives mic audio
a structured prompt or tool layer for intake fields
a session store keyed by patient or visit ID
an event log for auditability
When the agent collects a field, store both the raw text and the normalized value. For example, “I’m allergic to penicillin” should update the allergy list and preserve the original utterance. That makes downstream review and correction easier.
Also, be explicit about what the avatar should not do. In healthcare, the conversation layer should collect data and route, not diagnose. Make the prompt and the UI reflect that constraint.
Example: creating a session from Python
If you are orchestrating the flow from your backend, a Python SDK is the cleanest place to start. The exact request fields depend on your avatar and session configuration, but the shape is usually: authenticate, create a session, then pass the session details to your realtime client.
Use the session object as the handoff between your backend and the realtime frontend. In a real implementation you would store it with the patient visit, then stream the audio and state updates through your agent pipeline. Check the docs for the exact fields and supported session options at docs.protoface.com.
Example: REST API session creation
If you prefer to keep the orchestration outside the SDK, the REST API is straightforward. The main point is that API keys stay on the server, never in the browser.
This is the right layer for server-side control, webhook handling, and policy enforcement. For healthcare workflows, that matters because the server can decide whether a session should start, continue, or be terminated based on application state.
Where the Unreal avatar gets its sync from
The avatar itself does not need to know anything about your medical domain. It just needs a realtime speech stream and the control signals to animate correctly. That means the sync quality depends on three engineering choices:
Audio chunking: speech should be streamed in small enough pieces to keep latency down.
Prosody stability: if your TTS changes timing too much mid-utterance, the face will look off.
State transitions: speaking, listening, and idle states should be explicit and debounced.
For Unreal specifically, make the avatar component event-driven. Do not poll the backend every frame. Instead, update animation state on session events and audio callbacks. That keeps the frame loop deterministic and avoids fighting the engine’s render timing.
Protoface in the stack
This is where Protoface fits naturally: it provides the realtime avatar layer so you can keep your existing voice agent and focus on the intake logic. If your agent already lives in LiveKit, the quickstart examples show the general integration pattern, and the LiveKit plugin drops a synchronized talking face into the agent process. If you are driving sessions from your backend, the REST API and Python SDK are the more direct surfaces.
For Unreal, the practical benefit is that you do not have to build face streaming, session management, or avatar sync from scratch. Your game or app can consume the avatar as a managed realtime service while you keep the healthcare-specific logic in your own codebase.
Operational details you should not skip
Healthcare intake has non-technical constraints that affect the architecture:
Consent and disclosure: make it clear the user is interacting with an AI assistant.
Data minimization: collect only the fields required for the visit.
Timeouts: do not leave a session open indefinitely if the patient abandons it.
Rate limiting: protect the service from accidental replays or abuse.
Auditability: log enough to reconstruct the intake without storing unnecessary media.
If you expose an iframe-based entry point on a website, keep the backend and API key out of the browser entirely. The customer-managed embed model is useful when the goal is a drop-in experience, but for healthcare you still need to think carefully about origin allowlists, access control, and session duration.
Conclusion
The core pattern is simple: keep the avatar layer realtime and stateless, keep the intake logic structured and auditable, and keep the media transport out of Unreal’s gameplay code. That gives you a system that can collect patient information conversationally without turning the front end into a tangled mesh of video, voice, and business logic.
If you are implementing this now, start with a narrow intake flow, wire up session creation on the server, and validate the avatar sync under interruption and reconnect scenarios. Then expand the dialogue scope only after the latency and state machine are stable. The docs at docs.protoface.com are the best next stop for the exact SDK, API, and integration details.