How to Build a HIPAA-Aware Realtime Patient Intake Avatar in Unreal Engine
Build a HIPAA-aware realtime patient intake avatar in Unreal Engine with backend sessions, WebRTC, and PHI-minimizing state machine workflows.
Introduction
If you want a patient intake flow that feels immediate and human, a static form is usually the wrong shape. Patients hesitate, drop off, forget details, or need clarification on the spot. A realtime avatar can bridge that gap by combining conversational voice, synchronized facial animation, and an intake workflow that guides the user through a fixed set of questions while responding naturally to interruptions and clarifications.
This post shows how to build that experience in Unreal Engine with a HIPAA-aware architecture. The goal is not “HIPAA certified by default” — no vendor can hand you that by implication — but rather a design that minimizes exposure of protected health information (PHI), keeps secrets off the client, and lets you reason about data flow, session boundaries, and auditability. By the end, you should know how to wire Unreal to a realtime avatar session, keep the medical workflow deterministic enough for intake, and avoid the common security mistakes that turn a polished demo into a compliance problem.
Start with the right threat model
For patient intake, the main risk is not the avatar itself; it is the surrounding system. The avatar is just the presentation layer for a voice agent. The compliance questions are about what text is sent to the model, where audio is routed, how long session data persists, and whether any browser or game client can mint its own credentials.
A practical HIPAA-aware architecture usually follows these rules:
Do not expose API keys in Unreal, the browser, or mobile clients.
Keep PHI out of logs unless you have explicit policy and retention controls.
Minimize prompt scope; send only the intake context needed for the current step.
Separate identity, scheduling, and clinical data from avatar transport.
Prefer short-lived realtime sessions over long-lived, reusable tokens.
That means the avatar should be treated as a streaming endpoint attached to a backend-controlled session. In practice, your backend authenticates with the avatar service, creates a session for a specific patient encounter, and hands Unreal only the session material it needs to connect.
How the realtime path actually works
Under the hood, a realtime avatar is usually a media pipeline: audio goes in, transcript or intent processing happens on the agent side, and synchronized lip movement or facial animation comes back as video frames or a video-like stream. In a WebRTC-style setup, you care about latency more than throughput. For intake, 150–300 ms of extra delay can make the interaction feel noticeably less natural, especially when the patient is interrupting or correcting information.
There are three separate loops to think about:
Capture loop: microphone input from the patient, or synthesized agent speech if the agent is talking back.
Reasoning loop: the agent decides what to ask next, validates answers, and maintains workflow state.
Presentation loop: the avatar emits synchronized facial animation and video so the user sees an engaged, speaking face.
For patient intake, the reasoning loop should be structured, not open-ended. You want constrained prompts and explicit state transitions: demographic details, insurance confirmation, chief complaint, allergy check, consent acknowledgment, and handoff. If you let the model improvise too much, you’ll get a nicer conversation and a worse intake.
Design the intake flow as a state machine
The most reliable pattern is to model intake as a state machine with a bounded set of states and a small memory object attached to the session. Each state asks for one thing, validates the answer, and either advances or reprompts. The avatar is then just the face on top of that state machine.
For example:
Each state should define:
the prompt the agent should use,
the fields that can be filled,
validation rules,
the fallback when the user is unclear, and
whether the state can be safely repeated.
That structure matters for HIPAA because it limits accidental over-collection. If the user starts telling a long story during a medication check, the agent can redirect to the exact field it needs instead of trying to summarize an entire symptom narrative.
Build the Unreal side as a thin client
In Unreal Engine, resist the urge to make the client “smart.” The engine should render the avatar, play audio, capture microphone input if needed, and maintain a realtime transport connection. It should not generate credentials or make policy decisions.
A good split is:
Backend: authenticates users, creates sessions, stores clinical data, issues ephemeral session metadata.
Unreal client: joins a pre-authorized session, renders the avatar, streams audio/video, and displays local UI.
If you are connecting Unreal to a WebRTC-based avatar session, the mechanics are familiar: acquire connection parameters from your backend, initialize the transport, attach incoming media tracks to a texture or media component, and forward microphone audio into the session. The avatar itself should be a remote media source; do not try to regenerate the face locally unless you are intentionally building a custom renderer.
One useful implementation detail: treat network errors as recoverable, but session expiration as terminal. If the user gets dropped mid-intake, you may want to reconnect to the same backend encounter; if the session token expires, mint a fresh one server-side and rebind only if the encounter is still active.
Keep PHI bounded in prompts and logs
The hardest compliance problems often come from convenience features. Debug logs, analytics, prompt traces, and crash reports all tend to capture text by default. For a patient intake system, that’s dangerous.
Use these practices:
Redact names, dates of birth, phone numbers, and free-text symptom descriptions from application logs where possible.
Store transcript data only in systems that are explicitly approved for PHI.
Keep model prompts small and contextual. For example, send “collect insurance policy number” rather than a full patient chart.
Use separate environments for development and production so test data never mixes with patient sessions.
Expire realtime sessions aggressively; keep only the minimum audit trail required by policy.
Also remember that “voice agent” does not mean “all conversation content belongs in memory forever.” If you need a running summary for the agent, generate a structured state object instead of storing every utterance verbatim. That reduces retention and makes handoff to the EHR or intake system much cleaner.
Where Protoface fits
This is the kind of setup Protoface is built for: you keep the clinical workflow on your side, and the avatar service handles the realtime face and session transport. For backend-driven Unreal integrations, the REST API is the relevant surface because your server can create and manage avatars and sessions without exposing secrets to the client. The docs are the right place for exact request fields and session parameters, but the pattern is straightforward: your backend authenticates with an API key, creates a session for a specific encounter, and hands Unreal a short-lived connection payload.
If you are already running a Python control plane for intake orchestration, the Python SDK is a good fit for creating avatars, sessions, and server-side automation. For lower-level reference and current fields, use the docs rather than guessing at payload shape.
For teams that prefer a voice-agent stack already integrated with LiveKit, the quickstart examples are a useful way to see the media and agent wiring in practice. Even if your renderer is Unreal instead of a browser, the transport and state-management ideas carry over.
Implementation gotchas that matter in production
A few things tend to break first:
Latency spikes: the avatar may appear smooth while the underlying agent has uneven turn-taking. Measure end-to-end turn latency, not just video FPS.
Partial utterances: users interrupt themselves constantly. Your NLU or transcription pipeline needs to tolerate corrections and sentence restarts.
Session drift: if the client reconnects, make sure the backend session state is still authoritative.
Consent timing: if recording or transcription is involved, consent should be explicit and captured before PHI-rich dialogue begins.
Fallback UX: if the avatar fails, the intake process still needs a non-avatar path, ideally a simple form or human handoff.
Also, test with noisy audio and awkward questions. Patient intake is a hostile environment for speech systems: accents, background chatter, paper shuffling, and people speaking over the agent are normal, not edge cases.
Conclusion
A HIPAA-aware realtime intake avatar is mostly an architecture problem. Keep the client thin, keep credentials server-side, constrain the conversational surface to a state machine, and minimize how much PHI enters the realtime path in the first place. Unreal Engine is a good presentation layer for this if you treat it as a media renderer and not as the source of truth.
If you want to implement this quickly, start with a server-created session, a strict intake flow, and a single realtime avatar connection. Then harden logging, retention, and fallback behavior before you add more conversational flexibility. For API details, session fields, and current integration guidance, see docs.protoface.com.