Introduction

Healthcare triage is a good stress test for realtime AI interfaces: the user is often anxious, the conversation needs low-latency turn-taking, and the system has to feel responsive without pretending to be a clinician. A text-only chat box can work, but it leaves a lot of human cues on the table. A synchronized video face changes the interaction model: users can see when the agent is “listening,” “thinking,” and “speaking,” which makes a portal feel more like an actual intake desk than a static form.

This post shows how to embed a realtime triage avatar into a FastAPI-powered patient portal, and what matters technically when you do it. By the end, you should know how to wire up a patient-facing page to a realtime avatar session, keep credentials off the browser, handle session lifecycle cleanly, and avoid the common latency and security mistakes that make these systems feel brittle.

Start with the interaction model, not the avatar

For a triage workflow, the avatar is not the product; it is the front end of a voice agent that gathers symptoms, clarifies timing, and routes the user to the right next step. That means the important part is the turn-taking loop:

• The browser connects to a realtime media session.

• Audio is captured and streamed to the agent.

• The agent generates audio responses.

• The avatar lip-syncs the audio and renders the face in sync with speech.

In practice, the avatar should be treated as a media participant, not a decorative widget. If you are using a voice agent stack already, the avatar can sit alongside it and mirror its speech stream. If you are building the agent yourself, design the agent boundary so the avatar can consume the same synthesized audio that the user hears. That avoids drift between what is spoken and what is displayed.

FastAPI as the control plane

FastAPI is a good place to handle the pieces that should not live in the browser: auth, patient context lookup, session creation, and any per-visit policy decisions. The browser should never receive an API key. Instead, your backend can mint or request the session artifacts needed for the client to connect.

A common pattern looks like this:

1. The patient authenticates to your portal.

2. FastAPI verifies the user and loads any context you want the triage agent to know, such as appointment reason or queued intake fields.

3. Your backend creates a realtime avatar session or returns a short-lived session descriptor.

4. The frontend joins the session and renders the avatar.

Here is a minimal FastAPI endpoint that creates a server-side session using the Python SDK. The exact field names depend on the SDK version and the session schema in the docs, so treat this as a shape, not a copy-paste contract:

from fastapi import FastAPI, Depends

}

from fastapi import FastAPI, Depends

}

from fastapi import FastAPI, Depends

}

That endpoint should be protected like any other patient-data API. The important bit is that the browser receives only the minimum it needs to connect, not your long-lived API key or internal routing state.

Streaming details that actually matter

Realtime avatars feel good only when the media path is predictable. For voice-first triage, the critical constraints are:

• Latency: users will notice delays above a few hundred milliseconds in turn-taking, especially after they stop speaking.

• Jitter: inconsistent packet timing leads to awkward pauses and desynced facial motion.

• Audio continuity: the avatar should track the same synthesized audio buffer the user hears, not a separate approximation.

• State transitions: listening, thinking, and speaking states need to be explicit so the UI can reflect them.

In a patient portal, you also need to think about interruption handling. If the user cuts the agent off to correct a symptom or medication name, the agent should be able to stop speaking, process the interruption, and resume without carrying over stale context. That is mostly an agent design issue, but it has visible consequences in the avatar layer: if your speech pipeline is laggy, the avatar will keep talking after the user has already interjected, which feels broken fast.

For the frontend, keep the media session separate from page navigation. Patients will navigate between forms, instructions, and the triage interaction; you do not want the connection to depend on a full page reload. In a React or plain JS frontend, initialize the session once, attach cleanup handlers, and explicitly disconnect when the visit ends or the user logs out.

Security and compliance guardrails

Healthcare workflows force you to be disciplined about two things: credentials and data exposure. Do not embed API keys in client-side JavaScript. Do not log raw transcripts to places that are not approved for protected health information. And do not assume that a polished avatar UI changes your compliance obligations; it does not.

At minimum, your design should account for:

• Server-side session issuance: the backend authorizes the user before any realtime connection is created.

• Scoped context: only pass the patient-specific data the triage agent actually needs.

• Short-lived session state: expire sessions when the visit ends or times out.

• Auditability: keep a clear record of who started the session and when.

If you are also rendering the avatar inside an iframe on a portal page, you should still treat the embedded surface as untrusted from a credential perspective. The browser should get an embed URL or session token, not secret API credentials. That separation is the difference between a controllable integration and a browser-exposed backend.

Where Protoface fits in this architecture

This is the layer where Protoface is useful: it gives you a developer-facing avatar API, a Python SDK, and a managed realtime media surface so you can attach a synchronized face to your triage agent without inventing the avatar stack yourself. For a FastAPI portal, the simplest path is to create sessions server-side, return a safe join artifact to the frontend, and let the browser connect to the realtime avatar session while your backend keeps ownership of auth and patient context.

If you are already running a Python voice pipeline, the SDK is the cleanest place to start. If you prefer to plug an avatar into an existing voice agent, the LiveKit plugin is the shortest route. For example, the LiveKit agent side can remain focused on speech and turn management while the avatar plugin handles the visual layer:

# Illustrative shape only; see docs for exact imports and configuration

# Illustrative shape only; see docs for exact imports and configuration

# Illustrative shape only; see docs for exact imports and configuration

For browser-only integrations, the customer-managed iframe embed is also relevant because it keeps your backend out of the client and avoids exposing API keys in the browser. That matters in healthcare, where the less secret material you ship to the front end, the better.

If you want concrete examples, the quickstarts in the public repo are the fastest way to see the integration patterns in context, and the docs cover the API and session lifecycle in more detail.

Practical implementation notes

A few details tend to save time in production:

• Use patient-scoped avatars sparingly. Most triage flows do not need a unique avatar per user; they need one stable agent identity and patient-specific session state.

• Keep the UI honest. Show “connecting,” “listening,” and “speaking” states explicitly, because it reduces user anxiety and makes failures visible.

• Plan for fallback. If the realtime session cannot connect, degrade to text intake rather than leaving the patient at a dead end.

• Measure end-to-end latency. Do not only profile your backend. Measure time from user stop-speaking to first agent audio and first visible mouth movement.

When debugging, separate media issues from application issues. If the avatar is out of sync, check the audio pipeline first. If the agent appears to ignore user input, check the turn detector or VAD. If the page loads but the session never attaches, check authorization and token expiry before you blame the frontend renderer.

Conclusion

A realtime triage avatar is not hard because of the video face; it is hard because it sits at the intersection of auth, streaming media, and conversational state. FastAPI is a solid control plane for session creation and policy, while the browser should only handle a short-lived connection and rendering. If you keep the backend authoritative, keep credentials out of the client, and treat the avatar as part of the agent pipeline rather than a cosmetic widget, the result is much easier to operate.

For implementation details, session shapes, and integration examples, start with the documentation at docs.protoface.com. If you want to see the moving parts in code, the public quickstarts and SDK repositories are the fastest path from concept to working prototype.