Introduction

There’s a common failure mode in healthcare triage systems: the backend is technically correct, but the user experience feels like filling out a form with a chat box bolted on top. For low-friction intake, symptom checks, or routing to a nurse line, text-only often creates avoidable drop-off. A voice agent helps, but a disembodied assistant can still feel sterile in workflows where trust, pacing, and clarity matter.

This post walks through what changes when you migrate a FastAPI-based triage app from text or voice-only interaction to a live voice + video avatar. The goal is not to “add graphics.” It’s to preserve your existing backend logic, keep the clinical flow deterministic, and attach a synchronized talking face that makes the interaction easier to follow for patients. By the end, you should understand the integration points, the WebRTC streaming model, and where to put the avatar layer without tangling it with your triage logic.

Keep the triage engine separate from the presentation layer

The first architectural decision is simple: do not move triage logic into the avatar layer. Your FastAPI app should still own intake state, risk scoring, escalation rules, audit logs, and handoff semantics. The avatar should only consume and present the agent’s output.

A clean separation usually looks like this:

• FastAPI handles patient session creation, authentication, triage state, and any EHR/CRM integrations.

• Voice agent handles speech-to-text, reasoning, and text-to-speech.

• Avatar layer renders the face and lip-sync in real time based on the agent’s audio/video output.

That separation matters operationally. If you later switch TTS providers, revise your prompt, or add a human handoff step, the avatar integration should not change. The avatar receives the live media stream; it should not be your source of truth for patient state.

What changes when you add a live avatar

Most developers think of this as a frontend problem. In practice, it’s a media-routing problem.

In a voice-only app, you typically have:

1. Browser or telephony client captures audio.

2. Voice agent transcribes, reasons, and generates speech.

3. Audio returns to the client.

With a live avatar, you add a synchronized visual participant that tracks the speaking turn. That usually means the avatar is attached to the same real-time pipeline that carries the agent’s voice. In WebRTC terms, you’re extending the participant graph, not rendering a prerecorded video. The important property is low latency: the mouth movement needs to follow the audio closely enough that users perceive one coherent speaker.

For healthcare triage, that low-latency coupling matters because the avatar should never appear to “talk over” the agent’s turn-taking. If the patient hears a pause, sees a pause. If the agent interrupts itself to clarify a symptom, the avatar should reflect that. In practice, this means you want the avatar to subscribe to the same agent stream rather than trying to synthesize a separate video narrative.

FastAPI integration pattern: expose a session endpoint, not a media pipeline

If your current app already creates triage sessions in FastAPI, that endpoint is the natural place to mint whatever session metadata the avatar layer needs. Do not make FastAPI stream audio/video itself unless that is already part of your design. A good migration path is to keep FastAPI as the control plane and hand media to the agent infrastructure.

For example, your app might create a triage session and return a token or session ID to the client:

from fastapi import FastAPI

}

from fastapi import FastAPI

}

from fastapi import FastAPI

}

The browser then joins the real-time voice session, and the avatar comes along as part of that agent session. Your FastAPI app can still enforce who is allowed in, how long a session lasts, and when to escalate to a human.

Managing latency, turn-taking, and clinical reliability

In triage, small UX mistakes compound. If the agent takes too long to respond, patients repeat themselves. If the avatar visibly lags behind the audio, the system feels broken. If the video face keeps animating while the agent is silent, trust erodes quickly.

There are a few practical rules:

• Keep response latency bounded. Stream partial transcriptions and generate responses incrementally where possible.

• Use explicit turn boundaries. The avatar should animate while the agent is speaking and settle when the agent yields the floor.

• Do not over-animate during silence. For healthcare, subtle idle behavior is better than exaggerated motion.

• Have a deterministic fallback. If video fails, the voice agent should continue without breaking the triage session.

Also remember that a face changes the perceived authority of the system. That can be helpful, but it increases the importance of prompt discipline and escalation guardrails. If the model is uncertain, it should say so plainly. If it recommends emergency care, the wording should be unambiguous. The avatar should never imply diagnosis; it should represent the agent’s delivery, not replace clinical judgment.

Example: using the LiveKit agent plugin to add the avatar

If your voice agent already runs on LiveKit, the cleanest path is to attach the avatar directly to that agent. The plugin surface is intentionally narrow: it drops a Protoface avatar into the LiveKit agent so the agent gains a synchronized talking video face. You keep your existing agent topology and add a visual participant to it.

A minimal setup looks like this:

from livekit.agents import WorkerOptions, cli

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

from livekit.agents import WorkerOptions, cli

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

from livekit.agents import WorkerOptions, cli

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

That snippet is intentionally schematic; the exact configuration, session creation, and authentication details are in the docs and the plugin examples. The important point is the shape of the integration: the avatar is a participant in the agent session, not a separate video overlay you have to synchronize yourself. If you want to see a working plugin-oriented setup, the repository and examples are here: GitHub and the integration docs are at docs.protoface.com.

When to use the REST API or Python SDK instead

Not every team needs a LiveKit-first integration. If you’re orchestrating sessions from your own backend, the REST API and Python SDK are often a better control plane for avatar/session lifecycle.

That’s especially useful if your FastAPI app already creates user-specific sessions and you want to pre-provision an avatar session before the patient lands in the waiting room. A typical flow is:

1. Create an avatar or pick an existing one.

2. Create a realtime session with per-session instructions or voice settings.

3. Return the session credentials or join data to the frontend or agent runtime.

An API request might look like this:

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","metadata":{"flow":"triage"}}'

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","metadata":{"flow":"triage"}}'

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","metadata":{"flow":"triage"}}'

The exact path and fields are in the docs, but the idea is straightforward: keep the sensitive keys server-side, create sessions from the backend, and treat the avatar as a managed real-time resource.

Practical migration checklist for a FastAPI triage app

If you’re moving an existing app, I’d do it in this order:

1. Preserve your current triage endpoint. Keep the same state machine, same risk thresholds, same escalation logic.

2. Add a realtime session layer. Your FastAPI app should mint session metadata and enforce access control.

3. Attach the avatar to the voice agent. Do this in the agent runtime, not in the business logic layer.

4. Test failure modes. Audio-only fallback, reconnects, slow networks, and agent restarts all matter more once video is present.

5. Review wording with clinical stakeholders. The avatar can improve clarity, but it also makes the system feel more authoritative.

One thing developers often underestimate is session lifecycle. Patients may abandon and rejoin. A nurse may need to take over mid-stream. Your session model should support cleanup, expiry, and handoff without leaking resources or leaving zombie media participants behind.

How Protoface fits without changing your backend shape

This is where Protoface is useful: it gives you a managed avatar surface that plugs into the real-time agent you already have, rather than forcing you to build lip-sync and video composition yourself. For a LiveKit-based triage agent, the plugin route is the shortest path. If you’re controlling session lifecycle from FastAPI, the REST API or Python SDK keeps the avatar/session management on the server side. The key point is that you can add the face without exposing API keys in the browser and without turning your app into a custom media server.

For teams that prefer to start small, the public docs and quickstarts are a practical way to validate the integration against your actual agent stack before you wire it into production triage flows. See the documentation at docs.protoface.com and, if you want implementation examples, the relevant open-source repos under GitHub.

Conclusion

Migrating a FastAPI healthcare triage app to a live voice + video avatar is mostly an architecture exercise. Keep triage logic in FastAPI, keep media handling in the voice agent runtime, and treat the avatar as a real-time participant that follows the agent’s audio. That gives you a system that is easier to reason about, easier to fail over, and much less likely to become a tangled one-off.

If you’re implementing this now, start with your existing session endpoint, attach the avatar to the agent layer, and test the boring cases first: reconnects, latency spikes, and fallback to voice-only. The docs at docs.protoface.com cover the current surfaces and exact request shapes, and the quickstart repos are the fastest way to see the integration end to end.