Introduction

If you are building a healthcare intake flow in Flutter, you already know the hard parts: patients abandon forms, voice UX needs to feel natural, and anything involving PHI has to be designed carefully from the start. A static chat widget can collect data, but it does not reassure people, guide them through an awkward intake, or make the interaction feel human.

This is where a realtime avatar helps. The pattern is straightforward: a voice agent listens, speaks, and drives the intake, while a synchronized video face makes the experience easier to follow and less sterile. In this post, we will wire that up in Flutter with a FastAPI backend and Python, and we will keep the architecture honest: the browser or mobile client should never see your API key, session state should live server-side, and media transport should be treated like a realtime system, not a normal REST request.

We will focus on the practical integration points: how to structure the Flutter app, how FastAPI brokers avatar sessions, how Python can create and manage those sessions, and what to watch for when you are moving audio/video across a network that may be imperfect.

Architecture: keep the media path separate from your app backend

For intake, the frontend should do as little as possible beyond rendering UI and participating in the media session. A clean setup looks like this:

• Flutter app: presents the intake experience, captures user audio, and renders the avatar video track.

• FastAPI backend: authenticates the user, creates a realtime avatar session, and returns only the short-lived data needed by the client.

• Protoface session: hosts the avatar and the realtime state that drives lip sync, speech, and animation.

The important design choice is that the backend owns the privileged token exchange. Your Flutter app should request a session from FastAPI, and FastAPI should talk to the avatar service using an API key stored in server-side environment variables. The client should receive only an ephemeral session payload or connection instructions, depending on the API shape you use.

For healthcare intake, this separation matters for a few reasons:

• It prevents long-lived API credentials from being embedded in the app.

• It gives you one place to enforce identity, session duration, and rate limits.

• It makes it easier to audit when a user started, stopped, or reconnected to an intake session.

FastAPI session broker: create the avatar session server-side

FastAPI is a good fit because session creation is a normal request/response operation, while the media itself is not. The backend can validate the patient, create the session, and return the data the Flutter client needs to join.

The exact request and response fields depend on the session API in the docs, but the pattern is usually the same: your backend makes an authenticated call to create a realtime session with instructions, voice choice, and any per-session constraints.

import os

return r.json()

import os

return r.json()

import os

return r.json()

This example is intentionally generic. In practice, you will map your own fields to the documented API shape. The main point is that session creation belongs in FastAPI, not Flutter.

Once the session is created, your backend can also persist your own intake record:

• patient or appointment identifier

• session id

• creation time

• selected avatar or voice profile

• status transitions like started, paused, completed, or failed

That lets you reconcile application state with media state later, which is useful when a session gets interrupted mid-intake.

Flutter client: join the realtime session and render the avatar

In Flutter, treat the avatar like any other realtime media participant. The client should call your FastAPI endpoint, receive the join data, and then connect using the SDK or transport mechanism your stack uses. The key is that the client only handles ephemeral session data, never your secret API key.

If your app is already using a WebRTC-style media layer, the UI side is usually familiar: subscribe to the avatar’s video track, render it into a widget, and send microphone audio upstream. The avatar is useful precisely because it is synchronized with the speech path; if the audio and video drift independently, the whole experience feels broken.

A basic Flutter flow looks like this:

1. User taps “Start intake”.

2. Flutter POSTs to your FastAPI backend.

3. Backend returns session connection data.

4. Flutter joins the realtime session and starts rendering the avatar.

5. Audio from the patient is sent to the agent, and the avatar speaks back with lip sync.

On the UI side, avoid overcomplicating the initial render path. The avatar surface should be a single media view with obvious states: connecting, listening, speaking, reconnecting, ended. Those states matter more than fancy transitions because network conditions in clinic environments are rarely ideal.

Two implementation details are easy to miss:

• Audio permission timing: request microphone access only when the user explicitly starts the intake. It reduces friction and avoids an awkward permissions prompt before they understand why you need it.

• Reconnection behavior: assume the stream can drop. Your client should be able to rejoin the same logical intake session without losing the backend record of progress.

Realtime healthcare intake constraints: correctness beats polish

Healthcare UX adds constraints that general chatbot demos can ignore. The avatar should feel calm and consistent, but more importantly it should be reliable and bounded.

Three practical rules help:

• Constrain the conversation. Use session instructions to keep the agent on intake-only behavior, with short questions and explicit confirmations where needed.

• Keep PHI out of logs. Do not dump raw transcripts, audio metadata, or session payloads into general application logs unless you have a defined retention and access policy.

• Enforce session limits. Intake should end deterministically, either after completion or after a strict duration cap. That protects cost, reduces confusion, and makes audit trails cleaner.

From a systems perspective, think of the avatar as a realtime worker with state, not as a UI component. That means you need operational controls: start, stop, timeout, failure handling, and observability. Even if the media layer is managed for you, your app still needs to know whether the intake is actually progressing.

One other subtlety: latency. Users tolerate some delay in intake, but they notice mismatches between what they hear and what they see. If your upstream audio path, model inference, or avatar render pipeline introduces too much lag, the avatar will look detached from the conversation. That is why session initialization should happen before the user is mid-question, and why you want lightweight backend round trips.

Where Protoface fits: a server-side avatar session API, not a browser secret

This is the part where Protoface fits naturally: it gives you a developer-facing realtime avatar layer that you can create and manage from your FastAPI service, while the Flutter app stays free of private credentials. The public REST API at api.protoface.com is the right surface when you want to create sessions, configure per-session instructions, and manage avatar state from code.

If you prefer Python, the SDK is a cleaner way to keep that logic idiomatic. A small example looks like this:

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

The SDK name, method names, and request fields should be verified against the current reference, but the shape is what matters: server-side code creates the session, client code joins it.

If you are already using a voice-agent framework, the same principle applies there too. For example, the LiveKit plugin adds a synchronized avatar face to an existing voice agent, which is useful when your intake pipeline already uses LiveKit for audio transport and agent orchestration. In that setup, the agent keeps handling conversation logic, while the avatar layer handles the visual presence.

For developers who want a faster path to validate the flow, the quickstarts in the public repos are the shortest route to a working end-to-end sample. They are especially useful if you want to compare a pure REST/session flow with a framework-integrated one before you commit to your production architecture.

Implementation notes and gotchas

A few issues come up repeatedly in realtime intake systems:

• Do not expose the API key to Flutter. Even in a private app, client-side secrets get copied eventually.

• Separate session identity from patient identity. Use your own appointment or encounter IDs in backend metadata, but keep those mappings server-side.

• Keep the first turn short. Long greetings make the interaction feel slower and delay useful data collection.

• Test mobile network transitions. Wi-Fi to LTE handoffs and backgrounding behavior are common failure modes.

• Design for fallback. If the avatar session fails, your app should degrade to a plain audio or form-based intake path instead of blocking the workflow.

Also, remember that “realtime” does not mean “continuous at any cost.” In a healthcare context, predictable behavior matters more than aggressive animation or elaborate personality. The avatar is there to improve completion and comprehension, not to become the product.

Conclusion

The clean way to embed a realtime healthcare intake avatar in Flutter is to keep the media session and the application backend decoupled: FastAPI brokers the privileged session creation, Flutter consumes ephemeral join data, and the avatar layer handles synchronized speech and video. That architecture is simple to reason about, safer for secrets, and much easier to evolve when you need auditability or stricter session controls.

If you want to implement this pattern, start with a minimal FastAPI session broker, wire a Flutter screen that can join and render a media participant, and then tighten the operational pieces: limits, reconnection, and logging policy. The docs at docs.protoface.com are the right place to confirm the exact API shape and integration options, and the linked examples are useful when you want a working baseline before you adapt the flow to your intake system.