Introduction

If you are building a healthcare triage flow, the hard part is usually not “can the model answer questions?” It is “can the system collect structured intake reliably, keep latency low enough to feel conversational, and present the interaction in a way patients actually trust?” A realtime AI intake avatar is the UI layer for that problem: a synchronized talking face wrapped around a voice agent that can ask questions, listen, clarify, and hand off structured context to downstream systems.

In practice, that means stitching together speech-to-text, an LLM, text-to-speech, low-latency media transport, and a video surface that stays in sync with the generated audio. The avatar is not the intelligence; it is the presentation and timing layer that makes the agent feel coherent. By the end of this post, you should be able to reason about the architecture, the failure modes, and where a developer platform like Protoface fits when you want to add a realtime face to a voice agent without building the media pipeline yourself.

What a realtime intake avatar actually is

A realtime intake avatar is best understood as a stateful rendering client attached to an agent loop. The avatar receives the agent’s generated speech and visual timing cues, then renders lip movement, head motion, and facial expression in lockstep with the audio. The healthcare-specific piece is the dialog design: the agent is constrained to gather structured data such as symptoms, duration, severity, medication, and red-flag conditions, rather than having an open-ended conversation.

The important implementation detail is that the avatar must track the same conversation state as the voice agent. If the agent interrupts itself, corrects a sentence, or asks for clarification, the avatar has to reflect that immediately. If the transport adds too much delay, the user perceives it as laggy or uncanny. If the audio and face drift apart, trust drops quickly.

For triage, this usually means the flow is:

1. Capture user speech from mic or telephony.

2. Transcribe incrementally or in short turns.

3. Run a dialog policy or LLM prompt that extracts intake fields.

4. Generate the next response as audio.

5. Drive the avatar with the same response timing so the face speaks in sync.

6. Persist structured intake data for a nurse, care team, or downstream EHR integration.

That last step matters. The avatar is only valuable if the intake is machine-usable, auditable, and easy to hand off.

Realtime architecture: the pieces that have to line up

There are three timing domains you need to think about: user input, model generation, and media playback. Most mistakes happen when teams treat them as one thing.

1. Input path: speech turns and interruption handling

In a healthcare setting, users interrupt themselves, change their minds, and answer out of order. Your agent should support partial turns and barge-in. That usually means the voice stack needs to stream audio frames in, detect end-of-utterance with some combination of VAD and turn detection, and support canceling or superseding an in-flight response.

For intake, don’t force the patient through a rigid script unless you have a strong reason. A better pattern is to collect required fields opportunistically:

• chief complaint

• onset and duration

• severity or pain score

• associated symptoms

• medications and allergies

• red-flag symptoms that require escalation

This lets the agent adapt to whatever the patient says first, then fill gaps with follow-up questions.

2. Generation path: the agent must emit both content and control

A voice agent for triage should not only return text to speak. It should also return machine-readable state: what question was asked, what slots were filled, and whether escalation criteria were met. In other words, the LLM output is not the product; it is an intermediate representation.

That distinction becomes important when you want the avatar to look responsive. If the model emits text in chunks, your TTS layer should support streaming synthesis so playback begins before the whole response is done. The avatar renderer then lip-syncs to that audio stream. If you wait for full completion before rendering, the system feels sluggish even if the model itself is fast.

For a triage bot, I recommend separating:

• conversation text — what the user hears

• structured intake state — what your backend stores

• presentation timing — when the avatar starts speaking, pauses, or stops

3. Media path: why “just embed a video” is not enough

A talking avatar is not a pre-rendered animation. It is a realtime media session that has to remain synchronized with generated speech. In practice that usually means WebRTC or a similar low-latency streaming transport, because you need:

• sub-second mouth-to-ear latency

• bidirectional communication

• adaptive network handling

• continuous playback without rebuffering every turn

WebRTC is a good fit because it is designed for interactive media, but it introduces its own operational concerns: signaling, NAT traversal, session lifecycle, reconnection, and browser autoplay restrictions. If you are building the whole stack yourself, you need to solve all of that before you get to the product work.

The avatar also needs consistent visual control signals. A good implementation does not just animate mouth shapes from audio amplitude; it uses phoneme timing or equivalent speech alignment, then layers expression and motion on top. That is what keeps the face from looking mechanically “bouncy” during long utterances or pauses.

Healthcare triage-specific constraints

Healthcare changes the design surface in ways that general-purpose sales or support agents do not.

First, you need explicit escalation logic. The agent should identify symptoms that require immediate human review and stop pretending it can safely continue alone. Second, you need conservative prompts. A triage agent should ask focused questions and avoid speculative diagnosis. Third, you need traceability. Every collected field should be attributable to the patient’s own words or clearly marked as inferred.

That means the backend should store more than the final transcript. At minimum, keep:

• turn-level timestamps

• transcript segments

• structured field extractions

• escalation decisions

• session metadata and agent version

If your avatar is the front end for intake, those logs are what let clinicians trust the output later.

Implementation sketch with a LiveKit voice agent

If you already have a voice agent running in LiveKit, the lowest-friction way to add a talking face is to attach a realtime avatar plugin to the agent pipeline. The plugin handles the video side while your agent keeps doing speech, turn handling, and tool calls.

The shape of the code is simple: initialize your agent as usual, configure the avatar, then attach it to the voice loop. Exact field names and environment setup live in the docs, but the integration pattern looks like this:

from livekit import rtc

session.run()

from livekit import rtc

session.run()

from livekit import rtc

session.run()

The practical upside is that you keep the agent logic where it belongs and let the plugin manage the synchronized video surface. If you want examples and package details, start with the plugin repository on GitHub and the setup notes in the documentation.

For teams using Pipecat instead of LiveKit, there is a dedicated integration path as well: see the Pipecat service reference and the plugin repository.

When you need direct API control

There are cases where you do not want the avatar embedded in the agent runtime. For example, you may want to create sessions from your own orchestration service, manage avatar metadata centrally, or build a custom web client. In those cases, a REST API is the right surface.

A minimal session-create request might look like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

And from Python, you would typically do the same thing programmatically with the SDK:

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

The exact request and response fields are documented in the docs. The key design point is that the API gives you a clean control plane for avatars and sessions, while the realtime media remains handled by the underlying runtime.

Operational gotchas worth planning for

A few details tend to surprise teams the first time they ship this kind of flow:

• Latency budgets are cumulative. Speech recognition, model response time, TTS, and video sync each add delay. Keep each step as incremental as possible.

• Interruptions are normal. The patient may cut off the avatar, and the avatar may need to stop speaking immediately when the user starts talking.

• Backpressure matters. If the agent emits too much text, your TTS and avatar layers can fall behind.

• Rate limiting is a product feature. For external-facing embeds, you want explicit usage boundaries rather than open-ended sessions.

• Privacy and security are not afterthoughts. Healthcare flows should keep credentials server-side, minimize browser exposure, and clearly separate session data from model prompts.

The more your agent behaves like a real intake nurse, the more you have to treat it like a realtime system rather than a chat widget.

Where Protoface fits in this architecture

This is the part where a platform is genuinely useful: it removes the video-avatar transport and synchronization work so you can focus on the triage logic. Protoface exposes the control surfaces you actually need as a developer: a REST API for sessions and avatars, a Python SDK for orchestration, and a LiveKit plugin for dropping a synchronized face into an existing voice agent. If you are building a web-facing intake experience, the iframe embed option keeps API keys out of the browser and lets you constrain usage with parent-origin allowlists and session limits.

For most teams, the productive path is to prototype the conversational flow first, then add the avatar once the intake schema and escalation logic are stable. That lets you validate the medical workflow before you worry about facial rendering details. If you want to dig in, start with the public docs at docs.protoface.com and the quickstarts linked from the repository README.

Conclusion

A realtime AI intake avatar is not just a talking head. It is a synchronized UI for a voice agent that has to collect structured information, stay low-latency, and behave safely under interruption. The main engineering challenge is lining up the input, generation, and media paths so the conversation feels continuous while still producing reliable structured output.

If you are implementing this for healthcare triage, start by designing the intake schema and escalation policy, then choose the runtime surface that matches your stack: LiveKit plugin, REST API, Python SDK, or embedded iframe. From there, keep the media path boring and the conversation logic strict. That combination is what makes the system usable in practice.

For setup details and examples, see the documentation.