Migrating from STT + TTS Chatbot to a Streaming AI Avatar for Patient Intake
Migrate patient intake bots from STT+TTS to a streaming AI avatar with low-latency voice, sync, and session patterns.
Introduction
Most patient-intake bots start as a standard speech pipeline: speech-to-text captures the user, an LLM decides what to ask next, and text-to-speech reads the response back. That works, but it leaves you with a disconnected experience: the agent has a voice, but no presence. In healthcare workflows, that often matters more than in a purely transactional chat flow. Patients need pacing, turn-taking, reassurance, and a sense that the system is actually listening.
A streaming AI avatar changes the interaction model without changing the core conversation logic. Instead of synthesizing audio and then separately rendering a static UI, you stream the agent’s speech to a live, lip-synced face in real time. By the end of this post, you should understand the architectural differences between STT+TTS chatbots and avatar-based voice agents, the main integration patterns, and the practical trade-offs involved in moving a patient-intake flow to a realtime avatar.
Why STT + TTS feels brittle in intake workflows
Classic voice bots usually have three disjoint pieces:
ASR/STT for user speech capture.
LLM or dialog logic for deciding what to ask next.
TTS for response playback.
The problem is not just “no face.” It is that each piece has its own latency, buffering, and interruption behavior. If the STT endpointing is too aggressive, you cut off patients mid-answer. If TTS is slow to start, the agent feels hesitant. If you add visual UI later, it often lags behind the audio and becomes decorative rather than functional.
For intake, this creates concrete issues:
Turn-taking drift. The agent responds before the patient finishes, or pauses too long and invites interruptions.
Loss of conversational context. If the audio and UI update on different clocks, the user cannot easily map what they heard to what they see.
Lower perceived trust. In healthcare, a disembodied voice can feel less attentive than a synchronized conversational presence.
A streaming avatar does not remove those problems by magic. It just forces the user-facing layer to share the same realtime transport as the voice agent, which makes the interaction easier to coordinate and easier to reason about.
What actually changes with a streaming AI avatar
Conceptually, the agent remains the same: the model still receives user speech or text, produces the next turn, and emits audio. The difference is that the output is no longer “just audio.” The synthesized speech is also used to drive a live video face through a streaming session. That means you need a transport with low end-to-end latency, incremental updates, and a way to keep the avatar synchronized with the spoken response.
In practice, this usually means a realtime media stack such as WebRTC or a similar streaming channel. The important characteristics are:
Low-latency delivery. Audio should start playing quickly; visual mouth motion should follow the same stream.
Incremental generation. The agent should be able to begin speaking before the full reply is finalized.
Interruptibility. When the patient interrupts, both audio playback and avatar animation need to stop cleanly.
Session identity. The avatar, voice agent, and conversation state need to stay tied to one realtime session.
That last point is where a lot of integrations get messy. If you glue a TTS service to a video renderer yourself, you end up with a fragile coordination problem: token refresh, media renegotiation, avatar state, session lifecycle, and authentication all become your problem. For patient intake, where the conversation is typically long-lived and stateful, that complexity shows up fast.
Integration patterns that actually work
There are three practical ways to add an avatar to an existing voice agent:
1) Wrap an existing voice agent with an avatar layer
If you already have a LiveKit-based agent, this is the least disruptive path. Your existing agent logic stays where it is; the avatar becomes another downstream consumer of the agent’s speech. That keeps the state machine, tool calls, and intake flow logic intact while giving the agent a synchronized face.
This is a good fit if your bot already handles:
identity verification or patient lookup,
structured intake questions,
handoff to a human staff member,
or multi-turn clarification before form submission.
A minimal Python sketch looks like this:
The specific class names and parameters will depend on your stack, but the architecture is the important part: the avatar is attached to the realtime session, not bolted on as an afterthought. If you are using LiveKit Agents, the plugin repo is the right place to start: GitHub repo. If you are using Pipecat, the integration guide is here: Pipecat guide.
2) Create and manage sessions from your backend
If you need more control over avatar provisioning, usage tracking, or session orchestration, use the REST API from your backend. This is the right shape when you want to create sessions on demand, tie them to patient or appointment records, and keep credentials out of the client.
A typical flow is:
Authenticate your server with an API key.
Create or select an avatar.
Start a realtime session for the patient intake flow.
Pass the session details to your frontend or agent runtime.
Use this pattern if you need backend-enforced policy such as eligibility checks, per-clinic configuration, or logging that aligns with your EHR or scheduling system. The API details are in the docs, so treat the snippet above as illustrative rather than exact.
3) Embed directly in the browser when you do not need a custom backend
For simpler web flows, an iframe embed can be the fastest path. This is particularly useful for intake kiosks, appointment pages, or pre-visit screening where you want a self-contained avatar experience and do not want to expose any server credentials to the browser.
The important properties are operational, not cosmetic: parent-origin allowlisting, per-embed instructions, and rate limits by IP and duration. That keeps the browser-side integration simple while still giving you control over who can host the embed and how it is used.
This pattern is not for every deployment. If you need deep EHR integration or custom workflow branching, use a backend-managed session instead. But if the goal is “add a conversational intake assistant to the page without building a client runtime,” an iframe is hard to beat.
Patient intake-specific design considerations
Moving from STT+TTS to an avatar does not change the clinical or product requirements. It just makes some of them more visible.
Keep the interaction structured. Intake works best when the avatar asks one thing at a time and confirms the result. Patients do not tolerate multi-clause prompts well, especially when they are already trying to remember medication names or symptoms.
Design for interruption. In voice UX, the patient should be able to say “wait” or “no, that’s not right” and have the system stop cleanly. That means your agent logic, audio playback, and avatar animation must all respond to barge-in events consistently.
Separate conversational state from presentation state. The avatar is presentation. The intake form, validation rules, and workflow state belong elsewhere. If you couple them too tightly, you will make future changes painful.
Plan for escalation. Real patient intake is full of edge cases: unclear speech, multilingual users, partial answers, and sensitive disclosures. A good avatar should hand off gracefully to a human or a conventional form when the interaction stops being efficient.
Measure latency end to end. The user experiences one thing: time from speaking to receiving a useful response. Measure STT latency, LLM time-to-first-token, TTS start time, and avatar render delay separately. Otherwise you will optimize the wrong layer.
How Protoface fits without re-architecting the whole stack
Protoface is useful here because it sits at the presentation/session layer rather than forcing you to rewrite your agent. If you already have a working realtime voice agent, the LiveKit plugin is the lowest-friction path: drop in a synchronized avatar, keep your existing orchestration, and let the agent speak through a face. If you need operational control, the REST API and Python SDK let your backend create and manage realtime sessions directly, while the dashboard gives you session visibility and a playground for testing.
The main thing to keep in mind is that the avatar should be treated as part of the realtime media session, not as a separate UI widget. Once you model it that way, the implementation becomes much cleaner: your agent owns the conversation, your backend owns policy and identity, and the avatar layer handles synchronized presentation.
If you are evaluating this for production intake, start with the docs, wire up a small internal flow, and measure what actually changes in completion rate and interruption handling before expanding the rollout: docs.
Conclusion
Migrating from an STT + TTS chatbot to a streaming AI avatar is mostly an integration problem, not a modeling problem. The core agent logic can stay the same, but you need a realtime transport, a clean session model, and a way to keep speech and facial animation synchronized under interruption and latency pressure.
For patient intake, that shift is usually worth it when you care about turn-taking quality, perceived attentiveness, and a more coherent conversational experience. The practical next step is to prototype one flow end to end: a single intake script, a single avatar, and instrumentation around latency and barge-in behavior. Then expand only if the interaction actually improves.
Start with the documentation, pick the integration surface that matches your stack, and keep the agent logic separate from the avatar layer. That tends to scale much better than trying to retrofit a face onto an already fragile voice bot.