How Do Voice, STT, and TTS Work Together in a Realtime Medical Intake Agent?
Realtime medical intake agent architecture: streaming STT, LLM turn state, TTS, interruption handling, and avatar lip sync.
Introduction
A realtime medical intake agent is a three-way coordination problem: speech recognition turns audio into text, the language model decides what to ask next, and speech synthesis turns the response back into audio. If you add a visual avatar, you also need lip sync and frame delivery to stay aligned with the audio stream. The hard part is not any single component; it is keeping all of them synchronized under low latency, partial audio, interruptions, and clinical-style turn taking.
By the end of this post, you should be able to reason about the end-to-end pipeline, understand where latency and state drift usually come from, and see how to wire voice, STT, and TTS into a realtime intake flow that feels coherent instead of fragmented. I’ll also show where Protoface fits when you want the agent to have a live talking face.
The realtime pipeline: audio in, text out, audio back
The simplest mental model is:
User speaks into a microphone.
Streaming STT emits partial and final transcripts.
The agent consumes those transcripts, maintains dialog state, and decides whether to ask a follow-up, confirm a symptom, or continue collecting data.
TTS synthesizes the response as soon as the assistant message is available, often token-by-token or sentence-by-sentence.
The audio is played back immediately, while the agent may continue listening for barge-in.
In a medical intake setting, the important detail is that these stages are not strictly sequential. You usually want streaming behavior at every boundary:
STT streaming so you can react before the user finishes a long sentence.
LLM streaming so the agent can begin composing a response before the full completion is known.
TTS streaming so playback begins with the first synthesized chunk.
Interrupt handling so the user can correct the agent mid-utterance without waiting for a full turn to complete.
This matters more in intake than in casual conversation. Users speak in fragments, correct themselves, and often answer multiple questions at once. If your pipeline waits for fully finalized transcripts before doing anything, the experience becomes sluggish and the agent starts to feel like a form with a delay rather than a conversation.
How STT should behave in a clinical-style dialog
Streaming STT usually produces two kinds of output: partial hypotheses and finalized segments. Partial output is useful for responsiveness, but you should not treat it as ground truth. In a medical intake flow, that distinction matters because a wrong partial transcript can send the dialog down the wrong branch if you are too eager to commit state.
A practical pattern is to keep two layers of state:
Ephemeral transcript state for partial hypotheses, used for live UI feedback or early turn detection.
Committed intake state for finalized transcript segments, used to update fields like symptom onset, medication names, allergies, and duration.
That gives you a way to be responsive without being brittle. For example, if the user says, “I’ve had chest pain for about… actually, more like three days,” your UI may show the evolving partial text, but the intake record should only update after the final segment is stable or after you explicitly resolve the correction.
There is also a turn-taking issue. In voice agents, silence detection is often as important as recognition accuracy. If your end-of-utterance threshold is too short, you cut people off mid-thought. Too long, and the agent feels unresponsive. For medical intake, a slightly conservative endpoint is usually better than aggressively low latency, because patients pause while thinking through symptoms or medication lists.
How TTS and lip sync stay aligned
Once the assistant decides to speak, TTS becomes the pacing source for the rest of the turn. The important thing is to treat the synthesized audio as the canonical clock for both playback and avatar animation. If the avatar’s mouth motion is driven by text timing while the audio is delayed, or vice versa, the result looks off immediately.
The common implementation is:
The assistant generates response text, often as a stream.
TTS converts that text into audio chunks.
The client or media layer plays the audio in realtime.
The avatar renderer consumes the same audio timing information to drive visemes or mouth-open/close motion.
The practical gotcha is buffering. A little buffer is necessary to smooth jitter, but too much buffer adds visible latency and makes interruptions feel broken. If the user starts speaking again, the system should stop playback quickly, cancel the remaining synthesis if possible, and transition back to listening. In other words, the conversation engine should own turn state, while the media layer should be a fast consumer of that state.
For medical intake, you also want the assistant’s speech style to be concise and confirmation-oriented. Long, multi-clause spoken responses increase the chance that the user interrupts or loses track of what to answer. A good pattern is to ask one question at a time, summarize back what you heard, and explicitly confirm uncertain values:
“I heard a penicillin allergy. Is that correct?”
“You said the pain started three days ago, right?”
“Are you taking any blood thinners?”
Where the avatar fits in the realtime stack
The avatar is not the conversation engine; it is the synchronized presentation layer. That distinction is easy to miss. Your agent can function perfectly well as voice-only, but once you add a face, the visual stream has to stay in lockstep with the audio turn model. Otherwise the mouth keeps moving after the agent has been interrupted, or the face is visibly idle while audio is still playing.
That is why avatar integration should attach to the voice pipeline rather than sit beside it as an independent media widget. You want the avatar to observe the same turn boundaries as the voice agent: listening, speaking, interrupted, and idle. In practice, the avatar should reflect the current audio state, not invent its own notion of dialog state.
From a developer’s perspective, there are three useful integration patterns:
Embedded avatar in an existing voice agent when you already have STT, LLM, and TTS wired up.
Hosted session or REST-controlled lifecycle when you need to create sessions, manage avatars, or provision client access ahead of time.
Browser embed when you want a customer-facing experience without shipping backend credentials to the client.
For a voice-first application, the first pattern is usually the cleanest because it preserves your existing agent logic and adds only the visual layer.
Example: attach a Protoface avatar to a LiveKit voice agent
If your voice agent already runs on LiveKit, the GitHub organization includes the plugin path you would expect. The plugin drops an avatar into the agent so the spoken output is mirrored by a synchronized talking face. The exact constructor fields and session details are documented, but the shape is roughly:
That is the key pattern: the voice pipeline remains your source of truth, and the avatar subscribes to the same realtime session. If you are using Pipecat instead of LiveKit, the integration is similar in spirit; see the plugin docs and examples in the relevant repository for the exact adapter shape.
Minimal API flow: create an avatar and start a session
If you are managing avatars and sessions directly, the REST API is the right surface. The workflow is straightforward: authenticate with an API key, create or select an avatar, then start a realtime session tied to that avatar. The docs have the exact request and response shapes, but a typical request looks like this:
That style of flow is useful when you need server-side control over session lifetime, rate limiting, or per-tenant configuration. It also keeps credentials off the client, which is usually the right choice for regulated workflows.
Operational gotchas that matter in production
There are a few failure modes that show up quickly in realtime intake systems:
Transcript drift: partial STT text gets committed too early, and the intake record becomes noisy.
Turn overlap: the agent speaks while the user is still talking, so the user repeats themselves or stops answering.
Audio/video skew: the avatar lags behind the spoken turn, which makes the system feel fake even if the content is correct.
Overlong prompts: the model generates too much explanation, making the interaction slower than a human intake nurse.
Poor interruption handling: the user cannot correct a medication name or allergy without waiting for the current turn to finish.
The fix is mostly architectural discipline: stream aggressively, commit conservatively, and keep one authoritative turn state in the agent runtime. The avatar should follow that state, not compete with it. For medical workflows, prefer confirmation questions, short responses, and explicit retries when a transcript confidence threshold is low or the model extracts a risky field.
Conclusion
Voice, STT, and TTS work together in a realtime medical intake agent by forming a single low-latency control loop: listen, transcribe, decide, synthesize, speak, and yield back to listening as quickly as possible. The hard parts are not syntax or SDK calls; they are state management, interruption handling, and keeping the media layers synchronized with the conversation state.
If you are building this stack, start with a voice-only agent that handles streaming STT and TTS correctly, then add the avatar as a presentation layer once the turn logic is stable. If you want implementation details, integration examples, or session management APIs, the docs are at docs.protoface.com.