FastAPI Tutorial: Streaming a Conversational Triage Avatar with WebSocket and STT/TTS
FastAPI WebSocket triage avatar tutorial: stream STT, partial transcripts, TTS audio, and synchronized realtime avatar state.
Introduction
If you are building a triage flow, the core problem is not “how do I talk to an LLM?” It is “how do I keep latency low enough that the conversation feels continuous, while also streaming speech in and out, preserving turn-taking, and giving the user a visible agent that reacts in real time?” A text-only chat UI can tolerate a lot of slop. A conversational avatar cannot.
This post shows how to build a FastAPI-backed triage avatar that receives microphone audio over a WebSocket, runs streaming speech-to-text (STT), forwards partial transcripts into a conversational loop, streams generated speech back through text-to-speech (TTS), and keeps the avatar synchronized with the audio. By the end, you should understand the moving parts well enough to wire your own production flow: session lifecycle, audio framing, turn detection, and the practical places where a realtime avatar layer fits into the system.
Architecture: keep the audio path separate from the control path
The first design decision is to separate control messages from media streaming. In practice:
WebSocket handles low-latency bidirectional signaling and audio chunks.
STT converts user speech into partial and final transcripts.
Agent logic decides when to respond and what to say.
TTS synthesizes the reply as streaming audio.
Avatar/video stays synchronized to the speech output.
That separation matters because “conversation state” and “media transport” have different requirements. Your control layer wants JSON messages, retries, IDs, and session metadata. Your media layer wants stable timing, small frames, and minimal buffering. If you mix them, debugging becomes unpleasant very quickly.
A simple flow looks like this:
For a triage bot, the most important behavioral rule is that you should not wait for a fully finalized user utterance unless you have to. Streaming partial transcripts lets you react early, detect interruption, and start preparing a response before the user finishes speaking.
FastAPI WebSocket: streaming audio and control messages
FastAPI is a good fit here because it gives you a straightforward async WebSocket API and plays nicely with background tasks. The simplest version is a websocket endpoint that accepts binary audio frames and JSON control messages.
In a real implementation you will likely carry 16 kHz mono PCM or Opus from the browser. PCM is easier to reason about; Opus is usually better for bandwidth. The exact transport is up to you, but keep the chunk size small enough to stay under human-perceptible latency. Roughly 20 to 100 ms per chunk is a common range.
Two details are easy to miss:
Backpressure: do not blindly accumulate audio chunks in memory. If STT or downstream agent work slows down, you need bounded queues or you will trade latency for unbounded RAM growth.
Session identity: every message should be tied to a session ID so reconnects, transcript continuity, and avatar state can be managed deterministically.
STT, turn detection, and why partial transcripts matter
In a triage workflow, STT is not just about transcription; it is also about turn detection. You need to know when the user is still speaking, when they paused, and when their utterance is probably complete. Most speech stacks expose partial hypotheses and final segments, and that is the signal you want to drive the agent.
A practical pattern is:
Append partial transcripts to a rolling buffer.
Use endpointing or silence detection to decide when a turn is final.
Trigger the agent only on final turns, unless you explicitly support interruption or early clarification.
Reset or trim state when the user barge-ins while the agent is speaking.
For triage specifically, make the first response structure very crisp. The model should ask one clarifying question at a time, or route quickly to a human if confidence is low. Long, meandering responses create a bad experience even if the transcription quality is good.
Here is a minimal control message shape that keeps the transport clean:
If you are already using a voice stack, the same principle applies: don’t let the STT layer directly “own” the conversation. Let it produce events, then have your agent decide what to do with those events. That is what keeps the system testable.
TTS and avatar synchronization: the real realtime problem
Once the agent has a reply, the next step is streaming speech back to the client. The common failure mode is letting text generation, TTS, and video animation drift apart. If audio starts before the avatar is ready, or if the video uses stale mouth cues, users notice immediately.
To avoid that, treat TTS output as the source of truth for speaking state. The avatar should begin speaking when audio begins, continue until the last audio frame is drained, and stop only when playback is actually complete. If you support interruption, you also need a clean cancellation path so that the avatar can stop mid-utterance without visual lag.
Practically, that means your pipeline should support:
Start-of-speech events so the UI can show “thinking” versus “speaking”.
Streaming audio rather than waiting for a full synthesized clip.
Cancellation when the user interrupts or the agent reroutes.
State reset when the session transitions from listening to speaking and back again.
When debugging, capture timestamps at every boundary: chunk received, transcript emitted, reply generated, first audio byte sent, and final audio byte played. In realtime systems, a few hundred milliseconds of unexpected buffering is enough to make the experience feel broken.
Where Protoface fits: a synchronized avatar surface on top of your voice agent
This is the point where a realtime avatar layer becomes useful. Instead of building your own lip-sync/video pipeline, you can drop in a Protoface avatar surface and keep the rest of your voice stack focused on STT, triage logic, and TTS. The most direct integration path for a voice agent is the LiveKit plugin, which is designed to add a synchronized talking video face to an existing LiveKit agent.
If your agent already runs in LiveKit, the plugin approach keeps the media graph where it belongs: audio flows through the agent, and the avatar stays aligned with the agent’s speaking state. The exact setup varies by your agent and session model, but the pattern is typically “attach avatar to the active voice session, then drive it from the agent’s output stream.” The examples in the plugin repo are the right place to copy from rather than inventing your own timing code. See the plugin and examples in the GitHub organization and the docs at docs.protoface.com.
If you prefer to manage sessions directly, the REST API lets you create and manage avatars and realtime sessions with API-key auth. A minimal create call will look like this conceptually:
The exact request/response fields are documented in the API reference, but the useful part is the model: you create a session once, then bind it to your conversational runtime and keep media and avatar state aligned through that session handle.
Implementation details that matter in production
A few gotchas show up repeatedly:
Latency budget: the sum of websocket buffering, STT delay, model time, TTS startup, and video sync is what the user feels. Optimize the whole path, not individual services in isolation.
Idempotency: reconnects happen. Make session startup safe to repeat so you do not create duplicate conversations or orphaned avatar sessions.
Interruptions: if the user speaks over the agent, stop TTS and mark the current turn as canceled. Do not let the avatar keep talking after the user has taken the floor.
Observability: log session IDs, transcript boundaries, and speak/stop events. Realtime bugs are timing bugs, and timing bugs are impossible to diagnose without timestamps.
If you want to avoid wiring all of this by hand, the Python SDK is useful for programmatic session management, while the LiveKit plugin is the cleaner choice when your voice agent already lives there. The underlying design is the same either way: a conversational runtime produces text and audio events, and the avatar surface renders them with minimal delay.
Conclusion
A good conversational triage avatar is mostly an exercise in disciplined realtime engineering. Keep transport and control separate, stream partial STT results, treat TTS as the authority for speaking state, and make interruption and session identity first-class concerns. Once those pieces are stable, adding a synchronized avatar is straightforward rather than fragile.
If you are implementing this now, start with a minimal FastAPI WebSocket loop, get end-to-end audio timing working, and then layer in avatar synchronization. The public docs at docs.protoface.com and the relevant quickstarts in the Protoface ecosystem are the fastest way to match your code to the supported surfaces without guessing at the API shape.