Getting Started with ASR Streaming for Realtime AI Avatars in FastAPI
Learn streaming ASR in FastAPI with WebSockets, partial transcripts, low-latency audio, and realtime AI avatar integration.
Introduction
Streaming automatic speech recognition (ASR) is the difference between a voice system that feels responsive and one that feels like a batch job. For realtime AI avatars, that distinction matters even more: the avatar should start reacting while the user is still speaking, not after the full utterance has been transcribed.
If you’re building a voice agent in FastAPI, the core problem is usually the same: accept audio incrementally, pass partial hypotheses downstream, and keep the whole pipeline low-latency enough that the avatar’s mouth, the agent’s reasoning, and the user’s expectations stay aligned. By the end of this post, you should have a practical mental model for ASR streaming in a FastAPI app, understand where the latency comes from, and know how to wire a streamed transcript into a realtime avatar flow.
What “streaming ASR” actually means
In a non-streaming setup, the client records audio, sends a complete file, and the server runs ASR after the fact. That is simple, but it adds unavoidable delay and makes turn-taking clumsy.
In a streaming setup, the client sends small audio frames continuously over a long-lived connection. The ASR service consumes those frames and emits:
Partial transcripts while the user is still speaking.
Final transcripts when the end of an utterance is detected.
Sometimes speaker events or voice activity detection signals, depending on the ASR stack.
The key idea is that your backend should treat ASR as a stream of events, not a single response. That means your FastAPI app needs to handle long-lived connections, backpressure, and cancellation cleanly.
FastAPI architecture for low-latency audio
FastAPI is a good fit here because it gives you async request handling and WebSocket support. For realtime voice systems, WebSockets are usually the right transport if the browser or client is sending audio directly to your backend. HTTP chunked upload can work, but it’s not as ergonomic for bidirectional eventing.
A common architecture looks like this:
The client captures microphone audio in small frames, often 20–30 ms each.
Those frames are streamed to a FastAPI WebSocket.
Your server forwards the frames to an ASR engine or agent pipeline.
Partial and final transcripts are pushed back to the client and/or to your agent logic.
When a final transcript arrives, your voice agent can generate a response and the avatar can start speaking as soon as the first audio tokens are ready.
Two practical constraints matter more than framework choice:
Frame size: too large and latency increases; too small and overhead rises.
Turn detection: if end-of-utterance detection is too aggressive, you’ll cut off users; too conservative, and the UI feels sluggish.
A minimal FastAPI WebSocket loop
The example below is intentionally simple. It shows the shape of the problem: accept audio frames, feed them into your ASR layer, and emit transcript events back to the client. The exact ASR SDK calls will depend on your provider.
In a real implementation, you usually want separate tasks: one task reads incoming frames, another forwards them to ASR, and a third emits ASR events. That separation prevents your receive loop from blocking if the downstream service is slow.
Streaming ASR gotchas that show up in production
Most early failures are not model quality problems. They are pipeline problems.
1. Don’t buffer too much audio before sending it. If you accumulate 1–2 seconds of audio before forwarding, you’ve already lost the realtime feel. Aim for small chunks and continuous transport.
2. Be explicit about lifecycle. A streaming ASR session needs start, ongoing frames, and shutdown. If the client disconnects, cancel outstanding tasks and free the session promptly. Leaked sessions will become expensive quickly.
3. Treat partial transcripts as unstable. Don’t commit business logic on the first partial hypothesis unless you’re intentionally building speculative UX. Partial text can and will be revised.
4. Separate transcript state from conversation state. You may receive multiple final segments for one turn, or a final segment followed by silence detection. Keep a small state machine rather than assuming a single “message complete” event.
5. Measure end-to-end latency, not just ASR latency. In a voice avatar system, the user experiences the sum of capture delay, network transit, ASR, LLM or intent processing, TTS, and video synthesis or lip-sync alignment.
Wiring ASR into a realtime avatar pipeline
Once you have streaming transcripts, the avatar path is straightforward conceptually:
User speaks.
ASR produces partial text, then a final transcript.
Your agent decides whether to interrupt, ask a clarifying question, or continue the current turn.
Text-to-speech or a realtime speech model generates audio.
The avatar renders synced talking video from that audio.
The important point is that the avatar should not wait for a “conversation complete” blob. It should be attached to the same realtime turn model as the voice agent. If the agent begins speaking after 300 ms instead of 2–3 seconds, the avatar feels alive rather than scripted.
Also, keep in mind that lip-sync quality depends on stable audio timing. If you batch audio on the server, you can create visible drift between audio and video. Preserve the stream semantics all the way through the stack.
How Protoface fits in
This is where Protoface is useful: it gives you a developer-facing realtime avatar layer that can sit on top of your streaming voice stack. If you already have FastAPI handling ASR, you can focus on the transcript and turn logic, then attach the avatar to the response path instead of building video-face synchronization yourself.
For server-side integrations, the documented surfaces are the REST API and the Python SDK; for LiveKit-based agents, the plugin is the shortest path. The plugin is especially relevant if your voice agent already runs in LiveKit and you want to drop in a synchronized talking face with minimal glue code. See the plugin repo for examples and the package on PyPI: GitHub examples and PyPI.
A representative REST call looks like this, with the exact request schema defined in the docs:
And a Python SDK flow will typically look like this at a high level:
If you want the detailed request/response shapes, authentication examples, and lifecycle semantics, use the docs at docs.protoface.com. That’s the right place to confirm the exact fields rather than guessing from snippets.
FastAPI implementation notes that save time later
When you put this together, a few design choices are worth making up front:
Use async end to end for websocket handling and downstream API calls where possible.
Keep audio transport binary; don’t base64-encode frames unless you have to, since it adds overhead.
Tag every session with a stable identifier so you can correlate audio, transcripts, avatar events, and logs.
Budget for silence detection; it is not optional in conversational systems.
Test with real microphones and network jitter; localhost is too clean to reveal the problems you’ll hit in production.
If you are building on top of a larger agent stack, the same principles apply. Whether the avatar sits in a browser iframe, a LiveKit voice agent, or a custom FastAPI backend, the underlying requirement is unchanged: keep the user’s speech, the transcript stream, and the avatar’s response tightly coupled in time.
Conclusion
Streaming ASR is mostly about respecting the realtime nature of speech. Send small audio frames continuously, handle partial and final transcripts separately, keep your connection lifecycle explicit, and avoid buffering your way into latency. Once that pipeline is stable, the avatar layer becomes much easier to integrate because it can react to the same live turn events your voice agent already uses.
If you want to go from this pattern to a working implementation, start with the FastAPI websocket skeleton above, validate your ASR event flow, and then connect the avatar surface that fits your stack. The docs at docs.protoface.com and the quickstarts in the GitHub repo linked from the project README are the fastest way to move from concept to working prototype.