Building Low-Latency Turn Detection for AI Avatars with Python and FastAPI
Build low-latency turn detection for AI avatars in Python/FastAPI with streaming VAD, ASR partials, and stateful event handling.
Introduction
Turn detection is one of those systems problems that looks trivial until you try to make it feel natural in a realtime voice experience. If your avatar starts speaking too early, it interrupts the user. If it waits too long, the conversation feels laggy and robotic. For AI avatars, that delay shows up visually as well as audibly: the face needs to begin moving at the right moment, not after a noticeable pause.
This post walks through a practical way to build low-latency turn detection in Python with FastAPI, with enough detail to make the implementation useful in a production voice agent or avatar pipeline. By the end, you should be able to design a server-side turn detector, expose it through an HTTP/WebSocket interface, and understand where to place the latency budget so your avatar feels responsive rather than reactive.
What “turn detection” actually means in a voice avatar system
In a realtime conversational system, turn detection answers a simple question: is the human done speaking, or are they still holding the floor?
That question usually cannot be answered by a single signal. In practice, you combine several weak indicators:
Voice activity detection to know when speech is present.
End-of-utterance timing to detect a pause long enough to count as a handoff.
Streaming ASR partials to learn whether the user is still producing words.
Conversation state so you can treat short backchannels like “yeah” or “mm-hmm” differently from a real completion.
The key constraint is latency. If your detector waits for a full final transcript, the assistant will feel sluggish. If it triggers on the first short pause, it will barge in. The useful target is usually “fast enough to feel live, conservative enough to avoid interruptions.” For avatars, that means your face animation should begin only after the system is reasonably confident the user has yielded the floor, but the decision path itself should stay on the order of tens of milliseconds, not hundreds.
Designing a low-latency detector around streaming events
The simplest robust architecture is event-driven:
The client streams audio frames to your backend over WebSocket or WebRTC-adjacent infrastructure.
A VAD layer marks speech start/stop on small windows, typically 10–30 ms.
Your turn detector maintains state per session: current speech segment, last speech timestamp, recent partial transcripts, and an interruptibility flag.
When the detector decides the turn is over, it emits a single “user turn ended” event to the agent or orchestration layer.
That state machine matters more than any single model choice. A decent rule-based detector with good timing usually beats a fancy classifier that is fed stale events. The major tuning knobs are:
Padding after last speech: how long a silent gap you require before considering the user done.
Minimum speech duration: avoid treating clicks, coughs, and one-frame noise as turns.
Partial transcript stability: if the ASR is still revising the last few tokens, delay the turn boundary a bit.
Agent interruption policy: if the assistant is already speaking, decide whether to barge in, hold, or queue the reply.
In production, the detector usually runs as a lightweight async service. Don’t block the event loop with audio processing. Keep the hot path in memory, and treat persistence as secondary. If you need observability, emit timestamps and decisions asynchronously.
A minimal FastAPI implementation
Below is a stripped-down example that shows the shape of the code. It assumes that some upstream component is already feeding you speech activity updates and partial transcripts. The goal here is to make the state transition explicit, not to prescribe a specific VAD or ASR provider.
This is intentionally conservative. A real system will usually add hysteresis, a short debounce, and a notion of “agent is speaking” so you do not generate spurious turn-ends while the user is trying to overlap. If you already have ASR timestamps, use them. They are more reliable than trying to infer everything from a stop event.
Reducing latency without making the detector sloppy
Most turn-detection latency is self-inflicted. The common failure modes are:
Oversized audio chunks: if you buffer 500 ms before processing, your detector cannot react faster than 500 ms.
Serial dependencies: waiting for VAD, then ASR, then a database lookup on the critical path.
Cross-region hops: putting the detector far from the media path.
Heavy per-session work: recreating objects or model sessions on every event.
A few practical rules help:
First, keep the decision path in memory. Session state should live in a process-local cache or a low-latency store, not a relational database. If you need durability, write it out after the decision.
Second, make the detector stateless at the transport boundary and stateful inside the process. FastAPI is a good fit because it gives you straightforward async WebSocket handling, health checks, and HTTP endpoints without forcing you into a heavyweight framework.
Third, separate “confidence to stop listening” from “confidence to start speaking.” These are not the same thing. For example, you may choose a shorter silence threshold to stop capturing audio, but a slightly longer threshold before handing the turn to the avatar. That reduces false positives without materially increasing perceived latency.
Finally, measure the system end-to-end. The metric that matters is not VAD latency alone; it is time from the user’s last meaningful phoneme to the avatar visibly reacting. Put timestamps at audio receipt, speech stop, turn decision, TTS start, and avatar frame start. If you only instrument one layer, you will optimize the wrong thing.
Integration pattern for a realtime avatar pipeline
A useful mental model is that turn detection sits between media ingestion and agent response generation. Once the detector emits a turn boundary, the rest of the stack can do its job: finalize the transcript, generate the reply, synthesize audio, and drive the avatar animation in sync with the response audio.
If you are using a voice agent framework, the cleanest integration is usually a plugin or adapter at the agent layer rather than a bespoke glue service. That keeps the avatar in lockstep with the agent’s speaking state and avoids duplicated turn logic. For teams already on LiveKit, the LiveKit Agents plugin for Protoface is the natural place to wire a synchronized talking face into the agent. The same general pattern applies elsewhere: the detector decides when the human turn ends; the agent decides what to say; the avatar consumes the speaking state and audio timing.
For lower-level debugging, you can also expose the detector as a simple API and inspect decisions directly. A minimal request/response endpoint makes it easier to reproduce edge cases like backchannels, interruptions, or very short utterances.
Exact fields vary by endpoint; use the docs for the current schema. The important part is that your session lifecycle remains explicit, so turn events and avatar playback can be correlated when you debug timing issues.
Where Protoface fits
If you want the avatar side to be the solved problem, Protoface gives you the realtime avatar surface so you can focus on turn logic and agent behavior instead of video-face plumbing. In practice, the most relevant integration for this topic is the LiveKit Agents plugin, which lets a voice agent gain a synchronized talking face with minimal glue code. The plugin lives in the GitHub organization’s examples and package ecosystem, and the broader API and session model are documented at docs.protoface.com.
A typical setup is: your agent handles audio, your turn detector decides when the user is done, and the avatar playback stays synchronized to the assistant’s speaking state. That division of responsibilities is what keeps the system debuggable. You can independently inspect turn decisions, agent prompts, and avatar output instead of treating the whole stack as one opaque realtime blob.
Conclusion
Low-latency turn detection is mostly an architecture problem. Keep the hot path event-driven, maintain per-session state in memory, use small audio windows, and be explicit about the distinction between “speech ended” and “safe to respond.” If you do that, the avatar feels responsive without talking over the user.
If you are building on an existing voice agent stack, start by instrumenting your current turn boundary and measuring its true end-to-end latency. Then tighten the state machine before reaching for more complex models. For the avatar integration and current API shapes, check the docs and the relevant quickstarts in the Protoface GitHub organization.