How to Use Agora Voice Activity Detection for Turn-Taking in Realtime AI Avatar Apps
Use Agora VAD for low-latency turn-taking in AI avatar apps with state-machine tuning, barge-in, and audio-sync tips.
Introduction
Turn-taking is one of the smallest parts of a voice agent stack and one of the easiest places to get wrong. If you start the avatar too early, you cut off the user. If you wait too long, the interaction feels laggy and unnatural. In realtime AI avatar apps, that timing problem is usually handled by voice activity detection (VAD): a low-latency signal that tells you when someone is likely speaking, when they stop, and when it is safe to hand the floor back to the agent.
This post shows how to use Agora VAD as the basis for turn-taking in a realtime avatar app. By the end, you should understand where VAD fits in the media pipeline, how to tune it for conversation rather than simple silence detection, and how to connect it to an avatar so the face and voice stay synchronized. I’ll also show where Protoface fits when you want the avatar side of this to be a solved problem instead of a custom media project.
What VAD does in a realtime avatar system
In a conversational app, VAD is not just “someone is speaking.” It is a control signal for state transitions:
User speaking: suppress agent output, keep listening, and avoid interrupting.
User stopped speaking: wait for a short post-speech hangover, then decide whether to respond.
Agent speaking: optionally mute or de-prioritize input processing, depending on your interruption model.
For avatar apps, the same signal also drives visuals. Mouth motion, gaze behavior, and talking-state indicators need to match the audio state. If the avatar starts “talking” before the agent’s first audio frame arrives, or keeps animating after the agent has stopped, the illusion breaks quickly.
Agora’s VAD is useful here because it is designed for realtime media sessions rather than offline transcription. You typically use it as an event source in your audio pipeline: frames come in, VAD classifies them, and your app transitions between states like idle, user_speaking, and agent_turn.
Build the turn-taking state machine first
Before you wire in any SDK, define the states and the events that move between them. A simple version looks like this:
idle: no active speaker detected.
listening: the system is ready to accept input.
user_speaking: VAD has detected speech above threshold for long enough to count as a turn.
waiting_to_respond: user speech ended, but you’re applying a short debounce to avoid responding mid-thought.
agent_speaking: the assistant is generating or playing audio; avatar should be in talking mode.
The key detail is that VAD should not directly trigger “respond now.” It should only tell you that speech started or stopped. Your app still needs a small amount of policy:
Activation threshold: how much speech energy or confidence counts as speech.
Start debounce: how long to confirm a speech onset before switching state.
End hangover: how long to wait after speech drops before declaring the turn ended.
Barge-in policy: whether user speech can interrupt the agent.
That policy matters more than the specific VAD algorithm. In practice, most “bad turn-taking” bugs are really policy bugs: thresholds too tight, end delay too short, or no handling for brief pauses inside a sentence.
Wire Agora VAD into your media loop
The mechanics are usually the same whether your audio arrives from WebRTC, a server bridge, or a browser capture stream: sample frames, feed them to VAD, and react to events. The exact Agora API varies by SDK, so treat the following as structural pseudocode rather than a copy-paste implementation.
That loop looks basic, but two details are easy to miss:
Drive the avatar from the same state machine as audio playback. Do not infer talking solely from websocket messages or LLM tokens. Use actual audio start/stop when possible.
Keep VAD and ASR separate. VAD decides who has the floor. ASR decides what was said. Mixing the two often adds latency and makes interruption handling worse.
Tuning for conversation, not just silence
Most VAD systems are easy to demo and hard to productionize. The defaults may work in a quiet room, but turn-taking in a consumer app has a different set of failure modes: fan noise, overlapping speech, short interjections, and users who pause mid-thought.
These are the knobs I would look at first:
Frame size: smaller frames reduce detection latency, but increase callback frequency and sensitivity to noise.
Start threshold: too low and you get false positives; too high and the agent feels sluggish.
End threshold / hangover: too short and you cut off natural pauses; too long and the bot waits awkwardly.
Cooldown after agent speech: useful if you do not want the avatar to react to its own echo or room bleed.
For a voice avatar, I usually think in terms of perceived latency, not raw model latency. If your VAD fires in 50 ms but your policy waits 700 ms to avoid false ends, the user experiences 700 ms. That is the number that matters.
You also need to decide what to do with overlap. In an assistant app, it is common to let the user barge in and immediately stop the agent. In a structured flow, like a payment or verification step, you may want to ignore barge-in until the system reaches a safe interruption point. That choice belongs in the app, not in VAD itself.
Keep the avatar synchronized with the turn state
An avatar is most believable when its visible speaking state matches the audio pipeline precisely. That means the avatar should enter a talking state when the first frame of synthesized speech is about to play, not when the language model starts thinking. Likewise, it should leave talking state after audio playback ends, not when text generation completes.
For turn-taking, I recommend three avatar states only:
listening while user speech is active or the system is awaiting input.
thinking only if you explicitly want a non-speaking “processing” state.
talking when agent audio is actually playing.
That simplicity helps because the avatar becomes a reflection of the media state machine, not a second, loosely correlated model of the conversation. If you have a lip-sync pipeline, it should consume the same outgoing audio stream used for playback. The talking animation then naturally stays aligned with the audio clock.
One practical rule: when VAD reports the user has started speaking, stop the agent at the next safe audio boundary, then switch the avatar to listening. Don’t wait for a full response to finish if your UX allows interruption. But do avoid hard-cutting in the middle of a phoneme unless your backend supports it cleanly.
Where Protoface fits
If you are building the voice and turn-taking logic yourself, the avatar layer is still a separate problem: streaming face video, synchronizing lip motion to audio, and exposing a control surface to the rest of your app. That is the part the Protoface docs cover, and it is where the LiveKit integration is especially useful.
For example, if you already have a LiveKit voice agent and you want it to gain a synchronized talking face, the livekit-plugins-protoface plugin lets you attach a Protoface avatar to the agent rather than building a custom media bridge. Your VAD logic still decides when the agent should speak or stop; the plugin handles the avatar presentation layer.
If you prefer to manage sessions directly, the REST API and Python SDK give you programmatic control over avatars and realtime sessions. That is useful when your app needs to create an avatar session on demand, persist session metadata, or connect your own backend state machine to the avatar lifecycle. Keep the audio policy in your app, and use the avatar API to reflect the result.
Common pitfalls
A few failure modes show up repeatedly in production:
Using VAD as a transcript substitute: it is not. It only tells you when speech is likely happening.
Coupling speech end to response start too tightly: natural pauses inside a sentence will make the agent interrupt the user.
Ignoring echo: if your client is playing audio over open speakers, VAD can falsely detect the agent’s own voice unless you handle echo cancellation or playback gating.
Driving visuals from text generation: the avatar should track audio, not tokens.
Also remember that low-latency turn-taking is a distributed systems problem. Browser capture, network jitter, server inference, TTS buffering, and avatar transport all contribute to the end result. The best VAD in the world cannot compensate for a pipeline that adds unnecessary buffering.
Conclusion
For realtime avatar apps, Agora VAD is best thought of as the floor-control signal in a conversation state machine. Use it to detect when the user starts and stops speaking, add a small amount of policy for debounce and barge-in, and keep the avatar synchronized to the actual audio state. That gives you fast, natural turn-taking without conflating speech detection, transcription, and presentation.
If you want to focus on the conversation logic and not on the avatar plumbing, Protoface provides the avatar/session layer and integrates cleanly with voice-agent stacks. Start with the docs at docs.protoface.com, and use the LiveKit plugin or the Python SDK depending on where your agent already lives. The quickest path to a good result is usually: keep VAD simple, make the state machine explicit, and make sure the avatar always follows the audio.