A Developer’s Guide to Turn-Taking Logic with Agora for Voice and Video AI Agents
Developer guide to turn-taking logic for voice/video AI agents: VAD, barge-in, latency, state machines, and synced avatars.
Introduction
Turn-taking is the hard part of building a voice or video AI agent. Audio streaming is the easy part. The real engineering work is deciding when the agent should listen, when it should speak, and when it should stop speaking because the user has started talking again. If you get this wrong, the system feels laggy, interrupts people, or talks over them.
This post covers the practical mechanics: how turn-taking works in realtime agents, how to think about speech activity and interruption, where latency enters the loop, and how to wire the logic into a production system with a synchronized avatar. By the end, you should be able to design a turn controller that behaves predictably across voice-only and video/avatar experiences.
What turn-taking actually means in a realtime agent
In a conversational system, “turn-taking” is not just detecting whether someone is speaking. It is a small state machine that sits between three subsystems:
Input: microphone audio from the user.
Reasoning: STT, LLM, and policy logic that decides whether to respond.
Output: TTS or a synthesized avatar stream that plays the response.
The controller’s job is to coordinate those systems with enough timing precision that the interaction feels human. In practice, you need at least four states:
Idle: no one is speaking.
Listening: the system is collecting a user utterance.
Thinking: the agent has enough input to start reasoning, but hasn’t started output yet.
Speaking: the agent is streaming audio and, if present, a synced face.
A common mistake is to make the state machine binary: “user speaking” or “agent speaking.” That works in demos, but breaks down once you have partial speech, pause gaps, barge-in, and streaming generation. Real conversations contain overlap, and your controller needs explicit rules for overlap.
Detecting speech without overreacting
The simplest turn detector uses voice activity detection (VAD): if the input audio crosses a threshold, treat it as speech. That is usually good enough to know when to open a microphone gate, but not enough to decide turn boundaries on its own.
Why not? Because speech is bursty. People pause mid-sentence, inhale, say “um,” or produce low-energy consonants that look like silence. If you commit to a turn end on every short gap, your agent will interrupt too early. If you wait too long, you add dead air after every response.
A better controller typically combines:
Speech onset detection: start listening quickly when the user begins talking.
End-of-turn delay: require a short silence window before finalizing the user turn.
Confidence thresholds: only act on speech segments with enough signal quality.
Contextual suppression: reduce sensitivity while the agent is speaking if you want to support barge-in differently from background noise.
Those rules are more important than the exact model you use underneath. In production, a slightly conservative end-of-turn delay often feels better than “instant” responsiveness, because the latter tends to clip user utterances and force repeated prompts.
Managing barge-in and interruption cleanly
Barge-in is the case where the user starts speaking while the agent is still speaking. For voice agents, this is usually a feature, not a bug. Users often interrupt because the answer is obvious, too long, or already wrong. The trick is to handle interruption without creating a feedback loop.
The basic flow looks like this:
Agent is speaking and streaming output.
VAD or another turn detector sees user speech onset.
Controller stops TTS playback and cancels any remaining output generation.
Controller records the new user utterance as the next turn.
There are two implementation details that matter:
Cancel at the source: stopping audio playback is not enough if the model is still generating tokens or your avatar stream is still being fed frames. You need to terminate the active generation pipeline.
Preserve turn context: if the user interrupts after hearing only part of the answer, the assistant should still know what it was saying. In practice, you often keep the partial assistant turn in conversation state and mark it as interrupted rather than discarding it entirely.
For video avatars, barge-in needs one more step: the face animation must stop in sync with the audio. If audio stops but the mouth keeps moving for another second, the illusion breaks immediately. Any avatar layer you add should be driven from the same turn state as the voice stack.
Latency budget: the part you actually feel
Users do not perceive “model latency” abstractly. They perceive delays at turn boundaries. A good answer that begins 1.2 seconds late feels worse than a mediocre answer that begins in 250 ms.
When you design turn-taking, break the latency budget into pieces:
Input capture latency: microphone buffering and transport.
Turn-end detection latency: silence window or end-of-speech inference.
Inference latency: STT, LLM, and any routing logic.
Output startup latency: first TTS chunk or first video frame.
Streaming helps because it moves the visible start of the response earlier. The agent does not need to finish reasoning before it starts speaking. But streaming also raises a control problem: if you start output too early and then detect that the user was still talking, you need to stop immediately. That means your controller should be able to cancel downstream work fast, not just queue it.
A practical heuristic is to treat the first 200–500 ms after a user stops speaking as the “decision window.” Use it to finalize the turn if the user truly stopped, but keep the system primed so the response can start as soon as the controller is confident. The exact numbers depend on your STT quality, domain, and acceptable overlap.
Designing the controller as a state machine
For most applications, turn-taking is easiest to reason about as an event-driven state machine rather than as a pile of callbacks. The events are typically:
user_speech_starteduser_speech_endedassistant_response_readyassistant_output_startedassistant_output_stoppedinterruption_detected
From there, your controller can enforce a few rules:
That pseudocode is intentionally simple. Real systems add timers, confidence thresholds, and debounce logic. But the shape remains the same: events in, state transitions out. If you keep the controller explicit, it becomes much easier to test edge cases like short utterances, overlapping speech, and forced cancellations.
One more practical point: log state transitions, not just raw audio events. If a user reports that the agent “kept talking over me,” you want to see whether your system detected speech onset late, ignored an interruption due to a threshold, or failed to cancel output.
Where Protoface fits: adding a synchronized face without changing your turn logic
This is where Protoface is useful: it gives you a realtime avatar layer that stays synchronized with your voice agent, so your turn controller can remain the source of truth for when speech starts and stops. The avatar should follow the same state transitions as the audio stack, not invent its own independent timing.
If you are already running a LiveKit agent, the livekit-plugins-protoface plugin is the most direct way to attach a talking face to the agent. The key idea is simple: once your agent enters SPEAKING, the avatar should animate; when you cancel output on barge-in, the avatar should stop immediately.
If you want to create or manage avatars and sessions directly, use the REST API with an API key from your backend. That keeps credentials off the client and lets you control lifecycle centrally.
For implementation details, payload fields, and current integration patterns, the documentation is the source of truth: docs.protoface.com.
Common gotchas
Do not tie turn completion to one subsystem. STT end-of-speech, LLM readiness, and TTS playback are separate concerns.
Do not let the avatar drift from the audio state. If the audio is canceled, the visual response must be canceled too.
Do not expose API keys in the browser. If you need a website embed, use a customer-managed iframe embed so the browser never sees your secret.
Do not over-tune silence thresholds from one conversation. Turn behavior varies by accent, microphone, room noise, and domain.
Also remember that a good conversational UX is not always the lowest-latency one. In customer support, a slightly slower but more reliable turn boundary is often preferable to a twitchy system that misreads pauses as the end of a sentence.
Conclusion
Turn-taking is the control plane of a voice or video agent. Get it right and the system feels responsive, polite, and coherent. Get it wrong and even a strong model will feel broken. The core work is to treat speech detection, interruption, and output cancellation as explicit state transitions with measurable latency budgets.
If you are adding a realtime avatar to an existing agent, keep the avatar synchronized with the same turn controller that drives audio. That separation of concerns scales better than bolting animation logic onto the edges of your app.
For integration examples and current API details, start with the docs at docs.protoface.com, and use the relevant quickstart or SDK repo when you want a concrete implementation path.