Debugging Turn-Taking Issues in Agora-Powered Voice Avatar Applications
Debug turn-taking bugs in Agora voice avatars with event tracing, barge-in cancellation, and audio-driven lip-sync.
Introduction
Turn-taking bugs are one of the easiest ways to make a realtime voice avatar feel “off” even when the speech model, the TTS, and the video rendering are all individually working. The usual failure mode is not a crash; it is subtle conversational drift: the avatar starts speaking over the user, cuts itself off too early, waits too long, or keeps lip-syncing after audio has already stopped. In a WebRTC-based voice avatar stack, those symptoms almost always come from event ordering, buffering, or duplicated state transitions across your voice agent, transport layer, and avatar renderer.
This post walks through how to debug those issues systematically. By the end, you should be able to identify where turn ownership is being lost, instrument the pipeline correctly, and choose the right fix instead of masking the problem with arbitrary delays. The examples use a Protoface-integrated voice agent, but the debugging approach applies to any realtime avatar application built on streaming audio and low-latency media transport.
Start by modeling the turn pipeline explicitly
The first mistake is treating “the agent is talking” as a single boolean. In practice, turn-taking spans several distinct states:
Input capture: the user begins or stops speaking.
ASR/VAD decision: the system decides whether the user currently holds the floor.
Agent generation: the LLM or dialog manager starts producing the response.
TTS/audio playout: synthesized speech is streamed to the transport.
Avatar animation: the talking head is driven by the outgoing audio timeline.
Those layers do not always transition at the same instant. If your avatar starts animating on “agent text started” but audio hasn’t actually begun, the mouth can move ahead of sound. If your turn detector releases the floor on the first short pause, the agent may barge in during a hesitation. If your video renderer depends on audio timestamps but the stream is buffered in a queue, lip sync will lag under load.
When debugging, log the state transitions, not just the raw audio. The most useful events are:
user speech start / stop
barge-in detected
agent turn started / cancelled
first synthesized audio frame enqueued
first audio frame played
avatar speaking start / stop
Once you can see those timestamps in one place, the failure mode usually becomes obvious.
Common failure modes in Agora-style realtime voice stacks
If you are using a low-latency conferencing or media transport layer, the two most common classes of issues are duplicate authority and late state propagation.
Duplicate authority means more than one component thinks it owns turn-taking. For example, an upstream conversation manager decides the user is done speaking, while a downstream TTS or transport event fires a second “agent started” transition. The result is typically a double-start, a clipped first phoneme, or an avatar that begins moving twice as fast because two overlapping playback sessions were created.
Late state propagation happens when the audio path has moved on but the avatar or UI is still reacting to stale state. In a streaming system, this often appears when you infer speaking state from received text instead of actual audio playback, or when you use a local timer rather than the media pipeline’s real playout events. A 150 ms mismatch is enough to make lip sync look wrong to users.
There are a few specific bugs worth checking early:
VAD hangover too short: the system decides the user finished speaking during micro-pauses, so the agent interrupts.
Cancellation not propagated: a user barges in, but the agent’s already-generated TTS keeps playing for another second.
Out-of-order events: a “stop speaking” event arrives after a new “start speaking” event because state updates are handled asynchronously without sequencing.
Queue buildup: the avatar is driven from buffered audio that is already stale by the time it is rendered.
The practical fix is to assign one component as the source of truth for floor ownership and make every other component react to it. Do not let the UI, the agent loop, and the avatar independently infer whether the assistant is speaking.
Instrument the timing, not just the content
For realtime debugging, content logs are useful but not sufficient. You also need time deltas between state changes. A simple pattern is to annotate each turn with a monotonically increasing sequence number and timestamps at every transition. That lets you detect reordering, cancellation races, and unexpectedly long gaps.
If you already have structured logs, add the media pipeline stage to each record: ASR, agent, TTS, transport, avatar. Then compare the duration between “first audio enqueued” and “first audio played.” If that delta changes under load, the problem is in buffering or transport. If it stays constant but the avatar still looks late, the problem is in how the animation layer is keyed off playback state.
Another good trick is to log the reason the assistant lost the floor. You want to distinguish:
natural end-of-turn
explicit barge-in
timeout
transport disconnect
agent cancellation due to new user speech
These are operationally different and should not collapse into one generic “stopped speaking” event. If they do, you will end up tuning the wrong threshold.
Fixing barge-in and end-of-turn logic
Most turn-taking bugs come down to one of two problems: you are detecting turn boundaries too aggressively, or you are not cancelling speech fast enough.
For end-of-turn detection, resist the urge to use a single silence threshold for every conversation. Short acknowledgements, interrupted speech, and backchannels all need different behavior. A robust implementation usually combines:
VAD for speech activity
pause duration for likely turn completion
semantic completion for cases where the user’s sentence is clearly unfinished
For barge-in, the key is immediate cancellation across the entire pipeline. When the user starts speaking, you generally need to stop three things:
future agent token generation
pending TTS synthesis and playback
avatar speaking state that depends on that playback
If you only cancel generation but let queued audio continue, the avatar will keep moving even though the assistant is logically silent. If you only stop the avatar, the user will still hear the tail of the prior answer, which makes the whole interaction feel unresponsive.
A practical rule: do not infer “assistant is talking” from text generation state. Use the actual media lifecycle. The source of truth should be the audio pipeline, because that is what the user hears and what the avatar should mirror.
How this maps to a Protoface-integrated agent
If you are using a LiveKit voice agent, the cleanest place to keep the avatar synchronized is in the agent layer itself rather than bolting animation onto the browser after the fact. The livekit-plugins-protoface plugin is designed for that: it drops a talking face into the voice agent so the avatar follows the same streamed audio that drives the conversation. That reduces the chance of the avatar getting ahead of, or behind, the real media timeline.
A minimal integration looks like this in spirit:
The important debugging point is not the exact constructor shape; it is the attachment point. You want the avatar to consume the same output stream that is actually being sent to the user. If you instead drive animation from a separate “assistant started speaking” event, you reintroduce drift.
If you need to create or inspect sessions programmatically, the REST API is the place to look. The docs at docs.protoface.com cover the exact payloads, but the pattern is the standard one: create a session, bind it to your agent flow, and use the returned session metadata to connect the avatar to the correct realtime conversation.
For developers integrating at the agent layer, the PyPI package and examples in the plugin repository are usually the fastest path to reproducing and fixing timing bugs. See the plugin examples in the relevant repo: https://github.com/protoface-ai/protoface-quickstart-agora. Even if your transport differs, the same diagnostic workflow applies: trace the turn lifecycle from VAD to playout, then compare those timestamps with avatar state changes.
Debugging checklist that actually helps
When a turn-taking issue shows up, I usually work through the following in order:
Verify whether the bug is audio timing, avatar timing, or both.
Log start/stop events with a sequence number and monotonic timestamps.
Confirm there is a single owner for turn state.
Check whether barge-in cancels queued audio, not just future generation.
Compare first-audio-enqueued vs first-audio-played latency under load.
Make sure the avatar listens to playback state, not token generation state.
If the issue only appears intermittently, it is usually a race. If it appears consistently whenever the user interrupts, it is usually a cancellation problem. If the avatar lags only on slower networks, it is usually buffering or a stale event source.
Conclusion
Turn-taking bugs in realtime voice avatars are rarely about one broken subsystem. They are usually caused by weak ownership of turn state, missing cancellation paths, or avatar animation that is detached from actual audio playout. The fix is to instrument the pipeline end to end, make one layer authoritative for floor ownership, and drive the avatar from the same media lifecycle the user hears.
If you are building on Protoface, the LiveKit plugin and the API surface are both useful entry points depending on where you want to attach the avatar. Start with the docs, reproduce the timing issue with structured logs, and work backward from the playback timeline rather than the text generation timeline. For implementation details and quickstarts, see docs.protoface.com and the examples linked from the quickstart repository.