Debugging Delayed Turn-Taking in LiveKit Voice Avatar Apps
Debug LiveKit voice avatar turn-taking delays with timestamps, endpointing tuning, TTS startup, and avatar sync checks.
Introduction
Delayed turn-taking in a live voice avatar app usually looks like this: the user stops speaking, the agent pauses too long before responding, and the avatar face keeps moving as if it heard something that no longer exists. The root cause is often not “the avatar” itself. It is usually the timing pipeline around speech activity detection, transcript finalization, agent reasoning, TTS start, and video sync.
This post is about debugging that pipeline systematically. By the end, you should be able to identify where the delay is introduced, measure it with a few timestamps, and make practical trade-offs between responsiveness and correctness in a realtime voice agent with a synchronized avatar from Protoface.
What “turn-taking” actually means in a live voice system
In a voice agent, turn-taking is the control logic that decides when the system should start listening, when it should consider the user done, and when it should begin producing its own response. In practice, the sequence looks roughly like this:
Audio from the user is captured and streamed.
Voice activity detection, endpointing, or an ASR finalization heuristic decides the user has stopped.
The agent generates a response, often after tool calls or LLM latency.
TTS starts producing audio.
The avatar video begins lip-synced playback aligned to that audio.
Any one of those steps can add delay. The common mistake is to attribute all latency to the model. In reality, many “slow response” bugs are caused by conservative endpointing, buffering, or waiting for a final transcript when the system already has enough evidence to respond.
For debugging, separate the problem into two questions:
When did the system decide the user was done?
How long after that did the first response audio actually leave the server?
If those timestamps are close, the issue is upstream in speech detection or orchestration. If they are far apart, the issue is in agent logic, network hops, or TTS startup. If the audio starts quickly but the avatar appears late or out of sync, the issue is in media transport or render timing.
Instrument the pipeline before changing behavior
Do not start by tuning thresholds blindly. Add timestamps at every boundary you control. You want to know the latency contribution of each stage, not just the total.
A minimal set of markers is:
user_speech_end_detectedasr_final_receivedagent_response_starttts_first_audio_chunkavatar_playback_start
Even if your stack hides some layers, approximate them. For example, if you can’t instrument the ASR internals, record when your application receives a finalized transcript versus when your agent decides to speak. That gap is often the one users feel.
A useful debugging pattern is to log a correlation ID per turn and print deltas rather than raw timestamps:
That one line usually tells you where the delay lives. If the gap from user_speech_end_detected to agent_response_start is the largest, you likely have an endpointing or reasoning problem. If the gap from agent_response_start to tts_first_audio_chunk is large, the bottleneck is model or TTS startup. If avatar_playback_start lags significantly behind the first audio chunk, the media path is buffering too much or the client is waiting for a synchronization boundary you do not need.
Also check whether you are collecting these metrics on the same clock domain. Server logs and browser logs can drift enough to make the story look worse than it is. For cross-device debugging, prefer relative per-turn deltas and, if needed, send a server-generated turn ID down to the client.
Endpointing is the most common source of “late” responses
For a live conversational app, the endpoint detector has to answer a hard question: is the user pausing, or are they done? If you set the system to wait for strong confidence before ending a turn, you reduce barge-in errors but increase latency. If you trigger too early, you cut off users and create awkward overlaps.
This trade-off is usually controlled by some combination of:
minimum silence duration before end-of-turn
minimum speech duration required to count as a turn
energy threshold or VAD sensitivity
whether partial ASR text can trigger response planning
When turn-taking feels delayed, look for these failure modes:
Silence window too long. The system waits 800–1200 ms of silence to avoid false positives, which is reasonable for dictation but often too conservative for conversation.
ASR finalization is blocking response planning. The agent waits for a fully stabilized transcript even though a partial transcript is already enough to answer.
Punctuation-based end detection is too slow. If your logic depends on a period in the transcript, you are effectively outsourcing turn-taking to the model’s text formatting.
Tool-call gating is too strict. The system refuses to speak until all tool results are back, even when it could acknowledge the user and continue.
A practical mitigation is to decouple “I think the user is done” from “I have perfect transcript certainty.” Let the response planner begin as soon as the end-of-turn heuristic is sufficiently confident, even if the final transcript is still being polished in the background. For many apps, shaving 200–400 ms here makes the interaction feel much more immediate.
Voice agents need overlap control, not just faster endpointing
Low latency alone is not enough. A good voice agent also needs overlap control so it can handle interruptions, corrections, and backchanneling. If the agent is too eager to speak, it will trample the user. If it is too conservative, it will feel inert.
The core design choice is whether your system supports:
hard end-of-turn only: respond after the user is clearly done;
speculative response start: begin planning or even synthesizing before final ASR;
barge-in: stop or truncate agent speech when the user starts talking again.
For debugging delayed turn-taking, inspect whether your app is artificially serializing these states. A common anti-pattern is:
wait for final ASR
wait for LLM completion
wait for TTS buffer to fill
start avatar playback
That is robust, but it makes the conversational loop feel sticky. A better approach is to overlap non-conflicting work where possible. For example, you can begin generating the response as soon as the user’s utterance is confidently “done enough,” and stream TTS audio as soon as the first tokens are available. That reduces perceived latency even if the end-to-end compute time does not change.
Be careful, though: overlapping work changes failure modes. If the user continues speaking and your turn-taking heuristic was too aggressive, you may need to cancel the in-flight response and restart. So whenever you optimize latency, verify that cancellation paths work cleanly and that your agent does not produce a partial response after the user has barged in.
What to check in the avatar/video layer
If the spoken response sounds prompt but the avatar feels delayed, the issue is probably not turn-taking. It is either audio/video alignment or client-side buffering.
For a realtime avatar, the face animation should generally track the audio start closely. Small offsets are normal, but large ones are usually a sign that the playback pipeline is waiting for too much data before starting. In a WebRTC-style flow, that can happen if the client or relay accumulates a buffer to avoid underflow. In a browser, it can also happen if the video element is not being fed frames fast enough or the page is competing for main-thread time.
Check these points:
Audio-first vs synchronized start. If the avatar waits for both audio and video readiness, a slow video frame can delay the whole response.
Excess buffering. A larger jitter buffer improves smoothness, but it also adds startup delay.
Client rendering contention. Heavy DOM work or animation on the page can make the avatar appear to “wake up” late.
Clock drift between audio and video pipelines. If timestamps are aligned incorrectly, the mouth movement can lag the speech even when network latency is fine.
The debugging move here is the same: measure the gap between the first response audio chunk and the first visible avatar frame. If that gap is small, your avatar stack is probably healthy and the user is reacting to speech timing instead. If that gap is large, focus on transport and rendering.
How Protoface fits: adding a synchronized face without reworking your voice stack
If you already have a LiveKit voice agent and want a synchronized talking face, the fastest path is the LiveKit plugin. The plugin drops an avatar into the agent flow so you can debug turn-taking and visual sync without building a separate media pipeline from scratch. The relevant examples live in the plugin repository, and the setup details are documented in the integration guide on docs. For Pipecat-based stacks, there is also a dedicated integration path in the Pipecat docs.
For example, if you are debugging a LiveKit agent, keep the agent logic unchanged and focus on where the avatar joins the stream. That lets you isolate whether the delay is in agent response timing or in avatar presentation. If the response audio is prompt but the face is late, you know to look at the video join and playback path instead of the LLM.
For direct API-driven workflows, you can also manage avatars and sessions via the REST API. A typical request shape looks like this:
The exact payload depends on the endpoint and the session model you are using, so treat this as a shape example rather than copy-paste code. The important part for turn-taking debugging is that you can create, inspect, and reproduce sessions deterministically while you measure latency.
If you prefer reading the source or examples first, the plugin repository is a good starting point: https://github.com/protoface-ai/protoface-quickstart-openai-realtime and the general documentation is at https://docs.protoface.com.
Practical debugging workflow
When a user reports “the avatar responds late,” run through this sequence:
Measure the turn boundary. Log when your system decides the user is done.
Measure response start. Log when the agent begins generating speech, not when it finishes.
Measure TTS startup. First audio chunk matters more than full synthesis time.
Measure avatar playback start. Confirm the face starts when speech starts.
Test barge-in. Make sure user interruptions cancel or preempt speech cleanly.
Reduce one buffer at a time. Do not change endpointing, ASR, TTS, and rendering together.
That last point is worth emphasizing. Realtime systems are full of hidden queues, and it is very easy to “fix” one delay by adding another. If you shorten silence detection, for example, you might make turn-taking faster but increase false starts. If you reduce video buffering, you might improve startup latency but introduce stutter. The right balance depends on the product: a support bot, an interview assistant, and an NPC do not need identical timing.
Conclusion
Delayed turn-taking is usually a pipeline problem, not a single bug. The fastest way to debug it is to put timestamps on every stage, separate endpointing from generation from media startup, and tune one boundary at a time. Once you can see where the delay comes from, you can decide whether to make the system more aggressive, more conservative, or simply better synchronized.
If you are integrating a live avatar into a voice agent, start with the docs at https://docs.protoface.com, then use the LiveKit plugin or SDK path that matches your stack. The key is to keep the conversational timing observable while you iterate, because once you can measure the turn, you can usually fix it.