Measuring Turn-Taking Latency in Voice + Video AI Avatar Systems
Learn how to measure turn-taking latency in voice and video AI avatar systems by instrumenting VAD, ASR, TTS, rendering, and playback.
Introduction
Turn-taking latency is the time between when a user finishes speaking and when the system starts responding in a way that feels coherent. In a text chat, that gap is mostly invisible. In a voice or video avatar system, it is the product. If the pause is too long, the conversation feels broken; if the system interrupts too early, it feels rude or unstable.
In a realtime avatar stack, turn-taking latency is not one number. It is the sum of several smaller delays: audio capture, speech recognition, endpointing, agent inference, text-to-speech or audio generation, avatar rendering, encoding, network transport, and playback. The practical goal is to measure each stage separately, understand where the time goes, and decide which trade-offs are acceptable for your product.
This post shows how to measure turn-taking latency in a way that is useful for debugging and optimization. By the end, you should be able to define your latency budget, instrument the right events, and spot the usual failure modes in voice + video AI avatar systems.
Define the latency you actually care about
People often say “latency” when they really mean one of three different things:
User-to-agent latency: time from the user stopping speech to the agent beginning its response.
User-to-usable-output latency: time until the first audible or visible response is produced.
End-to-end conversational latency: time until the response is fully rendered and stable on the client.
For turn-taking, the first one is usually the most important. That is the gap a human notices most directly. If the avatar speaks 150 ms after the user stops, it feels responsive. If it takes 900 ms, users start wondering whether the system heard them.
Measure from the end of user speech, not from the last network packet or from the agent receiving a “final” transcript. In real systems, those are related but not identical. The user’s perceptual stop point is what matters, and it is usually approximated by voice activity detection, speech endpointing, or a manual label in offline analysis.
Break the pipeline into measurable stages
A realtime voice/avatar pipeline usually has these stages:
Audio capture and transport: microphone audio is packetized and sent over WebRTC or a similar realtime transport.
Speech detection and endpointing: the system decides whether the user is still speaking and when the utterance is complete.
ASR: speech recognition converts the utterance to text, often streaming partial results before a final transcript.
Agent inference: the LLM or policy model decides how to respond.
Response generation: text-to-speech, audio generation, or token streaming begins.
Avatar synthesis: video face motion, lip sync, and frame generation are produced.
Client playback: the browser or app receives and renders the response.
The important thing is not just measuring the total, but attaching timestamps to boundary events. For example:
user_speech_endvad_endpointtranscript_finalagent_first_tokentts_first_audioavatar_first_frameclient_playout_start
Once you have those, you can compute stage-specific deltas like endpointing delay, model think time, audio synthesis delay, and render delay.
Instrument the system at the edges, not in the middle
A common mistake is to measure only inside the agent process. That misses network jitter, browser buffering, and client render delay. For turn-taking, you want timestamps on both sides of each boundary:
Sender side: when the system emits the event.
Receiver side: when the next stage actually observes it.
In practice, that means stamping events with monotonic time in each runtime and keeping the path short. Use a consistent clock source within a process, but do not assume clocks are synchronized across machines unless you are explicitly using NTP/PTP and accounting for drift. If you need cross-service correlation, include a request or session ID and compare relative intervals within each hop.
For example, when debugging an agent, log:
That gives you a trace you can slice by session, region, device type, or voice. Once you have enough samples, look at p50, p90, and p95, not just the mean. Mean latency hides the cases users complain about.
Understand the biggest sources of delay
In a voice avatar system, latency usually comes from one of four places.
1. Endpointing is too conservative. If your speech detector waits too long to declare the user done, everything downstream starts late. This is often the largest avoidable delay. Aggressive endpointing improves responsiveness but increases the risk of barge-in and truncated utterances. There is no universal threshold; you tune it based on interruption tolerance and expected user behavior.
2. Agent inference blocks on the wrong boundary. If you wait for a full final transcript when a partial transcript is already sufficient, you add avoidable time. Conversely, if you start too early, you may respond to incomplete intent. Streaming agents often need a policy for when to begin generation and when to revise output.
3. Audio/video generation is serialized unnecessarily. If the avatar waits for the complete response before starting to animate, you pay the full model time before any visible feedback occurs. Better systems begin rendering as soon as they have enough signal to animate plausible speech motion. The same principle applies to streaming TTS: first audio chunk matters more than full completion for perceived latency.
4. The client buffers too much. Browsers, media pipelines, and players often trade latency for smoothness. That can be fine for playback quality, but too much buffering makes turn-taking feel sluggish. A good system keeps buffers intentionally small and tolerates occasional jitter, especially in conversational use cases.
Measure perceived responsiveness, not just raw milliseconds
Raw elapsed time is useful, but human perception is the real metric. Two systems with the same measured latency can feel different if one starts with a visual cue, a subtle “listening” state, or a partial mouth movement sooner.
For example, if the avatar begins a slight preparatory motion before the first phoneme arrives, users perceive the turn as starting earlier. Likewise, a short audio cue can help signal that the system has taken the floor. These are product choices, but they affect how you interpret latency measurements. You should measure both:
First feedback latency: when the user sees or hears the system react.
First intelligible response latency: when the user can understand the actual content.
Those two can differ by hundreds of milliseconds. In many voice agent products, reducing first feedback latency matters more than shaving the last bit off full response completion.
Use a small benchmark harness
If you want numbers you can trust, build a repeatable harness that replays recorded utterances or scripted conversations and emits structured timing markers. Keep the test inputs stable so you can compare changes across model versions, endpointing settings, and transport configurations.
A minimal pattern looks like this:
This is intentionally simplistic. In production, you will also want per-session IDs, percentiles, and tags for region, voice, model, and client type. But a simple harness is enough to catch regressions before they ship.
Where Protoface fits
Protoface is useful here because it gives you a clear integration point for the avatar side of the pipeline. If your agent already exists and you want to measure the additional latency introduced by synchronized video face generation, the LiveKit plugin is a practical place to instrument. The plugin drops an avatar into a LiveKit voice agent, so you can compare the agent’s audio timing with the avatar’s first rendered frame without rebuilding your whole stack.
For deeper debugging, keep the timing logs in your agent process and correlate them with session-level behavior in the dashboard. The docs at docs.protoface.com are the right place for the exact integration details and event surfaces; the quickstart repos are useful when you want a working baseline and need to verify whether a latency change came from your app or from the integration layer.
If you prefer direct API-driven workflows, the REST API and Python SDK are better for creating sessions, managing avatars, and correlating request IDs with your own telemetry. That is especially useful when you are benchmarking session setup time or comparing quality tiers under load.
Common gotchas
A few mistakes show up repeatedly:
Measuring only local loopback tests. Real network conditions change buffering and endpointing behavior.
Using a single timestamp for “response time.” That hides which stage regressed.
Ignoring barge-in. A system that is fast but cannot recover when interrupted will still feel broken.
Comparing averages across different utterance lengths. Long inputs naturally take longer to recognize and respond to; segment by input duration.
Not separating setup latency from turn-taking latency. Session creation, avatar provisioning, and media negotiation matter, but they are different from ongoing conversational responsiveness.
Also remember that quality tier can affect latency. Higher-quality synthesis or rendering often costs more time. That is not inherently bad; it is a product decision. The right question is whether the added delay is acceptable for the user experience you are building.
Conclusion
Turn-taking latency is the most visible performance metric in voice and video avatar systems, but it only becomes actionable when you decompose it into stages and measure each boundary separately. Focus on user speech end, endpointing delay, agent think time, first audio, first frame, and client playout. Log those consistently, analyze percentiles, and benchmark with stable inputs.
If you are building on a realtime avatar stack, start with a single trace from user speech end to first visible response, then work backward from the slowest stage. For integration specifics, event surfaces, and quickstarts, see the documentation. If you want a concrete baseline, the LiveKit plugin and the Python SDK are the fastest way to get a measurable system up and running.