What to Measure in ElevenLabs Realtime Avatar Apps: Latency, Turn-Taking, Lip-Sync, and Failures
Measure latency, turn-taking, lip-sync, and failure rates in ElevenLabs realtime avatar apps to debug user-perceived performance.
Introduction
If you are adding a realtime avatar to a voice agent, the hard part is not “making video.” The hard part is making the whole interaction feel coherent under network jitter, synthesis delays, model latency, browser scheduling, and the inevitable failure modes of streaming systems. A talking face that trails the audio by 300 ms can feel broken even when the underlying ASR and TTS are fine. A system that cuts off users too aggressively can feel robotic. A system that recovers from transient failure without dropping the session can feel reliable.
This post focuses on what to measure in realtime avatar apps: end-to-end latency, turn-taking behavior, lip-sync quality, and failures. By the end, you should be able to define practical metrics, instrument your app, and reason about which part of the pipeline is actually causing a bad user experience.
Start with the full path, not one component
For avatar apps, “latency” is not a single number. A complete path usually includes:
audio capture or inbound voice stream
transcription or speech understanding
LLM / agent reasoning
text-to-speech generation
avatar animation and video delivery
browser playback and rendering
If you only measure model latency, you miss the actual user-facing delay. If you only measure media delivery, you miss whether your agent is slow to decide when to speak. Measure timestamps at boundaries between stages so you can separate “compute is slow” from “streaming is laggy.”
Latency: measure what the user experiences
The first metric worth defining is time to first audible response from the end of the user’s utterance. For voice agents, this is usually the most important number. It answers: after the user stops talking, how long until the system starts responding?
In practice, break it down into three useful components:
Input finalization delay: how long until your VAD or endpointing logic decides the user is done speaking.
Response generation delay: how long until the agent produces the first token or first TTS chunk.
Playback start delay: how long until the browser actually renders audio/video.
These are different failure domains. If response generation is fast but playback is late, your media pipeline is the issue. If playback starts quickly but the agent hesitates before speaking, your turn detection or model orchestration is too conservative.
Also measure steady-state latency while the agent is speaking. For avatars, latency drift matters. If the video stream grows stale because audio frames are arriving but animation frames are delayed, you can get subtle desynchronization that users perceive as “off,” even when no single metric is alarming.
That snippet is intentionally crude. The point is to timestamp at the boundaries you control. If you need a fuller picture, emit spans or structured events for each phase and correlate them by session id.
Turn-taking: measure interruption quality, not just speed
In conversational systems, bad turn-taking is often worse than raw latency. If the agent talks over the user, or waits too long and forces awkward pauses, the interaction feels brittle. The goal is not “respond as fast as possible”; it is “respond at the right time.”
There are three metrics that are worth tracking:
False cut-in rate: the agent starts speaking while the user is still speaking.
False wait rate: the agent remains silent after the user has clearly finished.
Interruption recovery time: how long it takes to stop or adapt when the user interrupts mid-response.
These numbers depend on your endpointing and barge-in policy. In practice, VAD thresholds, minimum silence windows, and partial transcript confidence all affect turn-taking. A low-latency system with bad endpointing can feel worse than a slightly slower system that consistently waits for the right boundary.
For debugging, annotate every turn with:
user speech start/end timestamps
agent speech start/end timestamps
whether the agent was interrupted
whether the user resumed speaking before the agent finished
Then inspect a small sample of “bad” turns. You will usually find one of four causes: aggressive endpointing, late transcript arrival, delayed model output, or playback buffering that made the agent appear to respond late.
Lip-sync: measure audio-video alignment, not just motion
Lip-sync is often discussed qualitatively, but you can make it more concrete. The relevant question is not “does the face move?” but “does mouth motion align with the audio that the user hears?” In realtime streaming, perfect frame-level alignment is unrealistic. What matters is staying inside a perceptual tolerance window.
A useful practical approach is to measure:
audio-to-video offset: how far the visible mouth motion lags or leads the audio playback clock
jitter: variation in that offset over time
dropout rate: missing frames, frozen video, or stalled animation while audio continues
If you already collect player telemetry in the browser, compare the timestamp of the audio output buffer with the frame timestamps for the avatar stream. A consistent offset may be acceptable. Large variance is not.
Also separate lip-sync quality from face quality. A high-fidelity avatar can still look wrong if the timing is off. Conversely, a simpler avatar with stable, low-jitter timing often feels better in production.
Two common gotchas:
Browser scheduling: the video frame and audio callback do not necessarily advance on the same clock.
Backpressure: if your client buffers too much video, the avatar may look “smooth” while actually being late relative to speech.
Failures: treat them as first-class metrics
Realtime systems fail in ways that are easy to ignore during development and expensive in production. You should explicitly measure failure rates and recovery behavior, not just success-path latency.
At minimum, track:
session setup failures — auth errors, invalid config, timeout creating the session
stream establishment failures — WebRTC negotiation problems, iframe load failures, media permission issues
mid-session disconnects — network drops, browser tab suspension, server-side errors
recovery success rate — whether the session reconnects cleanly or requires a full restart
For each failure, capture the state needed to reproduce it: session id, client browser, network type, avatar config, and which step failed. If you cannot tell whether a failure happened before media negotiation, during playback, or inside the agent, you do not have an actionable metric.
One good operational metric is successful conversational minutes per active session. This rolls up a lot of messy edge cases into a number that is easy to trend over time. If that number drops while average latency looks fine, you probably have a reliability problem rather than a performance problem.
How to instrument a production app
Use a small set of consistent event types and attach them to every session:
session_createduser_speech_starteduser_speech_endedagent_first_tokenoragent_tts_startedagent_speech_startedagent_speech_endedstream_errorsession_ended
From those, you can derive the metrics above without instrumenting every internal component forever. If you do need deeper visibility, add stage-specific spans for transcription, reasoning, synthesis, and media transport. Keep the session id and a monotonic clock in every event.
Where Protoface fits
For teams using Protoface as the avatar layer, the useful thing is that you can measure these same boundaries whether you are integrating through the REST API, the Python SDK, or the LiveKit Agents plugin. In a LiveKit-based voice agent, for example, the avatar is just another stage in the session, so you can timestamp when the agent starts speaking and when the avatar stream becomes visible in the client. That makes it easier to separate agent latency from video delivery latency.
A minimal REST call to create a realtime session will look like any other authenticated API request; exact fields depend on the session model in the docs.
If you are using the Python SDK or the LiveKit plugin, keep the same measurement model: log the moment the user stops talking, the moment the agent begins response generation, and the moment the avatar actually starts rendering speech. The integration details differ, but the metrics do not. If you need implementation examples, the public docs and quickstarts are the right place to start: docs.protoface.com and the relevant quickstart.
Conclusion
For realtime avatar apps, the useful metrics are the ones that explain user perception: end-to-end latency, turn-taking quality, lip-sync alignment, and failure behavior. Measure the whole path, not just model time. Log the session boundaries you control, then drill into the stages that are actually causing delays or glitches.
If you are building with Protoface, start by instrumenting a single session end-to-end and compare what your logs say against what the user actually sees and hears. Once that baseline is clear, the path to improvement is usually obvious. The docs at docs.protoface.com are the right reference for exact API fields, SDK usage, and integration details.