How to Minimize End-of-Utterance Delay in Realtime Conversational Avatars
Learn how to reduce end-of-utterance delay in realtime conversational avatars with VAD tuning, partial transcripts, streaming TTS, and barge-in handli
Introduction
When a conversational avatar feels “laggy,” the problem is often not the video pipeline itself. It’s the gap between the user finishing a sentence and the system deciding that it is safe to speak, render a facial response, and start the next turn. That gap is the end-of-utterance delay, and in realtime voice agents it matters as much as TTS latency or frame rate.
In this post, we’ll look at where that delay comes from, how to reduce it without making your agent talk over users, and what trade-offs you’re making when you tune endpointing, buffering, and turn-taking. The focus is practical: what to measure, what to change, and how to keep avatars feeling responsive rather than robotic.
What end-of-utterance delay actually is
In a realtime conversation stack, the user’s microphone audio is typically streamed to an ASR model or a speech-aware agent loop. At some point the system decides “the user is done speaking.” Only then does the agent finalize the current input, run reasoning, generate a response, synthesize audio, and often start driving the avatar’s lip sync and facial motion.
End-of-utterance delay is the time between the actual end of the user’s speech and your system’s turn boundary. It is not a single number, because several components contribute:
Client-side buffering: audio chunks are accumulated before being sent or processed.
Transport jitter: WebRTC or other realtime transports smooth packet delivery, which is good for quality but adds a little latency.
Endpointing / VAD: voice activity detection needs a confidence threshold and usually a hangover window.
ASR finalization: partial transcripts arrive quickly; final transcripts often lag slightly behind.
Agent policy: your orchestration may wait for a final transcript even when enough evidence exists to begin planning a response.
The practical question is not “how do I make it zero?” You can’t. The question is “how do I keep it low enough that turn-taking feels natural, while avoiding interruptions, clipped user utterances, or premature responses?”
Measure the right boundary
If you do not instrument the turn boundary, you will tune the wrong layer. The most useful timestamps are:
Last user audio frame received
VAD endpoint fired
Final transcript available
Agent response generation started
First response audio frame sent
Avatar playback begins
From these, two deltas matter most:
Endpoint delay = last speech frame to endpoint fired
Response start delay = endpoint fired to first outgoing audio
If endpoint delay is high, tune speech detection. If response start delay is high, the bottleneck is usually orchestration, model latency, or TTS startup. Do not blame the avatar layer until you’ve separated those.
A useful debugging pattern is to log turn events alongside the audio stream. Even a simple event timeline is enough to reveal whether your agent is “waiting to be sure” or “thinking too long after it already knows.”
Reduce endpointing latency without cutting users off
Most systems default to conservative endpointing because false positives are more annoying than a slightly delayed reply. That’s reasonable, but in avatar experiences the extra half-second is very visible. The goal is to keep the detector aggressive enough to feel immediate while preserving enough guardrail to avoid interruptions.
Tune VAD and hangover together
Voice activity detection usually depends on a rolling window and a silence threshold. Shorter windows reduce delay but increase sensitivity to brief pauses, breathing, and plosives. Longer windows are safer but slower.
In practice, endpointing is a pair of parameters:
Activation threshold: how confident the detector must be that speech is present.
Silence hangover: how much silence must follow speech before ending the turn.
If your users tend to speak in short bursts, a long hangover is usually the biggest source of delay. If your domain includes hesitations and self-corrections, make the hangover slightly longer and accept the extra latency rather than interrupting the user mid-thought.
For many conversational agents, the right answer is not a single fixed hangover. A better strategy is to adapt endpointing based on context:
Shorter hangover when the user has been speaking continuously and the next action is obvious.
Longer hangover when the agent expects enumerations, corrections, or dictation.
More aggressive endpointing for push-to-talk or clearly bounded tasks.
Use partial transcripts and semantic cues
Waiting for a final ASR transcript is often unnecessary. The agent can frequently infer intent from partial text and acoustic silence. For example, if the last few words indicate a complete clause and the user has paused for 300–500 ms, you may already have enough signal to begin planning.
This is where “semantic endpointing” helps. The system combines acoustic silence with transcript completion cues:
Did the user finish a syntactic unit?
Are there trailing fillers like “um” or “uh”?
Is the utterance likely to continue, based on punctuation or partial text?
Does the dialog state suggest the user is answering a question rather than narrating?
The trade-off is obvious: semantic endpointing reduces perceived latency, but it can be wrong in edge cases. The safest pattern is to use it as a hint, not as a hard override. If the agent starts preparing a response on a strong semantic signal, it should still be ready to abort or revise if the user resumes speaking.
Keep the response pipeline “warm”
Once endpointing fires, the fastest path is the one that does the least work after turn end. That means minimizing cold starts and avoiding unnecessary serialization between steps.
Three common optimizations:
Pre-initialize your model clients so you are not creating HTTP sessions or loading weights on every turn.
Start response planning on partial input when your UX can tolerate it; then finalize after endpointing.
Stream TTS and avatar playback immediately instead of waiting for the full sentence to be synthesized.
Streaming matters because the user perceives the start of motion and sound, not your backend completion time. If your TTS starts outputting audio after 150 ms instead of 600 ms, the avatar feels dramatically more alive, even if the total sentence duration is unchanged.
For lipsynced avatars, the first audio frame is also the trigger for facial animation. A delay here is more visible than in a pure voice agent, because the face is a high-salience cue. If the audio starts late, the avatar appears to “think” before it speaks, which reads as lag even when the system is technically correct.
Design for barge-in and overlap
Lowering end-of-utterance delay is only half the problem. If your system is too eager to respond, users will interrupt it. The right answer is to support barge-in cleanly.
That means:
Stop TTS immediately when the user starts speaking again.
Stop lip-sync animation or transition it into an idle/listening state.
Discard partial agent output for the interrupted turn.
Preserve dialog state so the resumed user turn is interpreted correctly.
In other words, the system should optimize for fast recovery, not only fast start. A slightly aggressive endpoint detector is much safer when interruption handling is solid.
Protoface in a LiveKit agent loop
For developers already using LiveKit Agents, a common pattern is to attach a synchronized avatar so the agent has a speaking face without changing the rest of the voice pipeline. The Protoface LiveKit integration follows the same principle: keep your agent loop and turn logic intact, then stream the avatar from the speech/audio events you already have.
That matters for end-of-utterance delay because the avatar should not introduce an additional turn boundary. If your voice agent already knows when speech ended, the avatar layer should react to that event, not re-detect it independently.
The implementation detail that matters is lifecycle alignment: when your agent finalizes a user turn, the avatar should transition immediately into speaking, and when the user barges in, it should stop just as immediately. If you make those transitions event-driven, you avoid duplicating endpoint logic in the avatar layer.
If you are building directly against the API, you can also create or manage sessions from your backend and keep your turn logic centralized:
The exact request schema and fields are in the docs, but the architectural point is the same: do the timing-sensitive work in one place, and let the avatar consume the result. That keeps your endpointing policy consistent across voice and video.
Common mistakes that add delay
Double endpointing: the ASR layer decides the turn ended, then the agent re-checks silence before responding.
Overly long silence thresholds: safe for dictation, bad for interactive dialog.
Waiting for final transcripts unnecessarily: partial text already provides enough signal in many cases.
Non-streaming TTS: holding audio until the full sentence is synthesized makes the avatar feel slow.
Recreating sessions per turn: session setup belongs outside the hot path.
If you see a consistent 500–900 ms “dead air” after the user stops talking, the issue is usually one of these, not the avatar rendering itself.
Practical tuning order
If I were debugging this in production, I’d tune in this order:
Measure endpoint delay and response start delay separately.
Reduce silence hangover until false interruptions become noticeable.
Enable partial-transcript-driven planning where safe.
Stream TTS and ensure the avatar starts on first audio, not on full completion.
Add robust barge-in so you can afford a slightly more aggressive endpoint.
This order works because it attacks the latency on the critical path first. Micro-optimizing avatar rendering while the system still waits an extra second to detect turn end is wasted effort.
Conclusion
End-of-utterance delay is mostly a turn-taking problem, not a graphics problem. The best results come from measuring the boundary precisely, tuning endpointing conservatively but not timidly, streaming response audio as early as possible, and making barge-in a first-class behavior.
If you’re building a realtime avatar into a voice agent, start by instrumenting the turn lifecycle and then reduce the silent gaps one layer at a time. The docs at docs.protoface.com cover the integration details, and the quickstarts linked from the project README are useful for seeing the realtime flow end to end.