How to Prevent Talk-Over and Cutoff Issues in Streaming Voice Avatars
Prevent talk-over and cutoffs in streaming voice avatars with explicit turn detection, debounce, and end-to-end interruption handling.
Introduction
Talk-over and cutoff problems are the same failure mode seen from opposite ends: the avatar starts speaking before the user is done, or it gets interrupted mid-utterance and never cleanly yields the floor. In a realtime voice avatar pipeline, that usually means your system is not treating turn-taking as a first-class protocol. It is mixing audio transport, ASR finalization, LLM latency, TTS streaming, and video lip-sync without a clear policy for when to start, pause, barge in, or stop.
If you are building a voice agent with Protoface, a WebRTC app, or any other streaming avatar stack, the practical goal is simple: make the agent feel responsive without speaking over the user. By the end of this post, you should be able to reason about where talk-over comes from, add guardrails in your agent loop, and choose the right interruption strategy for your application.
Understand the turn-taking pipeline
Most cutoff bugs come from one of four clocks drifting out of sync:
Audio capture: the client is still sending user speech, but the server thinks the turn ended.
ASR: partial transcripts are noisy; final transcripts arrive late; endpointing can be too eager.
Agent generation: the LLM starts planning a response before the user has really stopped.
TTS / avatar playback: the first audio chunk or lip-sync frame is emitted before the system has committed to the turn.
The core design principle is to separate intent to respond from commitment to speak. You can let the model think, but do not let it emit audible or visible speech until the user turn is considered closed by explicit policy.
Use explicit turn detection, not vibes
In practice, “the user is done” should be computed from a combination of signals, not a single silence threshold. The common pattern is:
VAD or endpointing to detect speech activity.
ASR partials to estimate whether the user is mid-clause.
Conversation state to know whether the agent is currently speaking.
Barge-in policy to decide whether user speech should interrupt the agent.
For example, a conservative implementation might wait for both short silence and an ASR final before starting the agent response. A more aggressive one might start generation on a strong partial transcript, but still hold audio output until the turn is stable.
A useful mental model is:
If you allow any downstream stage to ignore upstream state, you will eventually get talk-over. The fix is usually not “make the model slower” but “make the handoff explicit.”
Prevent premature speech with a two-phase response
The simplest reliable pattern is a two-phase response:
Think phase: the agent receives partial or final user text and produces a draft answer internally.
Speak phase: the system only starts TTS and avatar playback after a turn commit event.
This gives you room to implement debounce logic, interruption handling, and cancellation. It also prevents the classic failure mode where the agent emits the first 200 ms of speech, then discovers the user was still talking and has to stop abruptly.
A practical version of this is to keep a short “grace window” after the last user voice activity. The right value depends on your ASR and audience, but if you are seeing cutoffs, your endpointing is probably too aggressive. If you are seeing talk-over, your grace window is probably too short or your barge-in logic is too eager.
Do not confuse low latency with early commitment. A fast agent that waits 150 ms longer to speak will usually feel better than a fast agent that constantly interrupts.
Handle interruption as cancellation, not just pause
Once the avatar starts speaking, user speech should be treated as a cancellation signal when barge-in is enabled. That means you need to stop three things together:
the current TTS stream,
the avatar’s audio/video playback state,
any queued agent output that belongs to the interrupted turn.
If you only mute audio but keep generating tokens, the agent often resumes with stale context and produces a response that no longer matches the conversation. If you only stop video but let audio continue, lip-sync drifts and the avatar looks broken. Cancellation has to be end-to-end.
From a state machine perspective, think in terms of discrete states:
When speech overlaps with the user, transition to interrupted, stop the current output pipeline, and then re-enter listening with the latest user turn. That is usually better than trying to “resume” mid-sentence.
Debounce both start and stop events
Start and stop edge cases look different but are usually caused by the same thing: noisy endpointing. The cure is debouncing on both edges.
For start: require a stable user-speech signal before declaring the user is talking. This avoids false starts from brief noise or backchannels.
For stop: require a stable silence window before declaring the user has yielded the floor. This avoids the agent jumping in during natural pauses between clauses.
In code, that often means you keep timestamps for last speech activity and last final transcript, then gate output on both:
The actual thresholds depend on your ASR, network conditions, and user expectations. The main point is that a single event should not directly trigger speech without some stabilization logic.
Keep network and rendering jitter out of your turn logic
WebRTC and streaming TTS introduce timing noise that can masquerade as conversation problems. Audio packets arrive in bursts. Video frames can lag behind audio. Browser rendering can stutter. None of that should change who has the floor.
Two implementation rules help a lot:
Base turn decisions on server-side timestamps and conversation state, not on whether the browser has rendered the latest frame.
Use one authoritative controller for speaking state so the agent, TTS, and avatar do not independently decide when to start.
If you split this logic across multiple services, be very deliberate about who owns cancellation. A common bug is that the LLM service thinks it can still speak, while the avatar service already believes the turn was interrupted. That leads to exactly the kind of half-stuck behavior users notice immediately.
How Protoface fits into this
Protoface is useful here because it gives you a synchronized avatar surface that plugs into an existing voice stack instead of forcing you to build lip-sync, session management, and delivery plumbing yourself. In a LiveKit-based agent, the plugin path is the most direct way to attach a talking face to a voice agent while keeping the avatar aligned with the agent’s audio state. The integration point is the same one you want to control for talk-over: when to emit speech, when to stop it, and when to cancel it.
A minimal LiveKit-style setup looks like this conceptually:
If you are managing sessions directly, the REST API and Python SDK let you create and inspect avatar/session state from your backend, which is useful when you want server-side control over interruptions, timeouts, and per-session policies. The exact request shape depends on the endpoint, but the shape is straightforward: authenticate with an API key, create a session, then drive it from your agent orchestration layer.
For exact fields and examples, use the documentation and the quickstarts in the linked repos, especially if you are wiring this into an existing agent runtime rather than starting from scratch.
Practical checklist for eliminating talk-over
When you debug a bad conversation turn, walk this list in order:
Is user speech detection stable enough, or are you triggering on noise?
Are you waiting for a final transcript or a conservative silence window before speaking?
Does one component own the “speaker” state, or are multiple services racing?
When the user barges in, do you cancel TTS, avatar playback, and queued generation together?
Are you measuring latency at each stage so you can see where the delay or overlap is introduced?
If you can answer those five questions clearly, you will usually find the bug quickly. If you cannot, your system likely has implicit state transitions hidden inside a client, plugin, or model callback.
Conclusion
Talk-over and cutoff issues are not really “voice quality” bugs. They are turn-taking bugs. The fix is to make speech a controlled state transition: detect user speech conservatively, commit to speaking only after the turn is stable, and treat interruption as a full cancellation path.
If you are integrating a realtime avatar into a voice agent, start by tightening the state machine around the audio pipeline, then attach the avatar to that same control loop. The docs at docs.protoface.com have the concrete API and integration details, and the relevant quickstart repos are a good way to verify the behavior in a real agent before you wire it into production.