Why Your Support Avatar Feels Unnatural: Fixing STT, TTS, and Lip-Sync Timing Issues
Fix STT, TTS, and lip-sync timing in realtime support avatars with endpointing, streaming, and playback sync.
Introduction
If your support avatar feels “off,” the problem is usually not the face model. It’s timing. Human listeners are extremely sensitive to the relationship between speech audio, mouth motion, and conversational turn-taking. A system can have good STT, good TTS, and a decent lip-sync model and still feel unnatural if those components are not aligned at the frame and utterance level.
In practice, the common failure modes are predictable: the avatar starts speaking before the user finishes, pauses too long after text is generated, mouths words that do not match the audio onset, or keeps animating after the audio has already stopped. By the end of this post, you should be able to diagnose those issues, understand where the latency comes from, and structure a realtime avatar pipeline that feels much more human.
Where the uncanny feeling actually comes from
There are three timing domains in a voice avatar system:
Acoustic timing: when audio actually starts and stops.
Linguistic timing: when words or tokens are recognized or generated.
Visual timing: when mouth shapes, head motion, and eye motion are rendered.
Users do not perceive those independently. They perceive one conversation. If STT is delayed, the agent interrupts late. If TTS is delayed, the avatar “thinks” too long. If lip-sync is off by even a few hundred milliseconds, the speech looks synthetic even when the audio is fine.
There is also an asymmetry worth remembering: the visual system is forgiving of slightly lower fidelity, but very sensitive to timing mismatch. So the fix is usually not “better mouth shapes,” it’s “tighter synchronization across the pipeline.”
STT latency: the hidden cost of waiting for certainty
In realtime systems, STT is often run in partial, streaming mode. That gives you interim hypotheses before the user has finished speaking. The trade-off is confidence. If your agent waits for final transcripts only, you add end-of-utterance delay. If you react too early to unstable partials, you risk premature interruptions or wrong intent classification.
The practical fix is to treat partials as signals, not facts:
Use voice activity detection or endpointing to decide when the user is likely done.
Let partial STT drive speculative UI updates, but not irreversible actions.
Use final transcripts for response generation when correctness matters.
For conversational avatars, the biggest improvement usually comes from reducing endpointing delay. If your speech detector waits too long after the user pauses, the avatar will look polite but slow. If it cuts off too aggressively, it will feel interruptive. That balance is product-specific, but the engineering pattern is the same: detect turn boundaries explicitly instead of inferring them from transcript finalization alone.
TTS timing: text is not the same as speech
TTS introduces its own lag because the model or service needs input text, then synthesis time, then playback startup. If you wait for a complete response before starting speech, the user sees the avatar “thinking” in silence. That might be acceptable for long-form generation, but in support flows it feels sluggish.
The usual way to reduce perceived latency is to stream the response in chunks. This lets the TTS engine start playback on the first sentence while the language model continues producing later sentences. The key requirement is that your chunking respects prosody boundaries. Splitting in the middle of a clause or sentence often produces unnatural pacing, even if the audio starts earlier.
There are two important gotchas:
Do not over-chunk. Very small chunks create robotic prosody and can break natural pauses.
Do not buffer too long. Every extra buffer layer adds latency and makes the avatar feel detached from the conversation.
A good target is “start speaking as soon as you have a stable first clause.” That typically gives you the best balance between responsiveness and naturalness.
Lip-sync: render timing must follow audio onset, not text timing
Lip-sync should be driven by the audio playback clock, not by token timing or guessed word timing. That sounds obvious, but it is a common integration mistake. The mouth needs to open when the first phoneme actually plays, not when the text for that phoneme was emitted by the model.
Three issues show up repeatedly:
Audio starts late, mouth starts early: the avatar silently mouths the first words.
Audio starts early, mouth starts late: the face looks disconnected and “dead” at the start of speech.
Audio continues after mouth stops: the avatar appears to freeze mid-utterance.
To avoid that, synchronize lip motion to the same playback timestamps used by the audio renderer. If your stack exposes an audio frame clock or RTP timestamp, use it. If it exposes an explicit start event, align mouth animation start to that event. Do not infer timing from the moment you received the text or from when the TTS request was made.
Also remember that lip-sync quality is not just about phonemes. Head motion, blink timing, and micro-expressions all affect perceived naturalness. The avatar looks more human when the face is not mechanically frozen between speaking bursts. Small idle motion helps, but it should not drift independently from the speech state machine.
Designing a realtime pipeline that feels responsive
A good mental model is a state machine with three phases: listening, thinking, and speaking. The avatar should visibly transition between them with minimal delay, but not so fast that it looks jittery.
A robust implementation typically does this:
Stream user audio into STT.
Use endpointing to decide when the user has likely finished.
Begin response generation immediately after the turn boundary.
Stream the first response chunk into TTS as soon as it is stable enough to speak.
Start avatar animation on audio playback start, not on text generation.
Keep the speaking state active until playback fully drains.
The main engineering constraint is backpressure. If the language model, TTS, and video renderer all buffer independently, latency compounds. Keeping buffer sizes small and using explicit events between stages is usually better than “fire and forget” message passing.
How to debug the mismatch instead of guessing
If the avatar feels wrong, log timestamps for each stage and compare them on the same timeline. You want to answer four questions:
When did the user stop speaking?
When did STT finalize the turn?
When did TTS produce the first playable audio?
When did the avatar visually begin speaking?
Once those are captured, the bug is usually obvious. For example, if STT finalization is 700 ms after actual silence, you know endpointing is the culprit. If TTS starts quickly but the face lags behind, the problem is in the video or animation sync layer. If everything is late, you may have buffering in multiple places.
A useful rule of thumb: optimize the longest visible gap first. Human perception is nonlinear; shaving 100 ms off three different layers is often less noticeable than removing 300 ms from the single most visible pause.
Where Protoface fits
Protoface is useful when you want the avatar layer to behave like a managed realtime component instead of hand-rolling video timing yourself. For LiveKit-based voice agents, the LiveKit plugin and the PyPI package can drop a synced talking face into an existing agent so you can focus on endpointing, response streaming, and conversation policy rather than building a custom video pipeline from scratch.
For lower-level control, the REST API and Python SDK let you create avatars and sessions programmatically. The exact request and response fields are documented in the docs, but the shape is straightforward: authenticate with an API key, create a session, and attach it to your voice workflow. That keeps the avatar timing coupled to the same events your agent already uses.
Conclusion
An avatar feels unnatural when the conversation pipeline is temporally sloppy. The core fix is not cosmetic; it is systems work. Tighten STT endpointing, stream TTS in sensible chunks, and drive lip-sync from actual audio playback timing. Then instrument the pipeline so you can see where the lag is coming from instead of tuning by feel.
If you are building a voice agent or support experience, start by measuring the gaps between user silence, transcript finalization, audio onset, and visual onset. Once you can see those numbers, the next iteration becomes much easier. If you want a managed avatar layer that slots into an existing realtime stack, the docs at docs.protoface.com are the right place to start.