Introduction

Natural avatar conversations are mostly a timing problem. The model has to speak at the right moment, the avatar has to start lip movement when audio begins, the microphone pipeline has to stop talking over itself, and the visual layer needs to react quickly enough that the interaction feels continuous rather than stitched together.

If you are building a voice agent with a talking face, you are usually coordinating three streams at once:

• TTS: generated speech audio for the agent.

• STT: incoming user audio turned into text or partial hypotheses.

• VAD: voice activity detection that decides when someone is actually speaking.

By the end of this post, you should have a solid mental model for how those streams interact, how to avoid the common failure modes, and how to wire them together so the avatar feels responsive instead of laggy or interruptive.

The core problem: sync is not just lip sync

People often treat avatar sync as “start the mouth when audio starts.” That is necessary, but not sufficient. In a real conversation loop, the avatar is part of a turn-taking system:

1. The user speaks.

2. VAD detects speech onset and end.

3. STT produces partial and final transcripts.

4. The dialog engine decides when to respond.

5. TTS generates audio for the response.

6. The avatar renders that audio with a matching visual timeline.

If any of those steps drifts, you get one of a few predictable failures:

• Early mouth movement: the avatar appears to speak before audio is audible.

• Late mouth movement: speech starts, but the face stays still for a beat.

• Overtalking: the agent continues speaking while the user has already started.

• Turn clipping: VAD cuts off user speech too aggressively, harming STT accuracy.

• Dead air: the system waits too long for clean turn boundaries before responding.

The right design is usually not “perfectly sync everything.” It is “minimize latency, make turn boundaries robust, and let the avatar follow the audio clock.”

Understand the three clocks

There are three different timing domains you need to keep in mind.

1. The audio clock

This is the timing of the generated speech itself. For the avatar to look believable, the visual phoneme/mouth motion needs to line up with the actual playback schedule of the TTS stream, not with when your server happened to receive the first bytes.

Practical implication: if your TTS service streams audio chunks, hand those chunks to the avatar immediately and let the playback engine own the alignment. Avoid buffering “just to be safe,” because buffering creates visible lag.

2. The speech clock

This is the conversational state: who has the floor, whether the user is mid-utterance, and whether the agent should yield. VAD drives this state better than raw STT because STT can lag and can be wrong on noisy audio. VAD is what lets you decide “the user has started speaking, pause or cancel the agent now.”

For turn-taking, VAD should usually be treated as the fast signal and STT as the slower, semantic signal.

3. The semantic clock

This is the transcript and dialog state. Partial transcripts are useful for early intent detection and speculative responses, but they are not a great trigger for interrupting speech. Final transcripts are more reliable for deciding the content of the reply.

In practice, the pattern is:

• VAD starts the user turn.

• STT partials improve responsiveness.

• STT final closes the loop and stabilizes the agent’s answer.

How to wire STT and VAD without fighting each other

A robust pipeline usually follows this rule: VAD controls turn state; STT fills in meaning.

That means you should avoid a design where the agent waits for a final transcript before acknowledging that the user is speaking. If the microphone is hot and the user begins talking, VAD should immediately mark the agent as in-listen mode. Then STT can stream partial results while the user continues.

This separation matters because STT and VAD fail differently:

• VAD false positives can be tolerated if you use hangover and minimum speech thresholds.

• STT lag is expected and should not block floor control.

• STT errors can be corrected by the final transcript.

A practical turn-taking strategy is:

1. Use VAD to detect speech start quickly.

2. Require a short speech minimum before treating it as a real turn.

3. Use a hangover window at end-of-speech so short pauses do not cause premature cutoffs.

4. Feed STT continuously, but only commit final dialog actions when the utterance is complete.

Those thresholds are domain-specific. In customer support, you usually want a slightly longer hangover because interruptions are costly. In gaming or consumer assistants, shorter end-of-speech detection feels more responsive.

Synchronizing TTS with the avatar renderer

Once the agent decides to speak, the key is to make the avatar follow the audio, not the other way around.

Good systems start the visual speech state when the first audio frame is scheduled for playback, then continue animating from the audio stream. If the avatar system receives precomputed phoneme timings, those timings should be anchored to the same output timeline as the audio output device or WebRTC track.

When things go wrong, it is often because of one of these issues:

• Prebuffering too much audio: the avatar waits for a large chunk before starting, which increases first-token latency.

• Chunk boundaries too coarse: mouth motion advances in visible steps instead of continuously.

• Independent clocks: the TTS playback and the animation engine drift apart.

• Late cancelation: the user starts speaking, but the agent keeps a queued response and “talks over” them.

For conversational agents, low first-audio latency matters as much as perfect pronunciation. A small amount of phoneme jitter is usually less noticeable than a 300–500 ms visual delay.

Implementation pattern: cancel, then regenerate

The most natural experience usually comes from allowing user interruption. If VAD says the user has started a new turn, stop the agent’s current TTS immediately, preserve whatever partial context you have, and let the dialog manager re-plan.

This is more important than it sounds. Without interruption handling, you get the classic “I’m sorry, I didn’t catch that” overlap where the assistant and the user speak simultaneously, and the avatar keeps moving even after the conversation has already changed direction.

A common event flow looks like this:

# Pseudocode: event-driven conversation loop

stream_tts(reply)

# Pseudocode: event-driven conversation loop

stream_tts(reply)

# Pseudocode: event-driven conversation loop

stream_tts(reply)

The important detail is that the TTS stream is a disposable output, not a sacred artifact. If the user reclaims the floor, cancel it.

Example: LiveKit voice agent with a Protoface avatar

If you are already using LiveKit for your voice agent, the simplest path is to drop in the Protoface plugin so the agent gets a synchronized talking face without you having to build a separate video pipeline. The plugin is published on PyPI as livekit-plugins-protoface and the examples in the quickstart repo are a good reference point: GitHub quickstarts.

At a high level, the agent still does the same things described above: VAD gates turn-taking, STT provides transcript events, and TTS drives the outgoing audio. The plugin handles the avatar side of the output so the face tracks the speech stream in a way that stays aligned with the audio timeline.

from livekit.agents import JobContext

await agent.run()

from livekit.agents import JobContext

await agent.run()

from livekit.agents import JobContext

await agent.run()

That snippet is intentionally schematic: the exact constructor and avatar fields depend on the integration version. The important part is the architecture. Keep the agent’s speech pipeline event-driven, and let the avatar subscribe to the same audio/turn events rather than inventing a separate state machine.

When to use the REST API or SDK instead

If you are provisioning avatars or sessions from your backend, use the REST API or Python SDK rather than hard-coding resources in the client. That keeps API keys off the browser and gives you a clean control plane for creating sessions, tracking usage, and managing avatars.

import os

print(resp.json())

import os

print(resp.json())

import os

print(resp.json())

If you prefer Python, the SDK keeps the same idea but removes some HTTP plumbing. For production systems, this is often the better place to express your session lifecycle logic.

Practical tuning advice

There is no universal “best” VAD or STT setup. The right settings depend on your audio source, user environment, and how interruptible the agent should be. Still, a few rules hold up well:

• Prefer aggressive start detection over aggressive stop detection. You want to notice when the user begins speaking, but you do not want to clip the end of their utterance.

• Use partial transcripts for responsiveness, not control. They are helpful, but they are not stable enough to drive hard state transitions.

• Measure first-audio latency and barge-in latency. These tell you more about perceived quality than raw STT accuracy.

• Keep the audio pipeline streaming. Batch processing tends to look and feel slower, even if the average latency is acceptable.

• Test with real microphones and noisy rooms. Avatar sync that looks fine in a quiet dev environment often falls apart once users talk over background audio.

Also, do not over-optimize the avatar animation in isolation. If the agent pauses too long before answering, the face can be perfectly synced and still feel unnatural. Conversation timing is a system property.

Conclusion

Syncing TTS, STT, and VAD for a natural avatar conversation comes down to owning the conversation loop end-to-end: VAD decides who has the floor, STT supplies meaning, and TTS drives an audio clock that the avatar follows closely. When you treat those as separate signals with different latencies and responsibilities, the system becomes much easier to reason about.

If you are implementing this in a LiveKit-based voice agent, start with the avatar plugin path. If you are provisioning sessions or managing avatars server-side, use the REST API or Python SDK. For setup details, field names, and current integration examples, check the docs at docs.protoface.com.

For more implementation examples and quickstarts, the GitHub organization is a good place to look: GitHub.