Step-by-Step Guide to Streaming TTS Audio into a LiveKit Avatar Agent
Step-by-step guide to streaming TTS into a LiveKit avatar agent, covering audio sync, turn handling, cancellation, and latency pitfalls.
Introduction
Streaming text-to-speech into a live avatar is mostly a coordination problem: you need audio to start quickly, continue smoothly as tokens arrive, and stay aligned with the avatar’s mouth movements while the rest of your agent pipeline is still doing work. If you get it wrong, the user sees lip-sync lag, dead air, clipped utterances, or a face that keeps talking after the underlying turn has already ended.
This guide shows the practical shape of the problem and how to wire a streaming TTS source into a LiveKit-based voice agent so the avatar can speak incrementally instead of waiting for a full sentence. By the end, you should understand the audio flow, where timing breaks down, how to handle turn boundaries, and what the integration looks like in code.
What “streaming TTS into an avatar” actually means
In a typical voice agent, text is produced in chunks: a model emits partial or final text, a TTS engine turns that text into audio, and the audio is published into a real-time media session. For an avatar, there is a second requirement: the avatar renderer needs a synchronized audio stream so it can generate consistent lip motion and facial animation.
That leads to a simple mental model:
Your LLM or dialogue layer produces text incrementally.
Your TTS service converts that text into audio frames or segments as soon as they are available.
The audio is sent into the LiveKit session as the agent’s outbound speech track.
The avatar subscribes to that same speech stream and renders synchronized video.
The key detail is that the avatar should follow the same audio clock as the user hears. Do not try to “drive” lip sync from text alone if the actual TTS output may be delayed, reordered, or chunked differently than the text stream.
Step 1: Keep the agent pipeline streaming end-to-end
If your agent only emits final text, you lose most of the benefit of realtime TTS. The common pattern is to stream partial model output, buffer it into phoneme-friendly or sentence-ish chunks, and start TTS on the first stable fragment. You want to minimize time-to-first-audio without fragmenting the speech so aggressively that the voice sounds unnatural.
In practice, a good chunking strategy is:
Start TTS when you have a short, complete phrase or clause.
Avoid cutting across punctuation or obvious prosodic boundaries when possible.
Keep chunk sizes large enough that the TTS engine can produce natural intonation.
Keep the whole pipeline cancellable so barge-in or interruption can stop playback immediately.
For agents that support interruption, cancellation matters more than it first appears. If the user starts speaking, the agent should stop generating new TTS, stop publishing audio, and tell the avatar to stop mouth animation as soon as the active speech track ends.
Step 2: Publish audio on the same real-time path the avatar uses
With LiveKit, the outbound speech should be treated like a normal media stream. The avatar is not a separate “render the text” subsystem; it is an attached visual representation of the speaking participant. That means the most reliable integration point is the agent’s audio output track, not a side channel of text updates.
Your implementation needs to preserve:
Ordering: audio frames must arrive in sequence.
Continuity: avoid gaps unless the speaker really pauses.
Clock discipline: use a single media timeline for the utterance.
Cancellation semantics: when the turn stops, the stream stops.
If you are mixing TTS with other audio sources, do the mixing before the LiveKit publish step. Once packets are in the session, downstream rendering assumes the stream is authoritative.
Step 3: Handle the two failure modes that break lip sync
Most “avatar looks wrong” bugs fall into one of two buckets: timing drift and turn mismatch.
Timing drift
Timing drift happens when the avatar’s animation clock gets out of sync with the audio clock. Causes include buffering too much audio before publish, emitting irregular frame sizes, or switching tracks mid-utterance. The fix is usually boring but effective: keep audio frame cadence stable, avoid unnecessary re-encoding, and don’t interrupt an utterance unless you intend to end it.
Turn mismatch
Turn mismatch happens when the text generator, TTS service, and session state disagree about whether the agent is still speaking. For example, the model may have already ended the sentence, but TTS is still draining a queue of buffered chunks. Or the model may start a new turn before the previous speech stream has fully stopped.
The fix is to make turn state explicit in your app:
Mark a turn as “speaking” when the first audio for that turn is published.
Mark it complete only when the final audio chunk finishes.
Cancel both generation and playback on user interruption.
That state machine should live in your agent logic, not in the UI.
Short Python example: streaming text into a session
The exact SDK methods depend on your stack, but the shape is usually the same: create a session, attach the agent, and feed incremental text/audio into the live participant. If you are using a Python integration layer, keep the code focused on stream boundaries rather than the avatar itself.
Use the docs for the actual SDK call names and object model; the important part is that speech is streamed incrementally and tied to the same live session that carries the avatar video.
Operational details that matter in production
A few practical issues tend to show up once you move beyond a demo:
Backpressure: if TTS is slower than generation, your queue can grow and introduce latency. Cap it.
Fallback behavior: if the TTS provider errors mid-turn, decide whether to retry, apologize, or skip to the next agent action.
Latency budgets: measure time from user end-of-speech to first avatar audio, not just model latency.
Voice consistency: if you switch voices per turn, expect a visible change in animation pacing as well as sound.
For debugging, log three timestamps for every turn: when the model starts, when first audio is published, and when the audio track ends. Those three points usually reveal where latency is accumulating.
How Protoface fits into this flow
This is where a LiveKit-specific avatar layer becomes useful. The Protoface LiveKit plugin is designed to drop an avatar into an existing voice agent so the agent gains a synchronized talking video face without you having to build a separate rendering pipeline. If you are already on LiveKit, the integration point is the agent side, which is exactly where streaming TTS belongs.
In practice, that means you keep your agent logic, TTS streaming, and interruption handling in your application, then attach the avatar to the same realtime session. The plugin handles the avatar/video side of the synchronization. The relevant examples and package details live in the plugin repo and docs, and the integration notes are worth reading before you wire in production audio behavior: GitHub repo and documentation.
REST and SDK usage for session setup
If you prefer to create sessions programmatically before the agent joins, the REST API and Python SDK are the cleanest surfaces. You typically create or reference an avatar, start a realtime session, and then connect your media agent to it. Authentication uses API keys in the standard bearer format.
That request shape is illustrative; the exact request and response fields are documented in the API reference. The useful part is that session creation is a backend operation, so your browser never needs the key and your voice agent can join the session after the fact.
Common implementation mistakes
A few patterns reliably cause trouble:
Buffering the full response before starting TTS: this makes the avatar feel late even if the audio quality is fine.
Driving the avatar from text events instead of audio events: text is not the same thing as speech, especially when retries or rephrasing happen.
Ignoring cancellation: if the user interrupts, the avatar should stop immediately, not finish the queued sentence.
Over-chunking: sending tiny text fragments to TTS can create robotic prosody and unnecessary stream overhead.
If your TTS provider supports partial synthesis, tune for “first useful audio” rather than “first possible audio.” Those are not the same target.
Conclusion
Streaming TTS into a live avatar is mostly about respecting the realtime media model: stream text in manageable chunks, publish audio on a stable clock, and make turn state and cancellation explicit. Once you do that, lip sync becomes a consequence of the audio pipeline rather than a separate rendering problem.
If you are building on LiveKit, a plugin-based avatar integration is the simplest path because it keeps the avatar attached to the same speech stream your agent already uses. For implementation details, refer to docs.protoface.com, and start from the quickstarts in the Protoface ecosystem if you want a working baseline before adapting your own agent logic.