Streaming TTS vs Chunked TTS for Realtime AI Avatars: Which Integration Pattern Is Better?
Compare streaming vs chunked TTS for realtime AI avatars, with trade-offs in latency, lip-sync, interruptibility, and control.
Introduction
When you add a talking face to a realtime voice agent, text-to-speech is no longer just “generate audio and play it.” The audio pipeline becomes part of your animation pipeline. That means the way you feed TTS into the avatar matters: it affects latency, lip-sync quality, interruptibility, and how much control your application has over the user experience.
This post compares two integration patterns developers commonly use for realtime avatars: streaming TTS and chunked TTS. By the end, you should be able to choose the right pattern for a given agent, understand the trade-offs in latency and synchronization, and avoid the failure modes that make avatars look robotic or feel laggy.
What “streaming” and “chunked” actually mean
At a systems level, both patterns are just ways of delivering synthesized speech to an avatar renderer. The difference is in how much of the utterance exists before playback begins.
Streaming TTS: the model emits audio incrementally as tokens or short phonetic units are produced. Playback can begin before the full sentence is ready.
Chunked TTS: you split text into sentence-sized or phrase-sized chunks, synthesize each chunk independently, and play them in sequence.
For realtime avatars, these patterns influence three separate clocks:
Text generation clock — when the LLM decides what to say.
Audio clock — when bytes of PCM/Opus are available for playback.
Visual clock — when the avatar can start mouth motion and sustain it with believable timing.
The core engineering problem is not “which is faster?” in isolation. It is: which pattern keeps these clocks aligned enough to feel natural while remaining interruptible and predictable?
Streaming TTS: lowest perceived latency, hardest to coordinate
Streaming TTS shines when you want the avatar to start talking as early as possible. In a voice agent, that usually means the user experiences a quicker “I’m responding” signal, which is particularly useful for short acknowledgements, live conversation, and turn-taking in busy realtime sessions.
The upside is obvious: the system can begin audio output before the whole sentence is finished. The downside is that your avatar renderer and any downstream audio consumers need to tolerate partial, evolving speech. That complicates lip-sync timing and barge-in handling.
Typical streaming flow:
User speech is transcribed or otherwise turned into text.
The LLM starts generating the response.
TTS begins emitting audio frames for the initial segment.
The avatar starts animating on the first playable audio.
More audio arrives and the renderer keeps updating visemes or mouth cues.
Streaming is usually the right choice when:
turn latency matters more than perfect prosody boundaries,
responses are long enough that waiting for full synthesis would feel slow,
your agent may be interrupted frequently by the user, or
you can support mid-utterance cancellation cleanly.
But there are some gotchas:
Boundary instability: if the model revises the text mid-generation, audio that already played cannot be retracted.
Prosody drift: streaming systems may sound slightly less coherent across long clauses because future context is not available yet.
Visual jitter: if your lip-sync driver depends on unstable timing metadata, the face can appear to “chatter” or lag.
Complex cancellation: you need a clean way to stop audio and animation together when the user barges in.
In practice, streaming TTS is best when your runtime can treat speech as a live media stream rather than a finished asset.
Chunked TTS: simpler control, better predictability
Chunked TTS is the conservative integration pattern. Instead of streaming raw synthesis, you first segment the response into chunks that are semantically stable enough to speak independently. Each chunk is synthesized, buffered, and then rendered.
This approach adds a bit of latency, but it gives you much better control over timing and sequencing. You know where each chunk starts and ends, which makes it easier to align mouth motion, handle fallbacks, and implement retries without corrupting an in-flight utterance.
Chunking is often a good fit for:
deterministic assistant prompts,
systems that need clean sentence-level lip-sync,
applications where synthesis errors must be retried per chunk, and
pipelines that already batch text for moderation, translation, or post-processing.
The trade-off is that “chunked” does not automatically mean “good.” If your chunks are too large, the user waits longer. If they are too small, the speech can sound chopped or overly segmented. A bad chunker will split on punctuation too aggressively or break semantic units in awkward places.
Good chunking usually tries to respect:
sentence boundaries when available,
pause-worthy clauses,
acoustic continuity across short utterances, and
the ability to stop between chunks without sounding abrupt.
Chunked TTS is easier to reason about because each chunk is effectively an atomic unit. If a call fails, you can retry the chunk. If the user interrupts, you can stop at a clean boundary. If your avatar system wants explicit start/end events for a spoken phrase, chunking maps neatly onto that model.
Choosing between them: what actually matters in production
The decision usually comes down to your system’s dominant constraint.
Choose streaming TTS if your product is optimized for conversational responsiveness and you can tolerate more complexity in synchronization. This is common in voice agents, live support flows, and realtime conversational video where the avatar should feel “present” as quickly as possible.
Choose chunked TTS if you need predictability, easier retries, and cleaner media boundaries. This is often better for scripted assistants, longer expository responses, or applications where the avatar is only one part of a larger UI and the user is less sensitive to a few hundred milliseconds of extra delay.
Here is the practical rule I use:
If your agent frequently interrupts itself or the user interrupts it, streaming is usually worth the complexity.
If your agent speaks in relatively complete thoughts and correctness matters more than raw immediacy, chunked is usually safer.
Also remember that “streaming vs chunked” is not always a binary choice. Many production systems stream within a chunk boundary: they buffer a phrase, then stream audio internally to reduce start latency while preserving a stable utterance boundary to the rest of the app. That hybrid model often gives the best balance.
Implementation details that decide whether lip-sync looks good
For avatars, the biggest mistake is treating audio delivery as separate from mouth animation. Realistic lip-sync depends on consistent frame timing and a sensible mapping from speech audio to visemes or mouth states.
Keep these points in mind:
Use one source of truth for playback state. The audio player and the avatar renderer should agree on when speech starts, pauses, resumes, and ends.
Don’t let buffering hide latency indefinitely. If you overbuffer to smooth playback, the face may look delayed even when audio sounds fine.
Handle barge-in explicitly. When the user starts speaking, stop both audio and animation in the same logical event.
Measure end-to-end latency, not just TTS time. The real metric is time from user turn end to the avatar visibly speaking.
A useful debugging trick is to log timestamps for four events: response generation start, first audio byte available, first audio playback, and first visible mouth motion. If those diverge too much, the avatar will feel off even if each subsystem is individually “fast.”
How Protoface fits into this pattern choice
Protoface is designed for this exact integration problem: taking a voice agent and giving it a synchronized talking face without forcing you to build the avatar media layer from scratch.
If you are already running a voice agent in LiveKit, the LiveKit-based quickstarts and the LiveKit plugin are the most relevant surface area. The plugin lets you drop an avatar into the agent so speech and video stay coordinated at the media layer rather than trying to glue them together in application code. That is particularly useful if you are choosing between streaming and chunked TTS, because the avatar side can consume whatever audio pattern your agent produces while preserving synchronization.
If you want to create or manage sessions directly, the REST API and Python SDK are the cleaner fit. A minimal API call looks like this:
And the Python SDK is the natural choice when your app is already orchestrating agent state in Python:
The exact request fields and session lifecycle details are in the docs, but the integration idea is straightforward: keep your speech pipeline explicit, then let the avatar surface consume a stable session or media stream.
Conclusion
Streaming TTS gives you the lowest perceived latency and the most natural “instant response” behavior, but it requires tighter coordination and better cancellation semantics. Chunked TTS is easier to reason about and easier to debug, but it adds latency and can make the agent feel less alive if chunks are too coarse.
For realtime AI avatars, the right choice is usually driven by user experience goals: responsiveness versus determinism. If you are building a conversational agent that must feel live, streaming is often the better default. If you need cleaner boundaries, safer retries, or more predictable visual timing, chunked is a solid choice.
If you are implementing this now, start by defining your media contract: when does speech start, what event stops it, and how does the avatar learn about those boundaries? Then validate the whole path with real conversations, not just synthetic benchmarks. The docs at docs.protoface.com are the right place to map those concepts onto the available integration surfaces.