Introduction

When a voice agent has to show a synchronized talking face, the latency budget changes. Audio generation is no longer the only critical path: you also need to decide when to cut text into chunks, because each chunk tends to become one synthesis unit, one transport unit, and one lip-sync scheduling unit. Chunk too conservatively and the avatar starts late or speaks in awkward bursts. Chunk too aggressively and you increase boundary overhead, degrade prosody, and make the face look jittery because the system keeps restarting its alignment work.

This post is about optimizing text-to-speech chunking for lower avatar lip-sync latency without making the speech sound robotic. By the end, you should be able to choose a chunking strategy that fits a realtime avatar pipeline, understand the trade-offs between latency and naturalness, and apply the same principles whether you are streaming tokens from an LLM, feeding a TTS service, or driving a WebRTC avatar.

Why chunking matters in a lip-synced pipeline

In a realtime avatar stack, text usually flows through four stages:

1. The model emits tokens incrementally.

2. Your app accumulates tokens into TTS chunks.

3. The TTS engine returns audio, often as a stream or as short audio segments.

4. The avatar renderer schedules visemes or mouth motion against that audio.

The key latency is not just synthesis time. It is time to first usable audio plus time to first visually stable mouth motion. Those two often diverge. A TTS engine may begin streaming audio quickly, but if you hold text too long before sending it, the avatar cannot start at all. If you emit text too early, you may get more audio start latency from frequent request setup, and the avatar may have to re-anchor mouth timing at every boundary.

For most systems, the practical optimization target is:

• minimize the delay before the first semantically complete chunk,

• keep chunk boundaries aligned with natural speech boundaries, and

• avoid producing so many tiny chunks that network, TTS, and lip-sync schedulers thrash.

This is especially visible in WebRTC-driven avatars, where audio pacing, video frame cadence, and jitter buffering all interact. A good chunking policy reduces “dead air” while preserving enough linguistic context for the TTS engine to generate natural prosody.

Chunk on linguistic boundaries, not arbitrary token counts

The common mistake is to chunk by character count or token count alone. That works for throughput-oriented batch systems, but realtime speech needs more structure. A 20-token chunk might be a complete clause in one case and the first half of a noun phrase in another. The difference matters because TTS models use surrounding context to predict pauses, emphasis, and intonation.

A better policy is to accumulate text until you hit one of these boundaries:

• end of sentence punctuation: ., ?, !

• strong clause boundary: comma, semicolon, em dash

• natural pause words or discourse markers: “so,” “well,” “and then”

• timeout fallback when the model is being verbose and you do not want to wait forever

That gives you a first-pass heuristic:

buffer += new_text
buffer = ""

buffer += new_text
buffer = ""

buffer += new_text
buffer = ""

In practice, you usually want both a minimum and maximum chunk size. A minimum keeps you from emitting unnaturally tiny fragments like “Yes,” or “I think,” on their own. A maximum prevents pathological delays when the model produces a long run-on clause with no punctuation.

Useful starting points:

• Minimum chunk: enough text for at least a phrase, not a word; often 6–12 words.

• Maximum chunk: roughly 1–2 seconds of spoken content, depending on your TTS speed.

• Wait threshold: a short timeout, often 150–300 ms, to flush if punctuation is delayed.

Those numbers are not universal. They just keep you from overfitting to token boundaries that have no speech relevance.

Balance latency against prosody and restart cost

There are two competing failure modes:

Too-small chunks reduce the delay before the first audio frame, but increase request overhead and degrade speech quality. Many TTS systems need enough context to infer phrase-level prosody. If you send a tiny fragment, the result can sound clipped or produce an unnatural pause at the chunk boundary. For lip-sync, frequent chunk boundaries also create repeated scheduling decisions. The avatar may visibly “reset” its mouth shape if you do not merge timing across chunks.

Too-large chunks preserve prosody but push the first audio later. In a conversational agent, that is often the more user-visible problem. Humans notice silence immediately. If the avatar does not begin moving until 800 ms after the user stops speaking, the interaction feels sluggish even if the eventual speech is high quality.

The right answer is usually to split on boundaries that preserve meaning, then dynamically adjust. A simple adaptive strategy looks like this:

1. Buffer until punctuation or a clause boundary.

2. If the model stalls, flush after a short timeout.

3. If the current chunk is too short, merge it with the next one unless the turn is about to end.

4. Prefer earlier flushes near user turn transitions, where responsiveness matters more than perfect cadence.

That last point is important. In a back-and-forth voice agent, users are often more tolerant of slightly worse prosody than of slow turn-taking. If your speech pipeline is being interrupted by barge-in detection or user speech, you should bias toward faster partial output. If the agent is delivering a longer monologue, you can afford to wait for fuller clauses.

Implement chunking in the application layer

There is no universally correct chunker because the right policy depends on your language model, TTS, and avatar transport. So the best place to optimize is usually your application layer, where you can inspect token flow and decide when to flush.

A practical Python pattern is to attach a small state machine to your stream of text deltas:

import re

return None

import re

return None

import re

return None

This is intentionally simple. Real systems usually add more rules: avoid splitting numbers, abbreviations, URLs, and code; merge leading conjunctions with the next clause; and hold back a trailing fragment if it clearly completes the sentence with the next token.

There is also an architectural choice: whether the chunker should live next to the LLM stream, next to the TTS client, or inside a voice-agent framework. If you own the full stack, putting it close to the token stream gives you the most control. If you are integrating into an existing agent runtime, putting it near the TTS boundary is often easier because it sees the exact payload that will be synthesized.

Measure the right latency, not just TTS duration

Developers often measure TTS “response time” and stop there. For avatar systems, that hides the real problem. You should instrument at least four timestamps:

• first token received from the LLM

• chunk emitted to TTS

• first audio byte/frame received from TTS

• first visible mouth motion in the avatar

The gap between first token and chunk emitted is your chunking latency. If that number is large, the problem is in your buffering policy. The gap between chunk emitted and first audio is your synthesis latency. If that number dominates, you need a faster TTS tier or smaller requests. The gap between first audio and visible motion is your render/sync latency, which depends on how the avatar pipeline consumes audio and aligns mouth movement.

A useful debugging approach is to log chunk sizes alongside boundary reasons:

{
}

{
}

{
}

When latency is bad, this tells you whether the fix is linguistic, transport-related, or renderer-related. If all your chunks are ending because of timeout, your model output may be too sparse or your wait threshold too low. If chunks are ending correctly but the avatar still starts late, the bottleneck is somewhere else.

How this maps to Protoface

Protoface sits at the point where these trade-offs become visible to users. If you are using the LiveKit Agents plugin, for example, your voice agent can feed speech into a realtime avatar without you having to build the video-face side yourself. The same chunking principles still apply: the earlier and cleaner you hand off semantically complete text or audio, the sooner the avatar can produce stable lip motion.

For developers integrating a voice agent, the plugin and examples in the GitHub repo are the right place to see how the handoff is wired up: github.com/protoface-ai/protoface-plugin-pipecat. If you are working directly against the platform, the docs at docs.protoface.com cover the current API surface, SDKs, and session model. The exact fields and request shapes may change, so keep your chunking logic independent of any one provider’s payload format.

Conclusion

Lower lip-sync latency is mostly about respecting speech structure. Chunk on linguistic boundaries, enforce small timeouts so the agent never feels stuck, and measure chunking delay separately from synthesis and rendering delay. The goal is not “smallest possible chunks”; it is “the earliest chunk that still gives the TTS engine enough context to sound natural and lets the avatar start moving cleanly.”

If you are building a realtime avatar pipeline, start by instrumenting your token stream and experimenting with boundary-aware flushing. Then tune the minimum/maximum chunk sizes against actual conversations, not synthetic benchmarks. When you are ready to wire the avatar side in, the documentation at docs.protoface.com and the integration examples in the relevant repo should be enough to get you from theory to a production-safe implementation.