Introduction

If you’re building a voice agent with a visible face, the latency budget is no longer just “time to first token.” You are coordinating three separate streams:

• STT: turn incoming audio into text fast enough for turn-taking.

• TTS: synthesize speech with low first-audio latency and stable cadence.

• Avatar rendering: generate and ship lip-synced video without stalling the conversation.

The common failure mode is to optimize each subsystem in isolation and still end up with a sluggish interaction. The goal is a system that feels responsive at the human timescale: the user finishes speaking, the agent starts acknowledging quickly, and the face begins moving in sync with the voice without visible buffering. By the end of this post, you should have a concrete tuning approach for each stage, plus a mental model for where latency actually comes from and what trade-offs are worth making.

Start with the latency budget, not the model choice

For assistive conversational UX, perceived responsiveness matters more than raw throughput. A useful rule is to budget against the first 500–1200 ms after the user stops speaking. If the system can produce an acknowledgement inside that window, users typically perceive it as immediate enough for dialog. Beyond that, every extra hop is noticeable.

Break the end-to-end path into measurable segments:

1. Capture and VAD: microphone frame capture and voice-activity detection.

2. STT partials: first interim transcript and endpoint detection.

3. LLM turn: first response token or first sentence boundary.

4. TTS first audio: time to synthesized audio output.

5. Avatar render: time until the face starts animating and lip sync aligns.

6. Transport: how long media takes to reach the browser.

Don’t assume the slowest ML component is the only problem. I often see a “fast” TTS model lose to a slower one simply because the application waits for a full sentence before starting synthesis. Likewise, avatar rendering can add a perceptible delay if the video pipeline buffers too aggressively before playback begins.

Tuning STT for responsiveness

Use partials, but only if your turn-taking logic can handle them

Streaming STT is useful because it lets you detect intent before the user fully finishes a sentence. That can reduce both turn-taking delay and overall agent latency. The cost is instability: partial transcripts change, punctuation arrives late, and endpoint detection can be wrong on hesitations or short pauses.

A practical pattern is:

• Use interim transcripts for UI feedback and early intent inference.

• Use a stronger endpoint rule to decide when to commit a turn.

• Ignore minor transcript revisions unless they change semantics.

If your STT provider exposes phrase hints, custom vocabulary, or domain terms, use them. For assistive agents, those terms are often proper nouns, product names, or support-ticket jargon. A modest accuracy improvement in those tokens can reduce retries and downstream LLM ambiguity more than a generic model upgrade.

Endpointing is usually the hidden latency lever

Endpointing determines when the system decides the user is done speaking. Too conservative, and you add dead air. Too aggressive, and you interrupt natural pauses. The right setting depends on the interaction style:

• Fast, task-oriented flows: shorter end-of-utterance timeout, earlier commit.

• Open-ended conversation: longer timeout, more tolerance for pauses.

• Interruptible assistants: favor low latency, but support barge-in and cancellation.

In practice, a good implementation watches both audio energy and transcript stability. If the last few partials are consistent and silence has persisted for a short threshold, commit. If the transcript is still changing rapidly, wait a bit longer. This is more robust than relying on a fixed silence timer alone.

Tuning TTS for first-audio latency and naturalness

Optimize for time to first phoneme, not just total synthesis time

Users notice when an assistant “thinks” before speaking. For TTS, the metric that matters is first-audio latency: how quickly the system emits the opening phoneme or frame of audio after the text is available. Many engines are fast at producing an entire utterance, but that does not help if they buffer too much before streaming anything.

The most important optimization is to start synthesis early and incrementally. Instead of waiting for the LLM to finish a long response, begin TTS on the first stable clause or sentence. That means your orchestration layer should support streaming text-to-speech or at least chunked synthesis. You get lower latency at the cost of more complex prosody management, because the engine has less context for punctuation and emphasis.

Chunking strategy matters more than you think

If you split text too finely, the speech sounds robotic and sentence prosody suffers. If you split too coarsely, you delay the start of playback. A reasonable compromise is:

• Wait for a clause boundary or sentence boundary when possible.

• Allow early start on short acknowledgements like “Sure” or “One moment.”

• Avoid synthesizing token-by-token unless your TTS engine is explicitly designed for that.

Also pay attention to voice selection and audio format. A higher-quality voice can have slightly more synthesis overhead, and some codecs add startup cost. For browser delivery, the main question is how quickly your pipeline can emit decodable audio frames and whether the playback path can start before the entire response is buffered.

Watch for cancellation and interruption behavior

Assistive agents should not keep talking after the user starts speaking. That means your pipeline needs cancellation at multiple layers: stop TTS generation, stop audio transport if possible, and update the avatar state immediately. If cancellation only happens after the current sentence ends, the experience feels laggy even if your raw TTS latency is good.

This is one of the simplest places to improve perceived responsiveness: if the agent can be interrupted cleanly, users forgive a lot of backend latency because they retain control.

Tuning avatar rendering and lip sync

Keep the render path simple and stream-friendly

Avatar rendering adds a video pipeline on top of speech. The usual pitfall is to treat it like a static video asset rather than a realtime media stream. For conversational avatars, the relevant constraints are:

• How quickly the face begins moving after speech starts.

• Whether lip sync tracks the current audio stream with minimal drift.

• How much buffering is needed to avoid stutter or frame drops.

The best strategy is to keep the avatar path tightly coupled to the TTS audio timeline. If the renderer is waiting on too much future audio or video context, it can look smooth but feel delayed. If it starts too early with insufficient alignment, the mouth motion will be visibly off.

Balance frame rate against startup cost

Higher frame rates usually improve perceived smoothness, but they also increase compute and transport load. For a realtime avatar, the first visible frame is more important than maximizing sustained FPS. If you need to choose, prioritize:

1. Low time to first frame.

2. Stable lip sync on the frames you do render.

3. Enough frame rate to avoid obviously choppy motion.

If the avatar is rendered server-side and streamed to the browser, the network path matters too. WebRTC-style transport is generally preferable for realtime media because it is built for low-latency, jitter-tolerant delivery. But even a good transport cannot save you from excessive server-side buffering, so keep the media pipeline lean.

Measuring the right things

You cannot tune this well by feel. Instrument timestamps at each boundary and log them per turn. At minimum, record:

• user_speech_end

• stt_finalized

• llm_first_token

• tts_first_audio

• avatar_first_frame

• client_playback_start

Then compute deltas between them. This quickly shows whether your bottleneck is endpointing, model response time, speech synthesis startup, or media delivery. It also makes regressions obvious when you change providers or model versions.

One subtle but common issue: the “fastest” configuration can be worse in practice if it causes more interruptions, transcript corrections, or awkward prosody. Measure not just latency, but also correction rate and conversation success. A slightly slower turn that feels confident is often better than a twitchy one that repeatedly restarts.

Where Protoface fits in this pipeline

In a realtime voice agent, the avatar layer should not be a separate product decision. It should be a tightly integrated media target for your existing STT/LLM/TTS stack. Protoface is built for that role: you connect a voice agent, then let the platform handle the synchronized talking face over a realtime session.

If you are using LiveKit Agents, the quickest path is the Protoface plugin. The plugin drops an avatar into the agent so the speaking voice and lip-synced video stay coordinated without you wiring the media plumbing yourself. The same basic pattern applies whether you are starting from Python or from an existing voice-agent stack. For example:

from livekit.plugins.protoface import ProtofaceAvatar

)

from livekit.plugins.protoface import ProtofaceAvatar

)

from livekit.plugins.protoface import ProtofaceAvatar

)

If you are orchestrating sessions directly, the REST API is the cleanest integration point. You create or manage avatars and realtime sessions with bearer-authenticated requests from your server, which keeps API keys out of the browser. The SDK is useful when you want the same control flow in Python without hand-rolling HTTP calls. See the docs for the exact request bodies and session fields: https://docs.protoface.com.

For teams already on LiveKit, the relevant repo is also a good place to inspect end-to-end examples: https://github.com/protoface-ai. The important part is not the surface area, but the integration boundary: keep STT, LLM, and TTS decisions in your agent, and treat the avatar as a low-latency, synchronized media sink.

A practical tuning workflow

If you are starting from a slow or uneven experience, tune in this order:

1. Measure each stage with timestamps.

2. Reduce endpoint latency without causing frequent interruptions.

3. Start TTS earlier using stable text chunks.

4. Trim avatar buffering so motion begins as soon as audio does.

5. Test barge-in to confirm cancellation is immediate.

That order matters because it attacks the biggest perceived delay first. If the user is waiting for the agent to start speaking, shaving 200 ms off avatar rendering helps less than shaving 200 ms off STT endpointing or TTS startup.

Conclusion

Faster assistive AI responses are mostly a systems problem. STT determines when a turn is ready, TTS determines when speech becomes audible, and avatar rendering determines when the face feels alive. The best results come from tuning all three as one pipeline: low-latency endpointing, streaming synthesis, minimal video buffering, and robust cancellation.

Once you have timestamps for each stage, the bottlenecks are usually obvious. From there, it becomes an engineering exercise instead of guesswork. If you want a working avatar layer you can drop into an existing realtime voice stack, start with the docs at docs.protoface.com and the relevant integration examples in the linked repos.