Best Practices for Low-Latency TTS and STT in AI Shopping Assistants
Low-latency TTS/STT best practices for AI shopping assistants: streaming, barge-in, partials, and latency budgeting.
Introduction
Low-latency TTS and STT are not optional in an AI shopping assistant. If the assistant takes too long to respond, users interrupt it. If the transcript lags behind speech, the model loses turn-taking context, repeats itself, or answers the wrong question. In a shopping flow, that means abandoned sessions and a chat experience that feels brittle instead of helpful.
The goal is not just “fast.” The goal is predictable end-to-end latency: the assistant should detect when the user starts speaking, stop talking quickly, start decoding partial speech early, and begin streaming audio back before the full answer is ready. By the end of this post, you should be able to reason about the latency budget, choose the right streaming strategy, and avoid the common implementation mistakes that make voice shopping agents feel slow.
Start with the latency budget, not the model choice
Developers often begin by comparing TTS and STT providers in isolation. That is useful, but the user experience is dominated by the total path:
mic capture → transport → VAD/endpointing → STT partials → agent reasoning → TTS first audio chunk → playback
Each stage contributes a few tens or hundreds of milliseconds. When the budget gets blown, it is usually because of compounding delays:
Audio chunking: sending 1-second buffers instead of 20–100 ms frames delays detection and transcription.
Endpointing too late: waiting for long pauses before finalizing an utterance makes the agent feel unresponsive.
Serial processing: doing STT, then waiting for a full transcript, then calling the LLM, then starting TTS adds avoidable latency.
Blocking playback: holding TTS until the full response is generated prevents the user from hearing the first useful words early.
For shopping assistants, the practical target is usually: keep interruption handling under a few hundred milliseconds, start partial transcripts within a similar range, and stream the first TTS audio as soon as the first clause is ready. Exact numbers depend on your stack and network, but the architecture matters more than the nominal model speed.
Stream speech end-to-end, and let the agent operate on partials
The biggest win is to treat speech as a live stream, not a request/response blob. That means your STT provider should emit partial hypotheses continuously, and your agent should be able to reason over them without waiting for final punctuation.
In practice, that changes the shape of your application:
Use small audio frames from the client or transport layer.
Run VAD or endpointing to detect speech boundaries, but keep the thresholds conservative enough to avoid cutting off intent-bearing pauses.
Feed partial transcripts into the agent so it can prepare likely responses early.
Start TTS from the first stable chunk of the response instead of waiting for the full answer.
That last point is especially important for shopping assistants, where the response often has a natural prefix: “Sure — I found two options…” or “The size chart says…” You do not need to synthesize the entire paragraph before the user hears anything useful.
A good mental model is that the assistant should be able to overlap three things: listening, thinking, and speaking. If those are serialized, you will lose to latency even with excellent models.
Use short, interruptible audio turns
Shopping assistants need fast barge-in behavior. Users will interrupt when they hear a wrong item, a slow answer, or a pricing detail they already know. If your system cannot stop TTS promptly, the interaction feels broken.
Design for interruption at the transport and playback layers:
Cancel generation as soon as the user starts speaking. Do not wait for the current sentence to finish.
Stop audio playout immediately. Keeping 500 ms of queued audio after barge-in is enough to make the assistant feel unresponsive.
Preserve conversational state. The interrupted answer should not wipe out the next turn’s context.
On the STT side, make sure the system can distinguish speech start from background noise. In a shopping environment, you may have keyboard clicks, music, or other voices. Aggressive endpointing can reduce latency, but if it misfires, it causes truncation and retriggers, which are worse than a slightly slower finalization.
Optimize for first-token latency, not just total completion time
For TTS, the metric that matters most to users is often the time to first audible output. A slightly slower model that begins speaking earlier can feel faster than a “faster” model that buffers too much before emitting audio.
There are a few concrete tactics here:
Chunk responses into speakable units. Synthesize a clause or sentence fragment, not an entire multi-paragraph answer.
Keep prompts compact. Verbose system prompts and large conversation histories increase inference time and can delay response planning.
Precompute recurring phrases. For common shopping phrases like greetings, confirmations, or handoff messages, consider caching audio or at least warm-starting the voice pipeline.
Match voice quality to use case. Ultra-high-fidelity TTS can be worth it for premium experiences, but lower tiers may be the right trade-off for high-volume product discovery flows.
For STT, the equivalent optimization is reducing the time to stable partials. If the recognizer waits too long before emitting anything, your agent cannot start deciding whether the user asked about price, availability, or comparison.
Keep the voice agent state machine simple
Many latency bugs are really state bugs. A shopping assistant usually needs a small set of states: idle, listening, thinking, speaking, and interrupted. The transitions between them should be explicit.
A few practical rules:
Speaking should be cancellable. Do not treat speech synthesis as a fire-and-forget background task.
Listening should resume quickly after TTS ends. Long post-speech dead zones make the assistant feel inattentive.
Partial transcripts should update intent, not overwrite it blindly. If the user says “show me the blue one” after “show me the…”, the agent should be able to refine the current hypothesis.
Do not wait for perfect NLU before acting. In shopping flows, the safe move is often to start preparing candidate answers or product filters as soon as the intent is likely.
One subtle issue: if you generate long spoken responses from uncertain input, you can end up “committing” to the wrong answer too early. A common pattern is to keep the first spoken line short and confirm key details before diving deeper.
Implementation pattern: stream STT, stream TTS, and glue them together
Here is a simplified Python sketch showing the shape of the application. The exact SDK fields and event names depend on your stack, but the pattern is the same: receive partial transcripts, decide when the user has paused, and stream synthesized audio back as soon as you have a stable response fragment.
If you are integrating through a voice-agent framework, the same principle applies there as well: keep the speech pipeline streaming all the way through the agent and into the avatar layer. The important part is that the avatar never becomes a bottleneck in the audio path.
How Protoface fits into the stack
Protoface is useful when you want the assistant to have a synchronized talking face without building the video layer yourself. In a LiveKit-based voice agent, the LiveKit/agent quickstart pattern is the relevant one: attach the avatar to the existing voice pipeline so speech, lip sync, and playback stay aligned.
That matters for latency because the avatar should follow the same low-latency constraints as the rest of the stack. If your agent is already doing streaming STT and streaming TTS, the avatar layer should consume that audio in real time rather than forcing a separate buffering model. The operational details — API keys, session creation, and exact event fields — are documented in the docs.
If you are working in Python, the SDK is a reasonable entry point for programmatic session control and avatar management; if you are embedding in a browser, the customer-managed iframe approach avoids exposing any backend credentials in the client. For a shopping assistant, that iframe model is especially handy when you want to add a voice+face interaction to a product page without turning the page app into a media platform.
Common gotchas in production
A few issues come up repeatedly when teams ship realtime shopping assistants:
Overly long prompts: they inflate every turn and make latency worse.
No backpressure handling: when STT or TTS falls behind, queued audio keeps growing and turns become stale.
False endpointing: the assistant interrupts users mid-thought and asks clarifying questions too early.
Ignoring regional latency: if your users are global, the nearest media path matters as much as model choice.
Testing only happy paths: real shopping traffic includes interruptions, side conversations, bad microphones, and noisy rooms.
Measure with real traces. Look at time to first partial transcript, time to final transcript, time to first TTS audio, and interruption handling. Those four numbers will tell you more than a generic “latency” metric.
Conclusion
Low-latency TTS and STT are mostly an exercise in pipeline design: stream early, finalize carefully, overlap listening and speaking, and make interruption a first-class behavior. For shopping assistants, that is the difference between a voice UI that feels conversational and one that feels like a laggy IVR with a face.
Start by instrumenting your current turn path, then remove serialization one stage at a time. If you need a realtime avatar layer on top of an existing voice agent, check the relevant quickstarts, then verify the session and API details in the docs. From there, tune the budget with real users and noisy environments, not just synthetic benchmarks.