How to Keep a Voice-Driven Shopping Assistant Responsive Under Traffic Spikes
How to keep a voice shopping assistant responsive under spikes with backpressure, cancellation, bounded queues, and separate media scaling.
Introduction
Traffic spikes are where voice shopping assistants tend to fall apart: the ASR pipeline backs up, the LLM starts queuing, TTS latency creeps up, and the user hears dead air or gets interrupted mid-sentence. If your assistant also has a realtime avatar, you now have a synchronization problem on top of a latency problem.
This post is about keeping the experience responsive under load without pretending infinite scale is free. By the end, you should be able to reason about your bottlenecks, apply backpressure instead of letting requests pile up, and choose the right place to shed load so users still get a fast, coherent shopping interaction.
Start with the actual latency budget
For a voice-driven shopping assistant, “responsive” usually means three things:
Low time to first response after the user stops talking.
No stale turn-taking, meaning the assistant doesn’t answer to old audio after the user has already moved on.
Stable media playback, meaning audio and avatar video stay in sync even when downstream services slow down.
The simplest way to keep this under control is to budget latency per stage and treat each stage as an independently failing dependency:
Audio ingestion and VAD/turn detection
ASR
Dialog/LLM orchestration
Retrieval and commerce APIs
TTS
Avatar rendering and streaming
Under normal load, each stage can absorb a little jitter. Under a spike, the problem is not average latency; it is queue growth. Once a queue starts growing faster than it drains, end-to-end latency becomes unbounded until you cut off work.
Use backpressure, not unbounded concurrency
The most common failure mode is allowing each incoming voice turn to spawn a full request chain regardless of the system’s current state. That works until bursty traffic creates a convoy: ASR workers saturate, LLM calls pile up, and the assistant responds to requests that are already irrelevant.
Instead, define explicit admission control at the turn level:
Cap concurrent conversations per worker or region.
Cap concurrent turns per conversation, usually one active turn at a time.
Drop or defer low-value work when queues exceed thresholds.
Cancel in-flight work when barge-in or user interruption makes it obsolete.
That last point matters a lot for voice. If the user starts speaking over the assistant, continuing to synthesize the old response wastes compute and makes the conversation feel sluggish. Cancellation should propagate all the way down the stack: stop TTS generation, stop avatar playback, and stop any pending tool calls that are no longer needed.
Make turn state explicit
Voice agents feel “smart” when they manage state well, not when they maximize throughput. Track a small set of explicit states for each conversation:
listening
processing
speaking
interrupted
idle
That lets you implement sensible policies:
If the assistant is speaking and the user barges in, interrupt and reset to listening.
If an LLM call is still pending after a timeout, return a short fallback response instead of holding the floor indefinitely.
If a session already has a queued turn, coalesce duplicate transcripts or discard stale partials.
For shopping, this is especially important because user intent is often mutable. A request for “black running shoes” can become “actually make that size 11” halfway through. If your pipeline is still processing the first turn when the correction arrives, you want the correction to win.
Prefer short, composable prompts and bounded retrieval
At spike time, long prompts and unbounded retrieval are latency multipliers. Every extra token adds model latency, and every extra retrieval call adds variability. Keep the conversational context compact and current.
A practical pattern:
Keep a rolling summary of the conversation rather than replaying the full transcript.
Limit retrieval to a small top-k and short snippets.
Gate expensive tools behind intent detection. Don’t hit the catalog API unless the user’s request really needs it.
Prefer cached product metadata for common queries like pricing, availability, and shipping estimate.
When traffic is high, your goal is not to answer every possible question with maximum completeness. It is to answer the important ones quickly and degrade gracefully on the rest.
Stream early, but only if you can finish coherently
Streaming is one of the best ways to reduce perceived latency, but only if the downstream media path can keep up. For a voice assistant, that means you should start TTS as soon as the first stable chunk of the response is available, while continuing to generate the rest of the answer in small increments.
That trade-off has consequences:
Early audio reduces time-to-first-byte for the user.
But streaming partial responses makes cancellation and interruption more important.
If the assistant starts speaking too early and then changes direction, users notice incoherence immediately.
A useful rule: only stream when the response is sufficiently committed that the opening sentence will remain valid even if downstream tools are slow. For shopping, a safe opening is often a confirmation or a concise summary, followed by details once retrieval returns.
Keep commerce dependencies out of the critical path where possible
Shopping assistants often call product search, cart, inventory, shipping, or order-status services. Those APIs are rarely optimized for sub-second conversational latency, and they often become the bottleneck during traffic spikes.
There are a few ways to keep the assistant responsive:
Cache common reads with a short TTL, especially product details and availability.
Separate “answer now” from “complete later”. For example, say “I’m checking that now” and follow up when the slower API returns.
Use timeouts that match the conversation. A voice turn should not wait indefinitely for an upstream service.
Return partial results. If price is available but shipping isn’t, say so.
Do not let one slow inventory API consume the same worker pool that handles turn detection or audio streaming. Isolate the critical media path from the business-logic path. If you have to choose, keep the assistant talking cleanly and let the commerce lookup finish asynchronously.
Scale the media layer separately from the application layer
WebRTC-style media transport, audio generation, and avatar streaming have different scaling characteristics than your API and business logic. If you tie them together, a spike in one can starve the other.
A more robust layout is:
Ingress layer for audio and session setup
Turn-processing workers for ASR, orchestration, and tools
Media layer for TTS and video/avatar rendering
Shared control plane for session state, rate limits, and observability
This separation lets you scale the slow parts independently. It also makes it easier to apply different queueing policies. For example, you might allow a short queue for ASR but zero queue for playback, because playback delay is directly visible to the user.
Where Protoface fits
If your assistant needs a synchronized talking face, Protoface is the layer you add after you’ve made the voice pipeline predictable. The practical integration point for most voice-agent stacks is the LiveKit Agents plugin, which drops a realtime avatar into the agent so the video face stays aligned with speech delivery. See the plugin repo and examples in GitHub if you’re wiring this into an existing LiveKit-based stack.
The main operational advantage is that the avatar becomes part of the same session model as the voice agent. That means you can keep turn state, cancellation, and playback aligned instead of treating video as a separate best-effort feature. In a spike, that matters: if you shed load, you want to shed it consistently. Don’t keep rendering a face for a response you already dropped.
If you’re working lower-level or integrating programmatically, the REST API and Python SDK let you create and manage avatars and realtime sessions from your own control plane. Exact request fields and session shapes are in the docs, but the pattern is straightforward: authenticate with your API key server-side, create the avatar/session you need, then bind it to the active conversation.
Keep those calls on the server. For website embeds, the customer-managed iframe model is useful when you want to avoid exposing credentials in the browser and still enforce origin allowlists and rate limits. That is a good fit for low-friction demos and lightweight product experiences, but the same core rule applies: your assistant still needs backpressure and cancellation to survive traffic.
Operational guardrails that actually help
Once the code path is sane, the remaining work is operational:
Queue depth alarms on turn processing, not just CPU.
p95 and p99 latency per stage, so you can see whether the spike is in ASR, tools, or TTS.
Session-level logging with turn IDs so you can trace stale responses.
Rate limits and quotas per tenant or origin to prevent one client from starving everyone else.
Fallback UX such as “I’m still checking that” or a text handoff when the system is overloaded.
One subtle but important metric is staleness: the percentage of responses delivered after a newer user turn has already begun. That number often tells you more about perceived responsiveness than raw service latency.
Conclusion
Keeping a voice-driven shopping assistant responsive under load is mostly an exercise in controlling queues and cancellation. Cap concurrency, make turn state explicit, keep the critical path short, and separate media playback from slower commerce lookups. If you stream, stream early but only with responses you can still stand behind. If you shed load, shed it intentionally and consistently across voice and avatar output.
If you’re adding a realtime face to an existing voice agent, start with the docs at docs.protoface.com and wire the avatar in only after your backpressure and interruption logic are solid. That order saves a lot of debugging later.