Reducing Latency in a SvelteKit Realtime Shopping Assistant with WebRTC
How to reduce latency in a SvelteKit realtime shopping assistant with WebRTC, streaming, and synchronized avatars.
Introduction
When people say a “realtime shopping assistant,” they usually mean a voice-first agent that can answer questions, compare products, and guide a purchase while staying responsive enough that the interaction feels like a live conversation. The hard part is not just generating good answers. It is keeping the full loop tight: microphone capture, network transport, ASR, LLM inference, TTS, and any visual output all need to stay within a latency budget that users will tolerate.
If you add a talking avatar on top, the system gets more sensitive. A face that starts speaking late, or whose lip sync drifts behind the audio, makes the assistant feel broken even if the underlying answer is correct. In this post, we will look at where latency actually comes from in a SvelteKit realtime shopping assistant, how WebRTC changes the transport story, and what to do in the browser and backend to keep the experience interactive. We will also show where Protoface fits when you want a synchronized talking face without building the avatar pipeline yourself.
Start with the latency budget, not the UI
For conversational agents, “fast” usually means “first useful response within a few hundred milliseconds.” That is not the same as “full answer complete.” You want the system to start responding quickly, then continue streaming. A practical budget might look like this:
Audio capture and client-side VAD: < 50 ms
Transport to the agent: 20-80 ms, depending on path
ASR partials: 100-300 ms for first words
LLM first token: 100-500 ms
TTS first audio chunk: 100-300 ms
Avatar render start: ideally aligned with the first audio chunk
Those numbers are not guarantees; they are just a useful mental model. The main point is that latency compounds across stages. If you add a polling API call, a long server hop, or a blocking render step, you lose the feeling of continuity.
In a SvelteKit app, the browser should do as little work as possible before media starts flowing. Avoid building the assistant around request/response semantics. If the UX is realtime, the transport should be a realtime transport too. That means WebRTC for media, or at least a WebSocket-backed streaming path for control messages. WebRTC is usually the right choice when you are moving audio and video between browser and agent because it handles jitter, congestion control, NAT traversal, and low-latency media delivery.
Use WebRTC for media, not for everything
One common mistake is to push all state through the media plane. WebRTC is great for audio/video, but product catalog lookups, cart updates, and session metadata are better handled through ordinary HTTP or server-side application code. Keep the responsibilities separate:
Media plane: live microphone audio, agent audio, avatar video.
Control plane: product context, conversation state, session configuration, analytics events.
This separation matters because the media path has different failure modes and timing constraints. If your shopping assistant needs to fetch inventory or product details, do that in parallel with the conversation, not on the critical path for initial turn-taking. For example, if the user asks “What is the difference between these two headphones?”, you can start streaming a clarifying response immediately while the backend resolves product metadata.
In SvelteKit, the browser side usually benefits from a small amount of orchestration and not much else. A typical pattern is:
Initialize the page and fetch any session bootstrap data.
Start the media connection only after the user has interacted, to satisfy browser autoplay restrictions.
Keep the UI optimistic: show “listening” or “connecting” state immediately, then update as tracks become available.
Do not re-create the peer connection on every store update or component re-render.
That last point is important in SvelteKit. If your media connection lives inside a reactive component, make sure the connection object is stable and cleaned up explicitly on route changes or unmount. Reconnecting WebRTC every time a store changes is an easy way to add seconds of avoidable delay.
Reduce startup latency in the browser
The most visible delay in a shopping assistant is often the first turn. Users click “Ask,” wait, and then decide whether the system feels trustworthy. You can reduce that delay by making the browser ready before the user speaks.
Practical techniques:
Preload the assistant UI and any static assets before the user opens the modal.
Warm up the session by fetching configuration and signaling endpoints early.
Request microphone permissions proactively, but only in response to a user gesture if the browser requires it.
Keep the avatar container mounted so the first frame does not incur layout work.
Avoid expensive canvas or video processing on the main thread.
WebRTC itself can be fast, but only if ICE negotiation does not stall. If you are deploying behind restrictive networks, make sure your infrastructure is configured for TURN fallback. In practice, the difference between “works in the lab” and “works on hotel Wi-Fi” is often whether you have a robust relaying path for media.
For the avatar video specifically, the critical optimization is not “higher FPS at all costs.” It is “start rendering quickly and stay synchronized with audio.” A 15-24 FPS talking face that begins on time is better than a high-FPS face that appears late. Users are much more sensitive to startup lag than to moderate frame rate, especially in a conversational UI.
Stream the conversation, not the final answer
A shopping assistant should emit partial responses as soon as it has enough confidence to say something useful. That can be a short acknowledgment, a clarification question, or an initial recommendation. Streaming matters because it amortizes the user’s perceived wait time.
On the backend, this typically means your agent pipeline should support incremental updates from ASR, token streaming from the LLM, and chunked TTS generation. The avatar layer then consumes audio continuously rather than waiting for a complete sentence. If the audio and video are both derived from the same realtime session, lip sync becomes much easier to preserve because the avatar can key off the actual media timeline instead of trying to reconstruct timing after the fact.
There are two useful optimizations here:
Use partial ASR results to detect intent early. You do not need the full utterance before beginning a retrieval or recommendation path.
Stream TTS in chunks small enough to begin playback quickly, but large enough to avoid over-chunking overhead.
Be careful with cancellation. In a shopping assistant, users interrupt themselves constantly: “Show me the black one—actually, the blue one.” Your pipeline should support barge-in. When new speech is detected, stop generating or playing the stale response immediately, then switch the session state to the new turn. If you do not, the avatar will keep speaking outdated information while the user is already moving on.
Where Protoface fits: dropping a synchronized face into the agent loop
If your agent already runs on LiveKit, the cleanest low-latency path is to add a talking face at the agent layer instead of building a separate video stack. The livekit-plugins-protoface plugin, published on PyPI, is designed for exactly that: it lets a LiveKit voice agent produce a synchronized video avatar without forcing you to manage a second realtime system.
That matters for latency because the avatar can stay attached to the same conversation timeline as the agent audio. You are not shipping audio to one service and video to another and then trying to reconcile timing after the fact. The implementation details vary by setup, but the shape is usually straightforward: initialize the agent, attach the avatar plugin, then let the agent stream audio while the avatar follows along.
If you need to create or manage avatars and sessions programmatically, the REST API is the right surface. For example, a server-side job can create a session before the user joins, then hand the client only the session-specific connection details. Keep API keys on the server; do not expose them in the browser.
The exact payload fields are documented in the API reference, but the pattern is standard: authenticate from your backend, create the session, and feed the result into your realtime agent or browser session setup. If you are using the Python SDK for orchestration jobs, the same principle applies: keep the control plane server-side and use the SDK to manage lifecycle operations.
SvelteKit-specific gotchas that affect perceived latency
A few implementation details are easy to miss in a SvelteKit app and can make the assistant feel slower than it is:
Hydration delays: if the assistant component only becomes interactive after a full route hydration, move static shell rendering earlier and lazy-load only the media logic.
SSR mismatch: avoid rendering media-dependent UI on the server. Gate browser-only code behind
onMount.Reactive churn: isolate connection state so ordinary UI updates do not recreate tracks or peer connections.
Unnecessary re-fetching: cache catalog data and product context; do not re-query on every utterance.
Also watch the human side of the interaction. If the assistant is embedded on a product page, the user is likely already comparing items and expects fast feedback. A small delay can be acceptable if the UI makes progress visible: “Connecting…”, “Listening…”, “Looking up product details…”, then the first partial response. A blank screen is worse than a slightly delayed answer.
Conclusion
Reducing latency in a realtime shopping assistant is mostly about engineering discipline: keep the media path separate from application logic, start transport early, stream partials, support interruption, and avoid re-running expensive setup work in the browser. WebRTC gives you the low-latency media layer, but the experience only feels realtime if the rest of the stack respects the same timing constraints.
If you want to add a synchronized talking face to a LiveKit-based agent without building a custom avatar pipeline, the LiveKit plugin is the most direct integration point. If you are orchestrating sessions or avatars from your backend, use the REST API or Python SDK and keep your API keys server-side. For implementation details, integration notes, and current field names, start with the documentation at docs.protoface.com.