Guide to Building a Conversational Product Recommender Avatar with SvelteKit and LiveKit
Build a SvelteKit + LiveKit conversational product recommender avatar with realtime sessions, agent state, and lip-sync.
Introduction
If you want a product recommender that feels usable in a live session, a static chat widget is usually not enough. For discovery flows, users benefit from a conversational interface that can ask clarifying questions, maintain context, and show a face that matches the pacing of the conversation. The hard part is not “making it talk”; it is keeping audio, dialogue state, and visual feedback aligned enough that the experience feels coherent rather than bolted together.
This post shows a practical way to build a conversational product recommender avatar with SvelteKit on the frontend and LiveKit for realtime media transport. By the end, you should understand the architecture, the timing constraints, where the avatar fits into the agent loop, and how to wire it up without exposing secrets in the browser.
Architecture: separate the conversational loop from the visual layer
Think of the system as three independent pieces:
Frontend: a SvelteKit app that captures user input, renders UI state, and subscribes to realtime media.
Agent: your recommendation logic, which can be an LLM-driven voice agent or a scripted policy with retrieval over your catalog.
Avatar video: a synchronized talking face that mirrors the agent’s speech stream and mouth timing.
The key design choice is that the avatar should not be the source of truth for conversation state. It is a presentation layer. Your agent owns the turn-taking logic, prompt state, product ranking, and any commerce rules. The avatar simply renders the output with lip sync and expressions, ideally from the same speech stream that the user hears.
In practice, that usually means:
User speaks or types a question.
The agent extracts intent and constraints, such as budget, category, constraints, and preferences.
The agent queries your product index or recommendation service.
The agent speaks back a concise recommendation, while the avatar displays the synchronized face.
The frontend updates product cards, links, or comparison tables in parallel with the spoken response.
That split matters because speech synthesis, video rendering, and LLM token generation all have different latency profiles. If you couple them too tightly, the UI becomes fragile. If you keep the boundaries clean, you can swap models, tune prompts, or change avatar providers without rewriting the app shell.
SvelteKit frontend: keep session handling on the server, media on the client
With SvelteKit, the usual pattern is to create or mint any session credentials on the server side, then hand the browser only what it needs to join a room or connect to a pre-authenticated embed. Do not put API keys in client code. For a browser-based voice agent, the browser should only know about ephemeral session metadata, not your long-lived service credentials.
A straightforward UI might include:
a text input for the user query,
a product suggestion panel that updates as the agent speaks,
a video container for the avatar,
optional audio controls and connection status.
For a custom avatar integration, you can render the video surface directly in your app and connect it to the realtime session via the mechanism your backend establishes. The main thing to get right is lifecycle management: create the session once, reuse it while the conversation is active, and tear it down cleanly when the user leaves or times out.
In SvelteKit, that usually means server-side endpoints for session creation and a client component that subscribes to updates. Keep the transport concerns isolated from your recommendation logic so reconnects do not reset conversation state.
Recommendation logic: make the agent explicit about constraints and ranking
A product recommender is easiest to debug when the agent produces structured intermediate state, even if the end-user only sees natural language. For example, the agent can maintain a compact object with category, budget, must-have features, and excluded items. That object is then used to rank candidate products and produce the spoken response.
Two implementation details matter:
Normalize the query before retrieval. If a user says “something quiet for a small apartment,” convert that into explicit constraints such as noise level, footprint, and room size.
Separate explanation from ranking. Rank products using your actual catalog data or scoring model, then have the agent explain the result in plain language. Do not let the model improvise product facts you have not grounded.
For a voice-first flow, keep responses short and incremental. Realtime agents perform better when they can start speaking as soon as the first useful recommendation is available, rather than waiting for a verbose final summary. That also reduces the mismatch between audio and UI updates.
Transport and timing: why WebRTC-style realtime matters
Realtime avatars are sensitive to timing because the user notices the gap between spoken audio, lip motion, and turn-taking more than they notice small textual errors. In a voice agent, audio usually moves over a low-latency media transport, while model inference and product lookup happen concurrently. The avatar should follow the same speech timeline as the audio, not an independently generated animation.
A few practical rules:
Start with a low-latency path. If the agent is slow to first token or first audio frame, the avatar will feel unresponsive even if the final answer is good.
Stream partial results. If your agent can emit a short acknowledgement, followed by the actual recommendation, the UI feels much more alive.
Avoid double buffering. If you buffer text, then synthesize audio, then buffer again before video playback, latency compounds quickly.
Handle interruptions explicitly. If the user speaks over the agent, stop or truncate the current turn and reset the avatar state cleanly.
Also be careful with the recommendation UX itself. Users often ask follow-up questions mid-answer. A responsive avatar should tolerate mid-turn interrupts without losing state. That means the agent, not the UI animation, decides when a turn is committed.
Where Protoface fits: avatar rendering as a drop-in layer for your agent
This is the part that Protoface is meant to simplify. If you already have a LiveKit-based voice agent, the LiveKit Agents plugin can attach a synchronized talking face to that agent without changing your conversational logic. The plugin lives in the Python ecosystem and is the most direct path when your backend already uses LiveKit Agents for voice.
For a recommender, that means you keep your catalog retrieval, prompt logic, and business rules in your agent, then add a visual face as a presentation layer. The avatar tracks the agent’s speech, so you are not hand-rolling lip sync or video session management.
For example, the shape of the integration looks like this:
If you are wiring this into Python service code, the practical next step is to read the integration notes in the package docs and the examples in the relevant repo. The important part is that the avatar is attached to the agent’s speech output, not driven as a separate animation system.
If you prefer to inspect session primitives directly, the REST API also exposes avatar and realtime session management. A minimal request would look like this:
That example is intentionally generic: the exact fields depend on the session type and avatar configuration described in the docs. The useful part is the shape of the workflow. Create or reuse a session server-side, hand the browser only a safe session handle, and let the client connect to the media stream.
Operational details: security, rate limits, and failure modes
When you ship this pattern to real users, the non-obvious issues are usually operational:
Never expose service keys in the browser. Keep API-key-authenticated calls on the server. The browser should talk to your backend or use a customer-managed embed that does not require a backend.
Plan for reconnects. Mobile networks and tab suspends happen. Your session manager should recover gracefully and avoid creating duplicate active sessions.
Budget for rate limits and session duration. Interactive avatar flows are easy to leave open. Enforce sensible session timeouts and cleanup.
Surface partial failure. If product retrieval fails, the agent should say so and fall back to a narrower answer rather than stalling the media session.
One useful design pattern is to keep the recommendation engine deterministic enough that you can replay the same query and inspect why a given product was chosen. That makes it much easier to debug what the avatar said versus what the actual ranking logic produced.
If you are using a hosted embed flow for a website, the same security principle applies: the client should never see your API key, and embed access should be constrained by origin allowlists and session limits rather than by frontend secrecy.
Putting it together in SvelteKit
A solid implementation usually ends up with three server routes and one client component:
POST /api/session creates or fetches a realtime session.
POST /api/recommend runs your catalog logic and returns structured recommendations.
POST /api/turn coordinates agent state, if you want text-driven control from the frontend.
<AvatarPanel /> renders the live video surface and connection status.
That separation keeps the app maintainable. The API routes own secrets and data access. The component owns rendering. The agent owns the dialogue. If the avatar vendor changes, only the session layer changes. If your ranking model changes, the recommendation route changes. If you add products or new filters, the frontend stays mostly the same.
For a concrete starting point, I would recommend building the recommendation logic first with text-only responses, then adding the avatar once the turn-taking and ranking are stable. That order avoids debugging three moving parts at once.
Conclusion
A conversational product recommender works best when you treat the avatar as a realtime presentation layer for a well-defined agent, not as the product logic itself. SvelteKit gives you a clean server/client split, LiveKit gives you the low-latency media path, and the avatar layer makes the experience feel much more human without changing your core recommendation stack.
If you want to implement this pattern with a managed avatar layer, start with the docs at docs.protoface.com, then use the relevant integration repo or quickstart that matches your agent stack. For Python/LiveKit backends, the LiveKit plugin path is the most direct place to begin.