Introduction

Building a voice-and-video e-commerce agent is mostly an exercise in getting three real-time systems to behave like one: a conversational model, a low-latency audio pipeline, and a video surface that updates in sync with the agent’s speech. The tricky part is not generating a response; it’s keeping turn-taking, state, and transport aligned so the customer experiences a single coherent “person” instead of a stack of disconnected APIs.

This post shows one practical way to wire that up in SvelteKit with a FastAPI backend and WebSocket streaming. By the end, you should have a clear mental model for:

• how to split responsibilities between browser, backend, and AI services,

• how to stream partial responses over WebSockets without blocking the UI,

• how to keep voice playback and avatar animation synchronized, and

• where Protoface fits when you want the agent to have a realtime talking face.

System shape: keep the browser thin, the backend authoritative

For this kind of agent, I recommend a simple architecture:

• SvelteKit handles the storefront UI, capture of user intent, and rendering of the session.

• FastAPI owns session state, orchestration, and credentials to external services.

• WebSockets carry incremental events: transcript deltas, tool calls, inventory lookups, assistant text, and readiness signals for audio/video playback.

The browser should not decide business logic. It can submit user input, display the conversation, and play media, but the backend should be the source of truth for product availability, pricing, order status, and any privileged API calls. That matters in e-commerce because the moment the agent starts quoting prices or suggesting substitutions, you need deterministic access to backend state.

The realtime contract is usually event-based. A single user turn might produce:

1. partial ASR transcript events,

2. an intent classification or tool invocation,

3. streamed assistant text,

4. audio chunks or TTS-ready text, and

5. avatar state updates such as speaking, idle, or listening.

Do not try to “finish” the response before sending anything to the client. Latency is felt more as dead air than as imperfect partials. Early event emission is what makes the interaction feel live.

FastAPI orchestration with WebSocket streaming

FastAPI is a good fit for the session coordinator because it gives you a clean path for both REST setup and WebSocket streaming. A typical pattern is:

1. browser opens a websocket for a session,

2. backend sends initial state and auth context,

3. user messages are appended to a server-side transcript,

4. assistant output is streamed back as incremental events.

Keep the message schema explicit. Even if the model provider changes, your browser should only care about stable event types.

from fastapi import FastAPI, WebSocket

from fastapi import FastAPI, WebSocket

from fastapi import FastAPI, WebSocket

A few implementation details are worth getting right:

• Backpressure: buffer carefully. If the client is slow, don’t let a burst of deltas overwhelm the socket.

• Session affinity: keep a single source of truth for the conversation state, especially if you scale horizontally.

• Idempotency: websocket reconnects happen. Make sure the client can recover from a dropped connection without duplicating side effects.

SvelteKit client: stream into state, not into DOM hacks

On the client, the main job is to open the socket, append deltas, and keep the view reactive. In Svelte, that’s straightforward if you model the conversation as state and update it incrementally.

<script lang="ts">

</script>

<script lang="ts">

</script>

<script lang="ts">

</script>

For voice UX, the browser should not guess when audio is “done” based on text completion alone. If you are synthesizing speech, use a dedicated completion event from the backend. That allows you to keep the avatar speaking state aligned with audio playback rather than with raw tokens.

Also, treat microphone capture and playback as separate concerns. If the customer can interrupt the agent, you need barge-in behavior: stop audio playback, mark the assistant turn as interrupted, and start listening again. This is one of the places where realtime agents feel much better than request/response chat.

Synchronization: text, audio, and face need the same turn model

The hardest bugs are usually not in the model; they are in turn coordination. A voice agent with a face has at least three clocks:

• the text stream the user can read,

• the audio stream the user hears, and

• the avatar animation state that should match the audio.

In practice, define one authoritative assistant turn object on the server and derive all downstream media from it. For example:

• assistant text deltas accumulate into a canonical response,

• TTS starts only after a partial or complete response is available, depending on your provider,

• the avatar enters “speaking” when audio begins and leaves that state when playback ends.

Two common mistakes:

1. Starting the face too early: if the avatar animates before audio actually plays, the illusion breaks immediately.

2. Coupling the face to token streaming: tokens are not speech. If the model emits a long pause mid-sentence, the avatar should not flap around waiting for text.

For e-commerce, you also want tool results to be first-class events. If the agent asks the inventory system whether a size is in stock, stream that status separately from the final prose so the UI can surface it immediately.

Where Protoface fits: give the agent a synchronized talking face

This is the part where Protoface is useful: it lets you attach a realtime avatar surface to the voice agent without having to build lip-sync and face streaming yourself. The integration is intentionally developer-facing and comes in a few surfaces; for a FastAPI + SvelteKit stack, the important one is the REST API for creating and managing avatars and realtime sessions, plus a Python SDK if you want to orchestrate sessions server-side.

A minimal session creation flow from Python looks like this in shape:

from protoface import Client

from protoface import Client

from protoface import Client

If you are already using LiveKit Agents, there is also a plugin path that drops a synchronized avatar into the voice agent so the video face tracks the agent’s speech. That is useful when your existing stack is already centered on LiveKit and you want to avoid building a separate media pipeline.

For the browser, Protoface also supports customer-managed iframe embeds. That is a different trade-off: you get a self-contained avatar surface with no backend and no API key exposed in the client, which is attractive for marketing pages or low-friction demos. The iframe model uses parent-origin allowlisting and per-embed limits, so it is a good fit when you want a controlled surface rather than full custom orchestration.

If you need the exact request fields, session shapes, and auth headers, the authoritative reference is the documentation. For plugin examples, the relevant repository is GitHub, and the quickstart collection is useful when you want to compare integration patterns across stacks.

A concrete REST example for creating a session looks like this at the HTTP level:

curl -X POST https://api.protoface.com/sessions \

curl -X POST https://api.protoface.com/sessions \

curl -X POST https://api.protoface.com/sessions \

The precise payload varies by API version, but the shape is the same: create an avatar, create a realtime session, then attach that session to your agent runtime. Billing is by quality tier, so you should choose the lowest tier that meets the product requirement and validate latency and fidelity before you roll it out broadly.

Implementation notes that save time later

There are a few operational details that matter once this is in production:

• Keep secrets server-side. Your API key should live in FastAPI, not in SvelteKit client code.

• Instrument turn latency. Measure time from user speech end to first assistant audio and to first visible avatar motion.

• Handle disconnects deliberately. Reconnect should restore session state without replaying side effects.

• Separate conversational memory from ephemeral UI state. The former belongs in durable storage; the latter can stay in the browser.

For e-commerce specifically, avoid making the agent sound confident when the inventory or pricing service is uncertain. Emit a clear “checking” state, and let the backend decide whether the answer is authoritative enough to present.

Conclusion

The practical path to a voice + video e-commerce agent is not to weld everything into one monolith; it is to keep FastAPI authoritative, stream events over WebSockets, and let SvelteKit render a responsive client that reacts to those events. Once the turn model is stable, adding a synchronized avatar becomes a transport problem rather than a research project.

If you want to shortcut the avatar side of the stack, use Protoface for the realtime face layer and keep your time focused on orchestration, product logic, and UX. Start with the docs at docs.protoface.com, then wire up the surface that matches your architecture: REST API for custom orchestration, the LiveKit plugin for voice-agent stacks, or an iframe embed when you want the fastest path to a controlled browser experience.