Building a SvelteKit Virtual Receptionist with FastAPI, STT, and Realtime Avatar Responses
Build a SvelteKit virtual receptionist with FastAPI, streaming STT, TTS, and realtime avatar sync for low-latency voice UX.
Introduction
If you want to build a virtual receptionist, the hard part is not “making an LLM talk.” It’s coordinating a low-latency audio pipeline, speech-to-text, turn-taking, and a video avatar that can stay synchronized with the voice output without feeling mechanically detached. In practice, that means stitching together microphone capture, streaming STT, an agent loop, TTS, and an avatar layer that can render a talking face in real time.
This post walks through a practical architecture for a receptionist-style SvelteKit app backed by FastAPI. By the end, you should have a clear mental model for the data flow, the WebRTC/streaming constraints that matter, and where a realtime avatar service fits without contaminating your frontend with API keys or bespoke media code.
Architecture: split the problem into UI, agent, and media
The cleanest way to think about a virtual receptionist is as three cooperating systems:
Frontend in SvelteKit: captures mic input, plays audio output, and renders the avatar surface.
Backend in FastAPI: owns auth, conversation state, session lifecycle, and integration with STT/LLM/TTS services.
Realtime media layer: handles streaming transport and keeps audio/video synchronized closely enough that lip motion tracks the spoken output.
That separation matters because the frontend should stay thin. If you embed provider credentials in the browser or try to do all orchestration client-side, you end up with brittle security and awkward debugging. The backend should create sessions, issue any short-lived tokens, and decide when the agent is “speaking,” “listening,” or “idle.” The browser should just connect and render.
Why realtime receptionists are hard
Most voice apps fail for one of four reasons:
Latency creep: ASR, LLM, and TTS each add delay. If each step waits for the previous one to finish, the interaction feels sluggish.
Turn-taking bugs: users interrupt themselves, pause mid-sentence, or talk over the agent. Your state machine needs to tolerate that.
A/V desync: if the avatar updates on a separate clock from the synthesized speech, the mouth animation looks off immediately.
Client complexity: microphones, permissions, stream reconnection, and avatar rendering can become a pile of one-off browser code.
The right approach is usually streaming everywhere. Stream audio into STT incrementally, stream partial transcripts to the backend, stream generated tokens into TTS if your provider supports it, and keep the avatar tied to the same utterance lifecycle as the audio. You want the user to perceive one coherent turn, not four disconnected subsystems.
FastAPI session orchestration
FastAPI is a good fit for the backend because the responsibilities are straightforward: create a room or session, record metadata, hand the browser a short-lived connection artifact, and broker any calls to upstream AI services. You do not need a lot of framework magic here; you need predictable state transitions and a small number of endpoints.
A typical flow looks like this:
The browser loads the receptionist page and asks your backend for a session.
FastAPI creates a conversation/session record and returns the data needed to connect.
The browser joins the media session and starts streaming mic audio.
Backend receives transcripts, decides when to respond, and triggers TTS plus avatar playback.
Here is a minimal sketch of the shape, not a drop-in implementation:
Keep the backend stateless where possible. Persist only what you need to resume or audit a conversation: session identifiers, timestamps, transcript fragments, and enough metadata to reproduce problems later. The audio stream itself should stay off your database.
Streaming STT and turn detection
For a receptionist, speech-to-text is not just transcription; it is the trigger for the rest of the interaction. The backend needs partial transcripts quickly so it can decide whether the user is greeting the system, asking for a department, or interrupting the agent.
In practice, a good STT pipeline should expose:
Partial results for low-latency intent detection.
Final results for stable context updates.
Speech boundaries or VAD signals when available, so the system can determine when to answer.
Your agent loop should treat partial transcript events as hints and final transcript events as durable state. A common failure mode is responding to the first partial phrase too aggressively. Receptionists need a bit of patience: brief pauses are not necessarily turn ends. If your STT provider emits punctuation or endpointing signals, use them, but avoid hard-coding a single timeout as your sole turn detector.
A practical pattern is:
That last arrow is important: the avatar should follow the same utterance boundary as the voice, not a separate “talking” boolean that flips independently.
Frontend implementation in SvelteKit
On the frontend, keep the SvelteKit page focused on three jobs: request a session, connect the voice transport, and render the avatar surface. You do not want business logic spread across UI components.
For microphone capture, modern browsers will force a user gesture before recording. That means your UI should clearly expose a “Start” button and handle permission denial gracefully. Once connected, the client should stream audio continuously rather than buffering a large clip and uploading it all at once. Buffered uploads make latency worse and complicate barge-in behavior.
For the avatar view, use an iframe or a dedicated component that you can size explicitly. Realtime video faces need predictable layout, especially if you want the speech bubble, status indicators, and transcript region to remain stable while the avatar is rendering and updating.
A useful client-side guideline is to keep these states distinct:
Connecting: session created, media not yet established.
Listening: mic stream active, awaiting user speech.
Thinking: transcript received, response in progress.
Speaking: TTS and avatar are playing.
Recovering: reconnecting after an interruption or network loss.
That explicit state model makes debugging much easier than inferring everything from a single websocket flag.
Where Protoface fits
This is where Protoface is useful: it gives you the avatar/session side of the problem without making you build and synchronize the talking face yourself. For a FastAPI-backed receptionist, the usual pattern is to create or manage a realtime avatar session server-side, then let the frontend connect to that session as part of the broader voice workflow. The API surface is intentionally developer-oriented, with REST access for sessions and avatars, plus a Python SDK if you want to keep orchestration in backend code.
Example REST usage is straightforward; the exact request fields depend on the endpoint, so treat this as shape-only:
If you are already building the agent in Python, the SDK is the natural place to keep session creation and cleanup logic. The docs cover the concrete method names and payloads, so don’t guess at fields in production code; wire it up from the reference instead.
Operational details that matter in production
Once this is running, the production problems are usually around reliability rather than model quality:
Reconnect behavior: network blips should not drop the whole session if you can resume cleanly.
Rate limiting: if you expose the receptionist publicly, protect session creation and abuse-prone endpoints.
Observability: log timestamps for mic start, first transcript, response start, and response end so you can measure latency at each stage.
Fallbacks: if the avatar fails to load, the voice agent should still function, and the UI should degrade cleanly.
Also keep an eye on your media quality tier costs. Real-time avatar systems are sensitive to frame rate, synthesis quality, and runtime duration. If the receptionist is mostly used for short interactions, optimize for fast startup and predictable billing, not just maximum fidelity.
Implementation notes and gotchas
A few things are worth calling out explicitly:
Do not expose long-lived API keys in the browser. Keep session creation behind FastAPI.
Do not make the avatar animation independent from the voice event stream. One utterance should drive both.
Do not rely on one-shot uploads for “real-time” conversation. Streaming is the point.
Do not collapse partial and final transcripts into the same state. You will lose useful turn-taking signals.
If you want to add this to an existing agent stack, the LiveKit plugin path is also worth knowing about, especially if your current voice agent already lives in LiveKit. The plugin documented in the project repo lets the agent gain a synchronized talking video face without redesigning the whole application around a new media primitive. See the quickstart examples and plugin repo if your stack already speaks LiveKit.
For developers who prefer a more Python-native agent stack, the SDK and docs are the best starting points. The important decision is not “which model?” but “where does the session lifecycle live?” Once that is clear, the rest of the integration is mostly straightforward glue.
Conclusion
A usable virtual receptionist is mostly a systems integration problem: low-latency STT, a predictable backend state machine, streaming speech synthesis, and an avatar that stays synchronized with the spoken turn. SvelteKit is a good frontend for keeping the UI clean, FastAPI is a good backend for session orchestration, and a dedicated realtime avatar layer removes a lot of media-specific complexity you would otherwise have to own.
If you’re implementing this now, start with the session lifecycle and a single end-to-end turn: mic input, transcript, response, and avatar playback. Then add barge-in, reconnects, and observability. The reference docs at docs.protoface.com cover the concrete API shapes and integration options, and the GitHub examples linked from the quickstart repo are a good way to compare implementation patterns before you commit to one.