Introduction

If you are building a virtual receptionist, the hard part is not getting a model to answer questions. The hard part is making the interaction feel continuous: the user speaks, the agent hears, the avatar talks back with synchronized lip motion, and the whole thing handles interruptions cleanly when the user cuts in. In practice that means wiring together audio streaming, a realtime model, video rendering, and turn-taking logic without adding enough latency to make the experience feel broken.

This post walks through the architecture and implementation details you actually need: how voice and video stay synchronized, how interruption/barge-in works in a realtime agent, what to watch for in a SvelteKit app, and where a developer platform like Protoface fits when you want a talking face without building the avatar stack yourself.

Understand the realtime shape of the problem

A virtual receptionist is usually a duplex realtime system, not a request/response chatbot. The user’s microphone audio is streamed up continuously; the agent may stream synthesized audio back immediately; and the avatar video has to track the current speaking state closely enough that the mouth movement matches what is heard. If the user interrupts, you need to stop or down-rank the current response, flush any queued audio, and switch the UI state fast enough that the interruption feels intentional rather than glitchy.

There are three separate clocks to think about:

• Audio clock: the stream of user input and agent output, usually carried over WebRTC or a low-latency media transport.

• Conversation clock: turn-taking state, which decides whether the agent is listening, thinking, speaking, or being interrupted.

• Avatar clock: the video face, which needs to stay aligned with the agent’s synthesized speech, not just the text response.

If those clocks are loosely coupled, you get the familiar failure modes: the avatar keeps “talking” after audio stops, the agent talks over the user, or the interruption arrives but the UI doesn’t visibly react until a second later.

Model the conversation as state, not as a chat log

For a receptionist, the frontend should not treat every utterance as a static message bubble. Instead, keep explicit session state in SvelteKit and update it from realtime events:

• idle: waiting for mic input or a user click to start

• listening: user speech is active and being transcribed

• thinking: the agent has received the turn and is generating a response

• speaking: audio is flowing back to the client, avatar should animate

• interrupted: user barged in; stop playback and return to listening

This matters because interruption is a state transition, not just a UI event. When the user starts talking during speaking, you typically want to:

1. detect speech onset from the mic stream or VAD signal,

2. signal the agent/runtime to cancel the current response,

3. stop any buffered audio playback on the client immediately,

4. switch the avatar to a neutral/listening pose, and

5. start transcription on the new user turn.

That cancellation path should be idempotent. In realtime systems, you will see duplicate events, late events, and out-of-order events. The frontend should tolerate “stop speaking” twice, or a response completion arriving after cancellation, without leaving the avatar in the wrong state.

A SvelteKit implementation pattern that holds up

In SvelteKit, keep the media/session plumbing in a browser-only component and keep secrets out of the client. Your backend should mint whatever session token or ephemeral credential the media layer requires; the browser should only receive a short-lived value. Then use a store or local state machine to reflect session transitions.

A minimal pattern looks like this:

// +page.svelte or a client-only component

<div data-mode={mode}>{mode}</div>

// +page.svelte or a client-only component

<div data-mode={mode}>{mode}</div>

// +page.svelte or a client-only component

<div data-mode={mode}>{mode}</div>

The important part is not the button; it is that all media events funnel through a small set of state transitions. In a real build, those events come from your realtime transport, your voice agent runtime, or both. The avatar should subscribe to mode, not to raw transcripts.

Interruptions: implement them as media cancellation, not just UI state

“Barge-in” only feels good when the speaking side actually stops emitting audio. If the frontend merely hides subtitles or changes an animation while queued audio continues to play, the user will still experience overlap. The exact mechanism depends on your agent stack, but the principle is the same:

• detect new speech early, preferably with VAD plus a short debounce;

• cancel the current generation/stream on the agent side;

• clear client-side audio buffers;

• reset the avatar from speaking animation to neutral listening.

One pragmatic detail: don’t wait for the backend cancel acknowledgment before stopping local playback. If the user is interrupting, immediate visual and audio feedback is more important than preserving a response that is about to be discarded anyway. You can reconcile the session state asynchronously after the stop signal propagates.

Also make sure the UI distinguishes between “interrupted by user” and “agent finished speaking.” Those are different transitions and often deserve different downstream behavior. For example, an interruption might preserve partial context and route to a clarification response, while a normal completion might advance the conversation script.

Where Protoface fits: synchronized talking video without building the avatar layer

The part that is easy to underestimate is the avatar itself. Lip sync, speaking state, and session management are all separate from the language model. If you already have a voice agent and want it to present as a talking face, the simplest integration point is the LiveKit Agents plugin, or, if you are building around your own session lifecycle, the REST API and Python SDK.

For a LiveKit-based agent, the plugin approach is the least invasive: your agent stays responsible for ASR, LLM, and TTS, and the avatar layer is attached as a synchronized video face. The plugin lives in the relevant GitHub repo for the Pipecat integration path, and the general setup is similar in spirit across agent frameworks: create or select an avatar, initialize the session, then bind the speaking stream to the avatar renderer.

For direct API usage, the workflow is straightforward: create an avatar/session, keep the API key server-side, and pass the browser only an ephemeral session reference. Example:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

Exact request fields depend on the API surface you choose, so use the docs for the current schema. The key implementation point is security: your API key stays on the server, and the browser only receives whatever short-lived session artifact is needed to join the interaction.

If you prefer Python, the SDK is a cleaner place to orchestrate session creation from your backend:

from protoface import ProtofaceClient

print(session.id)

from protoface import ProtofaceClient

print(session.id)

from protoface import ProtofaceClient

print(session.id)

Again, treat the snippet as illustrative: use the SDK docs for the exact method names and fields. The useful pattern is that backend session creation is explicit, auditable, and separate from the browser runtime.

Practical SvelteKit concerns you should not skip

A few implementation details matter more than people expect:

• Use a browser-only boundary for audio capture and video rendering. Anything that touches window, microphone permission, or WebRTC should not run during SSR.

• Keep credentials off the client. Use a SvelteKit route or server action to mint session state; never embed long-lived keys in the frontend bundle.

• Plan for reconnects. If the websocket/WebRTC connection drops, the UI should re-enter idle or listening after rejoining rather than pretending the old session is still valid.

• Debounce interruption detection. A single cough or keyboard noise should not cancel every agent response.

• Separate transcript from state. Don’t derive speaking/listening behavior from the latest assistant text message.

One more practical point: test with real latency. A local-only demo can hide the fact that a 250 ms delay is enough to make a barge-in feel sluggish. Simulate a bad network, speak over the agent, and verify that the avatar stops moving and audio cuts fast enough to feel responsive.

Conclusion

A good virtual receptionist is mostly a realtime systems problem. The frontend has to track state transitions correctly, the agent has to support cancellation, and the avatar layer has to stay synchronized with speech rather than with text. Once you model that explicitly, SvelteKit is a perfectly workable shell for the UI and session orchestration.

If you want to avoid building the talking-face layer yourself, use the appropriate Protoface surface for your stack and keep the integration narrow: server-side session creation, short-lived browser credentials, and a frontend state machine that handles speaking, listening, and interruption as first-class events. The docs at docs.protoface.com are the right place to verify the current API shape and integration specifics.