Introduction

If you already have a Svelte interview practice app, the hard part is usually not capturing audio. It’s making the agent feel present: a face that speaks in sync with the model’s output, updates with low latency, and fits into your existing UI without turning the app into a media project.

This post walks through the architecture for adding voice, lip-synced video, and avatar sessions to a Svelte app. By the end, you should know how to wire a front end to a realtime voice backend, where the avatar rendering sits in the pipeline, and what trade-offs matter when you move from “it works on localhost” to a usable product.

What you’re actually building

For an interview practice app, the core loop is simple:

1. The user speaks into the browser.

2. Your voice agent transcribes and reasons over that input.

3. The agent returns audio plus a synchronized facial video stream.

4. The browser renders both in a way that feels like a single conversational session.

The important design constraint is that the face and voice must be driven from the same realtime session state. If audio and video are produced independently, lip sync drifts, expressions lag, and the experience quickly feels fake. In practice, this means your app should treat the avatar as part of the agent session, not as a separate animated widget.

For a Svelte app, that usually means one of two architectures:

• The browser connects to your voice backend, which then streams both speech and avatar media back to the client.

• Your backend handles the agent session and the browser receives a composited media stream or embed that already contains the avatar.

For an interview practice product, the first option is usually the better fit if you want custom UX, transcript display, note-taking, or session controls around the avatar.

Browser and transport basics: keep the media path simple

Video avatars for conversational agents are a realtime media problem, not a normal API problem. The transport tends to be WebRTC or a similar low-latency streaming path because you need:

• low end-to-end latency for turn-taking,

• audio/video synchronization,

• jitter tolerance, and

• session-level state that persists across turns.

That also means the browser should not be responsible for synthesizing the avatar locally. The browser’s job is to join the session, render the media track(s), and send user audio/input upstream. Svelte is a good fit here because it can keep the UI reactive without forcing your media pipeline into component state.

A practical Svelte structure looks like this:

• a component for the avatar video element,

• a component for audio capture and push-to-talk or VAD state,

• a transcript stream for user/agent turns, and

• a session state store with connection status, agent state, and errors.

Do not overcomplicate the front end by trying to “animate” lips from the audio waveform in the browser. That creates a second, unsynchronized visual system that will fight the real media stream.

Implementing the Svelte side: session bootstrap and rendering

For an interview practice app, the frontend usually needs to do three things:

1. ask your backend for a session token or join payload,

2. connect the browser to the realtime media session, and

3. bind the incoming avatar video track to a video element.

The exact join flow depends on your backend, but the shape is predictable. Here’s a minimal example of the state you want to manage in Svelte:

let connected = false;

}

let connected = false;

}

let connected = false;

}

Once you have a remote video track or stream, the rendering path is standard browser media handling:

<script>

<video bind:this={videoEl} autoplay playsinline muted></video>

<script>

<video bind:this={videoEl} autoplay playsinline muted></video>

<script>

<video bind:this={videoEl} autoplay playsinline muted></video>

Two details matter here:

• Use playsinline so mobile browsers don’t force fullscreen playback.

• Keep the element muted only if required by the browser policy. If the agent’s audio is separate, the video element can still be muted while the audio track plays through the media session.

In interview practice, you’ll also want transcript updates, because users care less about perfect visual fidelity than they do about conversational feedback and response timing. A good UX shows the agent’s spoken turn, the user’s answer, and optionally a “thinking” state while the backend is generating the next response.

Voice and lip sync: what the backend must preserve

The avatar face only looks right if the model’s output, the TTS system, and the video renderer stay coordinated. Conceptually, the pipeline is:

user audio -> ASR -> LLM reasoning -> TTS -> avatar rendering -> synchronized video/audio

user audio -> ASR -> LLM reasoning -> TTS -> avatar rendering -> synchronized video/audio

user audio -> ASR -> LLM reasoning -> TTS -> avatar rendering -> synchronized video/audio

If you swap TTS providers or insert extra buffering, you can easily add hundreds of milliseconds of latency. That may be acceptable for a demo, but it becomes noticeable in a live interview simulation where turn-taking matters.

The most common failure modes are:

• Audio/video skew: video updates lag behind speech or vice versa.

• Turn overlap: the agent starts talking before the user has finished.

• Session resets: the avatar state is lost whenever the UI rerenders or a WebSocket reconnects.

• Browser autoplay issues: audio fails to start without a user gesture.

A few practical mitigations:

• keep the session object outside transient component state,

• introduce explicit “listening”, “thinking”, and “speaking” states,

• handle reconnects at the session layer instead of reinitializing the whole app, and

• test on Chrome, Safari, and mobile early if you expect interviews on phones or tablets.

For interview practice specifically, latency is not just a media concern; it changes the perceived quality of the agent. A fast, slightly less expressive avatar often feels more natural than a beautiful but delayed one.

Where Protoface fits

This is the point where Protoface is useful: it gives you the avatar and session layer without forcing you to build the rendering stack yourself. If you already have a voice agent backend, the LiveKit plugin is the most direct integration path. It drops a synchronized talking face into an existing agent so your audio response gains a video avatar with the same session timing.

If your app is already using LiveKit Agents, the mental model is straightforward: keep your agent logic where it is, add the plugin, and let the avatar follow the same turn-by-turn output that drives speech. The plugin is published on PyPI as livekit-plugins-protoface, and the implementation details are documented in the public docs and repo examples. If you want to inspect the package or quickstarts, the relevant starting points are the docs and the GitHub organization.

For teams that want to create and manage avatars programmatically, the REST API and Python SDK are the other surfaces to know. The API is authenticated with bearer API keys, and the SDK is useful for provisioning avatars or sessions from your own backend. Example request shape:

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"avt_123","voice":"default"}'

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"avt_123","voice":"default"}'

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"avt_123","voice":"default"}'

The exact request fields depend on the endpoint and the current schema, so use the docs for the authoritative parameters. The point here is that session creation stays server-side; your Svelte app should receive only the minimal join data it needs.

Why iframe embeds are sometimes the better choice

Not every interview practice app needs deep media integration. If you want to place a self-contained practice avatar on a landing page, in a prototype, or in a low-risk customer-facing flow, an iframe embed can be the better trade-off. In that model, the avatar session lives inside the iframe, the parent app doesn’t handle API keys, and the browser integration is reduced to sizing, messaging, and origin allowlisting.

That’s especially useful when the product requirement is “give users a face-to-face practice session quickly” rather than “own every packet in the media path.” You keep the browser surface small, and you avoid exposing backend credentials. For developers who need a quick path from concept to working demo, this can be the fastest way to validate the experience before investing in tighter integration.

Operational details that matter in production

Once the prototype works, the real work is making it robust:

• Instrument session start time. Measure time from button click to first agent audio and first avatar frame.

• Log disconnect reasons. Realtime sessions fail for network, browser, and authorization reasons; they should not all look like “unknown error.”

• Control quality tiers intentionally. Billing by quality tier means you should choose the level that matches the use case rather than defaulting to the highest setting everywhere.

• Keep the avatar state server-owned. The client should be able to reconnect without recreating the entire interview state machine.

If you’re building interview practice, also think about session length and rate limits. Short practice sessions with frequent retries produce a different load pattern than long-form tutoring or sales demos, so the backend should be designed around short-lived realtime sessions rather than one monolithic connection forever.

Conclusion

Adding voice, lip sync, and a video avatar to a Svelte interview practice app is mostly an integration problem: keep the browser thin, keep the media session synchronized, and avoid splitting audio, video, and conversation into separate systems. The browser should render the experience; the backend should own the session and the timing.

If you want to implement this with less infrastructure work, start with the LiveKit plugin if you already have a LiveKit agent, or use the REST API and Python SDK if you need to provision and manage sessions yourself. The docs at docs.protoface.com are the right place for the current schema, quickstarts, and integration details.

Build the smallest end-to-end path first: one button, one session, one avatar, one transcript stream. Once that loop is stable, everything else is UI polish.