Introduction

If you already have a static video widget in your SvelteKit app, you probably know its limitations: it looks polished, but it does not respond to live conversation state. Once you want a voice agent, the hard part is no longer rendering video; it is keeping speech, animation, and network state synchronized without turning your app into a timing bug farm.

This guide shows how to migrate a static widget to a realtime AI avatar. By the end, you should understand the architecture changes involved, how to integrate the avatar into a SvelteKit app without leaking secrets to the browser, and how to choose the right integration path for your deployment model.

What changes when the video becomes realtime

A static widget is usually just media playback: an MP4, a looped animation, or a prerecorded talking-head clip. Your frontend owns rendering, and the backend mostly serves assets. A realtime avatar is different. It is driven by live audio or agent state, so the video face must stay synchronized with speech generation, turn-taking, and session lifecycle.

In practice, that means three systems need to cooperate:

• Conversation or voice agent runtime produces audio and turn events.

• Avatar session layer converts those events into a synchronized talking face.

• Web client renders the video stream and maintains the connection state.

The important technical point is that the avatar is not “an animation on top of audio.” It is a realtime media participant. That usually implies WebRTC or another low-latency streaming path, plus a server-side session token or API-mediated setup so the browser never has to hold long-lived credentials.

Migration pattern: keep SvelteKit thin, move session orchestration server-side

The cleanest migration is to treat the avatar as an external realtime service and keep the browser focused on display and user input. In SvelteKit, that usually means:

1. Create or fetch the avatar session in a server route or server action.

2. Return only the minimal client-facing connection data needed by the browser.

3. Render the remote video stream in a component that can react to session state.

4. Keep API keys and any privileged session management on the server.

This separation matters because a static widget often gets “upgraded” by stuffing an API key into client code. That works until it does not. If the service exposes session creation, avatar selection, or custom instructions behind an API key, those operations belong in a server endpoint.

Example SvelteKit server route for session setup

The exact request schema depends on your Protoface configuration and the docs, but the pattern is straightforward: your SvelteKit route calls the REST API from the server, then hands the client a short-lived session payload.

// src/routes/api/avatar-session/+server.ts

}

// src/routes/api/avatar-session/+server.ts

}

// src/routes/api/avatar-session/+server.ts

}

Two practical notes:

• Use environment variables for API keys, never SvelteKit client env.

• Keep the response minimal. If the browser only needs a short-lived token and a session ID, do not proxy more than that.

Rendering the avatar in the client

On the frontend, the goal is simple: mount the remote video stream and reflect connection state. Whether you render with a custom video element, an embedded frame, or a media SDK depends on the surface you choose. For a custom Svelte component, the implementation usually looks like standard realtime media plumbing: connect, attach tracks, detach on teardown, and watch for reconnects.

Even if the visuals are “just video,” the client component should still handle the edge cases that static widgets never see:

• Lifecycle: connect on mount, disconnect on destroy.

• Visibility: pause or downscale when the tab is hidden if your UX allows it.

• Fallbacks: show a neutral placeholder when the stream is not ready.

• Retry behavior: distinguish temporary network loss from hard session failure.

If you are embedding a third-party video surface, keep CSS constraints explicit. Realtime video widgets often fail for boring reasons: parent containers with zero height, object-fit mismatches, or layout shifts during reconnects. For a chat panel or support widget, reserve space up front and avoid letting the avatar resize unpredictably during speech turns.

Voice agent synchronization: the part that usually breaks first

The migration only feels successful when the avatar and the voice agent behave like one system. That means your audio pipeline must be coordinated with avatar session state. In a typical turn-based flow, the agent emits audio chunks, the avatar session receives them, and the client receives a synchronized video stream. If you short-circuit any of those steps, you get lip movement that is slightly ahead, behind, or completely disconnected from the current utterance.

There are a few implementation details worth watching:

• Latency budget: every hop adds delay. Keep transcription, LLM inference, TTS, and avatar rendering on a tight path.

• Turn boundaries: make sure the agent knows when the user has finished speaking before the avatar starts talking.

• State resets: if the agent interrupts itself or the user barges in, the avatar session should be able to stop and retarget quickly.

• Audio consistency: mismatched sample rates or buffering behavior can create visible lip-sync drift.

For developers already using LiveKit, this is where the integration feels natural: the voice agent can remain in LiveKit, while the avatar becomes a synchronized visual participant. The relevant plugin is the Protoface Pipecat integration for Pipecat users, and the LiveKit plugin is exposed as livekit-plugins-protoface. In both cases, the useful mental model is the same: the agent owns the conversation, and the avatar is attached as a realtime video surface that follows that conversation.

Using the REST API from a backend service

If your migration is driven by a backend service instead of SvelteKit alone, the REST API is the most direct control plane. You can create avatars, provision sessions, and manage runtime state without coupling UI code to session internals. That is useful when you need per-user configuration, usage tracking, or server-side authorization.

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

Use the docs for the exact request and response shape; the point here is the control flow. The browser should never send this request with a secret key. If you need a browser-only option, prefer a customer-managed iframe embed that keeps the key out of the client entirely.

When an iframe is the better migration path

If your current widget is mostly a drop-in UI element and you do not need deep frontend control, an iframe can be the fastest safe migration. It is especially useful when you want an interactive avatar on a marketing site or support page without standing up backend session logic in SvelteKit.

The trade-off is control versus isolation. An iframe reduces integration work and keeps the credential boundary clean, but you give up some direct access to the media pipeline and custom UI composition. That is often a good trade if your main requirement is to ship a working conversational avatar without exposing API keys or building a custom realtime stack.

In other words: if you need tight integration with your app state, use server-side session orchestration and a custom component. If you need a secure, low-friction embed, use the iframe path and let the embed handle session policy, parent-origin allowlisting, and per-embed limits.

Practical migration checklist

• Move any session creation or avatar management logic out of the browser.

• Replace the static video asset with a realtime render surface.

• Verify you can start, stop, and reconnect sessions cleanly.

• Test low-bandwidth and tab-switching behavior, not just the happy path.

• Watch for token leakage, especially in logs and client bundles.

• Measure end-to-end latency from user speech to avatar response.

If the old widget was “one component,” the new one is a small distributed system. That is normal. The trick is to keep the browser simple and make the server own the sensitive and stateful parts.

How Protoface fits into this

This is exactly the boundary Protoface is designed around: a developer-facing realtime avatar layer with a REST API, Python SDK, and integration surfaces for voice-agent stacks. For a SvelteKit migration, the practical advantage is that you can keep secrets server-side, create sessions from your app backend, and attach a synchronized talking face to a realtime conversation without inventing your own media control plane. The docs at docs.protoface.com have the concrete request fields and integration details.

Conclusion

Replacing a static video widget with a realtime avatar is less about visual polish and more about moving from asset delivery to session orchestration. In SvelteKit, the safest pattern is to keep the browser lightweight, create sessions server-side, and let the client focus on rendering and lifecycle handling. Once you do that, the remaining work is mostly about latency, turn-taking, and clean reconnect behavior.

If you want to implement this incrementally, start with a server route that creates a session, then swap your static widget for a realtime client component or embed. From there, validate the voice-agent sync and harden your reconnect logic. For concrete API shapes and quickstarts, go to the documentation and the relevant examples in the GitHub repos.