Introduction

Building a realtime AI banking avatar is mostly an exercise in latency management and state synchronization. You need a browser UI that can capture mic input, a transport that keeps audio/video low-latency and resilient, a voice agent that can respond incrementally, and a talking face that stays lip-synced with the audio stream instead of drifting behind it.

By the end of this post, you should understand the architecture of a SvelteKit frontend that connects to a realtime agent over WebRTC, uses WebSocket streaming for fast event updates, and presents a video avatar that stays tightly coupled to the model’s speech. I’ll also show where Protoface fits when you want the avatar layer without building it yourself.

Start with the transport, not the avatar

For a banking assistant, the avatar is the UI layer. The real system is a realtime conversation pipeline:

1. The browser sends mic audio to an agent.

2. The agent performs ASR, dialog policy, tool calls, and TTS.

3. The agent streams partial responses back as they become available.

4. The avatar renders those responses as synchronized speech and facial motion.

WebRTC is the right default for the audio/video plane because it gives you low latency, jitter buffering, NAT traversal, and a transport model designed for realtime media. WebSockets are still useful, but for control-plane traffic: session creation, partial transcription events, conversation state, typing indicators, and application-specific messages. In practice, you usually want both.

A clean split looks like this:

• WebRTC for mic audio in, agent audio out, and avatar video out.

• WebSocket for status, transcript deltas, tool results, and session metadata.

• HTTP for bootstrapping a session and obtaining short-lived credentials.

If you try to push everything through WebSockets, you will eventually hit awkward buffering, A/V sync, and reconnection problems. If you try to do everything over WebRTC, you will make debugging and application state harder than it needs to be.

SvelteKit client architecture

In SvelteKit, keep the page component thin and move the realtime logic into a client-only module or a store-backed service. The browser should own media capture and rendering, but not long-lived secrets. Your backend route should mint a session token or exchange the user’s authenticated session for a temporary agent session.

A typical flow is:

1. User opens the banking assistant page.

2. SvelteKit loads a chat shell and asks your backend to create a conversation session.

3. Your backend returns short-lived signaling data and any per-session instructions.

4. The browser connects to the agent over WebRTC and subscribes to the WebSocket event stream.

Minimal client-side state usually includes:

• connectionStatus: idle, connecting, connected, reconnecting

• transcript: final and partial messages

• speaking: whether the avatar should animate speech

• sessionId: for logs and debugging

For banking flows, this split matters because user identity and account context should stay on the server. The frontend should receive only the minimum conversation context required for the current session.

WebRTC media and lip-sync: what actually needs to stay aligned

When people say “realtime avatar,” they often mean three separate sync problems:

• Audio-to-text latency: how quickly the user’s speech is recognized.

• Text-to-audio latency: how quickly the agent starts speaking after generating text.

• Audio-to-face latency: how accurately the avatar’s mouth motion follows the spoken audio.

The third one is the one that makes or breaks the illusion. If the mouth motion is driven by text alone, it tends to look off. You want the avatar to respond to the actual TTS audio or to a tightly coupled phoneme/viseme timeline, not a guessed animation sequence.

On the browser side, a straightforward implementation is:

const pc = new RTCPeerConnection();

};

const pc = new RTCPeerConnection();

};

const pc = new RTCPeerConnection();

};

That snippet is simplified. Real implementations usually separate media tracks, negotiate codecs, and keep a dedicated video element for the avatar stream. The important part is that the avatar should be treated as a realtime media participant, not as a prerecorded animation.

Two practical gotchas:

• Autoplay policies: browsers often require a user gesture before playing audio. Design the UX so the user explicitly starts the session.

• Mic permissions and device switches: support revoking and re-granting permissions, and expose a clear “switch microphone” path.

WebSocket streaming for agent events

Even with WebRTC handling media, you still want a WebSocket event stream to drive the rest of the UI. Banking conversations benefit from precise state transitions: “identity verified,” “balance lookup in progress,” “handoff required,” or “secure message sent.” Those are better represented as events than as inferred UI behavior.

A common event model looks like this:

const ws = new WebSocket(session.eventsUrl);

};

const ws = new WebSocket(session.eventsUrl);

};

const ws = new WebSocket(session.eventsUrl);

};

There are two things worth being strict about here:

1. Event ordering. Partial transcripts, tool events, and speech-state events can arrive out of order under load. Your UI should tolerate late-arriving updates without corrupting the conversation timeline.

2. Idempotency. If the socket reconnects, you may receive repeated events. Key your state updates by event id or monotonic sequence number.

For a banking avatar, you usually also want a policy layer that can suppress or redact content before it is shown in the UI. The browser is not the right place to decide whether a message should be displayed verbatim.

How Protoface fits: avatar rendering without building the media pipeline

This is where a dedicated avatar service saves time. Instead of generating lip-sync and face animation yourself, you can drop the avatar layer into a voice agent and keep your application focused on conversation logic and banking workflows. One common integration path is the LiveKit Agents plugin, which adds a synchronized talking face to an existing voice agent.

If you are already using LiveKit-based agents, the plugin is the shortest path. The packaging on PyPI is livekit-plugins-protoface; the repo and examples are linked from the project materials, and the Pipecat integration has a dedicated guide as well. If you want to stay close to the agent runtime, that is typically the cleanest place to add the avatar.

# Illustrative only; check the docs for exact constructor arguments

# Illustrative only; check the docs for exact constructor arguments

# Illustrative only; check the docs for exact constructor arguments

For session and avatar management outside the agent runtime, the REST API is the control plane. You create or manage avatars and realtime sessions server-side, using an API key in the Authorization header. That keeps keys out of the browser, which matters a lot for a public banking UI.

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avatar_123","metadata":{"channel":"banking-support"}}'

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avatar_123","metadata":{"channel":"banking-support"}}'

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avatar_123","metadata":{"channel":"banking-support"}}'

The exact request/response shape is documented in the docs, but the pattern is what matters: server-side session creation, short-lived client access, and a browser that only receives what it needs to render the conversation.

Security and product constraints for banking

Banking apps have different constraints from a generic support bot. A few are worth calling out explicitly:

• Do not expose long-lived secrets in the browser. Use your backend to mint sessions or proxy requests.

• Separate identity from conversation. The avatar session should not be the source of truth for account access.

• Minimize PHI/PII in transcripts. Redaction should happen before storage when possible.

• Expect handoffs. The avatar should support a graceful transfer to a human agent or secure flow.

If you are embedding the avatar in a website without a backend, an iframe-based managed embed can remove a lot of risk because the API key never enters the browser context. That pattern is useful when you want the avatar on a public landing page or an authenticated portal with strict origin allowlisting and per-session limits.

Operational details that save time later

There are a few implementation details that are easy to miss until production:

• Measure end-to-end latency. Track mic capture to first token, first token to first audio, and first audio to first visible mouth movement.

• Handle tab suspension. Background tabs can affect timers and media behavior.

• Retry signaling separately from media. A dropped WebSocket should not always tear down the WebRTC peer connection.

• Log session ids everywhere. You will need them when debugging audio drift or state mismatches.

In SvelteKit, this usually means a small realtime service module plus a few stores, not a giant monolithic component. Keep the connection lifecycle explicit, and make teardown deterministic when the user leaves the page.

Conclusion

A realtime banking avatar is not hard because of the face; it is hard because of the transport, sync, and operational constraints around the face. If you design the system around WebRTC for media, WebSockets for event streaming, and server-side session management for security, you can keep the implementation understandable and the UX responsive.

If you want to skip building the avatar rendering layer yourself, start with the live-agent integrations and the API surfaces documented at docs.protoface.com. For a production SvelteKit banking assistant, that usually means: keep secrets on the server, keep media realtime, and keep the avatar as a thin presentation layer over a well-structured voice agent.