Introduction

Building a realtime fintech avatar sounds simple until you wire up the actual moving parts: a language model or voice agent that streams audio, a video face that stays synchronized to that audio, a browser frontend, and a backend that can safely create sessions and control policy. The hard part is not generating a talking head; it is keeping the media pipeline low-latency, authenticated, and operationally sane.

This post shows one practical way to do it with FastAPI as your application backend, LiveKit as the realtime transport for the voice agent, and a web experience that presents the avatar to end users. By the end, you should understand where each component belongs, how the audio/video synchronization works, and how to create and initialize avatar sessions without leaking credentials into the browser.

Start with the system boundaries

For a fintech use case, the boundaries matter more than the demo. You usually need three layers:

1. Control plane: your FastAPI app issues short-lived session metadata, enforces user/auth policy, and talks to the avatar provider’s REST API.

2. Realtime plane: LiveKit carries voice audio, agent events, and any additional media between the agent and the client.

3. Presentation plane: the browser renders the avatar video and handles user interaction.

The important design rule is that the browser should not hold long-lived secrets. If you are authenticating against an avatar service with API keys, that should happen server-side in FastAPI. The client should receive only session-scoped data or ephemeral tokens.

For fintech, this separation is also how you keep auditability and rate limiting under control. Session creation, policy decisions, and per-user limits belong behind your backend. The realtime media path should remain narrow and predictable.

How the media pipeline actually works

An interactive avatar is a synchronization problem. The voice agent produces audio in chunks; the avatar renderer has to turn that into lip-synced video frames with minimal delay. If the audio is laggy, the face feels disconnected. If the video lags too much, the user notices the mismatch immediately.

In practice, your agent stack sends text or tool-call outputs into a speech pipeline, which emits audio frames. The avatar service consumes that audio and generates a synchronized talking face. The browser receives the video stream and, depending on the implementation, may also receive the audio track directly or via the same realtime session.

Two latency sources are worth watching:

• Turn latency: time from user speech end to agent response start.

• Render latency: time from audio emission to video frame availability.

For fintech workflows like onboarding or support, keep responses short and predictable. Long “thinking” pauses are more jarring in video than in plain audio. If the model needs to call tools, acknowledge quickly and then continue after the tool returns.

FastAPI: create and authorize sessions on the server

Your FastAPI app should own all privileged actions: creating avatars, provisioning realtime sessions, and deciding which user can access which session. The exact request shape depends on the API, but the flow is straightforward: your backend calls the avatar REST API with a bearer key, receives session data, and returns only the safe parts to the browser.

import os

}

import os

}

import os

}

The point of this server-side call is not the specific field names; those can change, and the docs should be the source of truth. The point is to keep API keys out of frontend code and to gate session creation behind your own authz logic.

If your app already has user accounts, bind the avatar session to your internal user ID and store the mapping. That makes it much easier to reconcile usage, investigate incidents, and enforce per-customer limits.

Connect the browser without exposing secrets

For customer-facing fintech flows, the safest frontend pattern is an iframe embed backed by a server-created session. The browser gets an embed URL, not your API key. That lets you put the avatar in a dashboard, support portal, or onboarding page without turning the page into a secret store.

This model matters if you allow per-embed configuration such as voice selection or custom instructions. Those settings should be attached when you create the session, not assembled in JavaScript from sensitive values. Keep any parent-origin allowlist strict, especially if the avatar appears inside an authenticated app.

<iframe
></iframe>

<iframe
></iframe>

<iframe
></iframe>

If you need to integrate the avatar into an existing app shell, make sure the iframe size, autoplay policy, and CSP headers are tested early. Most “it works locally but not in production” issues in realtime media come from browser permission rules rather than the avatar stack itself.

LiveKit: attach a synchronized face to the voice agent

When the avatar is part of an agent workflow, LiveKit is a good place to anchor the media graph. The agent handles the conversation; the avatar is just another rendering target for the agent’s speech. The quickstart examples are useful if you want to see end-to-end session wiring, but the main idea is simple: initialize the avatar plugin in the same runtime as your voice agent so spoken output can feed the video face with minimal delay.

from livekit.agents import AgentSession

await session.start()

from livekit.agents import AgentSession

await session.start()

from livekit.agents import AgentSession

await session.start()

The plugin approach is especially useful if you already run a LiveKit-based voice agent and want to add a face without redesigning the stack. You keep the conversational logic where it is, and you let the avatar layer subscribe to the same speech events.

Operationally, this is cleaner than bolting a separate browser-side video generator onto your agent. The synchronization boundary stays inside the agent runtime instead of being split across browser JavaScript, a backend, and a separate media pipeline.

Practical fintech concerns: policy, reliability, and limits

In financial workflows, you should assume that the avatar is part of a regulated user interaction, not a novelty widget. That means you need explicit controls around usage, content, and access.

• Authentication: never ship API keys or long-lived service credentials to the browser.

• Session scope: make sessions short-lived and tied to a single authenticated user or support case.

• Rate limiting: protect against abuse and unexpected usage spikes.

• Timeouts and fallbacks: if the avatar session fails, degrade gracefully to audio-only or text.

Reliability-wise, keep your FastAPI handler idempotent where possible. Session creation should be safe to retry, and your frontend should be able to recover if the iframe or LiveKit session reconnects. Also, treat avatar quality as a product choice: higher quality is usually more realistic, but it also changes bandwidth and cost. That trade-off is real when you are serving lots of customers.

Where Protoface fits cleanly

This is the part that removes most of the glue code. Protoface gives you a REST API for server-side session management, a Python SDK for programmatic control, and a LiveKit plugin for dropping a synchronized talking face into an existing voice agent. For a fintech app, that means FastAPI can own auth and policy while the agent runtime focuses on conversation, not avatar rendering.

For backend automation, the REST API is the right surface. For application code that provisions avatars or sessions, the Python SDK is the nicer fit. For an existing LiveKit deployment, the plugin keeps the integration close to the media graph instead of spreading it across your web app. If you want exact request fields, session options, and current integration details, start with the docs.

Useful references: docs and the plugin examples in the relevant GitHub repository.

Conclusion

The reliable way to connect FastAPI, LiveKit, and a realtime avatar is to keep each layer narrow: FastAPI handles identity and session creation, LiveKit handles realtime conversation transport, and the browser renders only the session it was granted. That gives you a clean security boundary and a media path that can stay low-latency under load.

If you are implementing this now, start by standing up the server-side session creation flow, then add the LiveKit agent integration, and only then wire the frontend iframe or media UI. Keep credentials server-side, keep sessions short-lived, and test reconnect behavior early. The docs at docs.protoface.com are the right place to confirm the exact API shapes and current quickstarts.