Introduction

If you want an AI agent to feel present in a product, a text bubble is not enough. A realtime talking avatar has to do three things at once: stream video at low latency, keep lip movement synchronized with speech, and stay responsive to new user input without falling apart under network jitter or state drift.

This post shows how to build that kind of avatar-backed experience in Python with FastAPI and WebSockets. By the end, you should understand the architecture, the event flow, and the practical trade-offs involved in moving from a local prototype to a production-shaped service.

We’ll focus on the backend mechanics: how FastAPI can accept websocket connections from a browser, how to bridge those events to a realtime avatar provider, and how to keep the session state coherent enough for interactive use. Where it helps, I’ll show how Protoface fits into that flow without hand-waving over the details.

What “realtime talking avatar” actually means

At a systems level, a talking avatar is a media pipeline with at least four moving parts:

• Input transport: user messages, voice activity, or control events arrive over HTTP or WebSockets.

• Agent orchestration: your backend turns those inputs into a response plan, usually by calling an LLM, TTS service, or speech pipeline.

• Video generation: an avatar renderer produces a face video stream whose mouth motion tracks the spoken audio.

• Media delivery: the stream reaches the browser with low enough latency that turn-taking still feels conversational.

The critical point is that “realtime” is mostly about latency budget and synchronization. If the text arrives quickly but the avatar starts speaking too late, or the video stream lags the audio by several hundred milliseconds, the experience feels broken. In practice, your backend needs to minimize coordination overhead and avoid treating the avatar as a batch job.

For a developer-facing app, FastAPI is a good control plane because it gives you async request handling, websocket support, and a clean way to expose session lifecycle endpoints. WebSockets are useful because the browser can keep a single bidirectional connection open while the conversation evolves, which is much easier than polling when you need to stream state changes, partial transcripts, or control messages.

FastAPI and WebSockets: the minimal backend shape

The simplest useful backend has two responsibilities:

1. Accept a websocket connection from the client.

2. Relay session events between the client and your avatar/video service.

In a real implementation, you would usually also keep some session record in memory or in a database, but the transport layer is the part that matters first.

Here is a stripped-down FastAPI websocket endpoint that illustrates the shape of the code. It does not depend on a specific avatar provider yet; it just shows the orchestration pattern.

from fastapi import FastAPI, WebSocket, WebSocketDisconnect

pass

from fastapi import FastAPI, WebSocket, WebSocketDisconnect

pass

from fastapi import FastAPI, WebSocket, WebSocketDisconnect

pass

This endpoint is intentionally boring. That’s a feature. Your real complexity should live in the services that manage audio generation, avatar rendering, and session state. The websocket should stay thin enough that you can reason about backpressure, retries, and disconnects.

Designing the session flow

There are a few patterns that work well for avatar sessions:

• One websocket per interactive session is usually enough for a browser client. It keeps event ordering straightforward.

• Separate control plane from media plane. Use FastAPI for session creation, auth, and orchestration; keep media streaming handled by the avatar service or WebRTC stack.

• Make session state explicit. Store the avatar id, voice choice, prompt/instructions, and lifecycle timestamps somewhere you can inspect later.

• Assume disconnects. Browsers sleep, networks flap, tabs close. Your backend should be able to resume or clean up without leaking resources.

The most common mistake is to let the websocket handler become a catch-all for business logic, media state, and provider API calls. That works for a demo and then becomes painful to debug. Keep the websocket message schema small and define a few stable event types:

• session.created

• user.text or user.audio

• agent.partial

• avatar.status

• session.closed

Even if your actual event names differ, the idea is the same: structure the session as a state machine, not as a stream of ad hoc JSON blobs.

Where the avatar rendering belongs

For a realtime avatar, the backend usually should not be rendering video itself unless you are building media infrastructure. Instead, your application passes intent and configuration to a service that is purpose-built for synchronized face generation.

That service is responsible for things like:

• aligning speech and mouth motion

• managing session-scoped video output

• returning stream metadata or embed URLs

• enforcing quality tier and usage constraints

In a browser-based product, you typically present the rendered avatar as a video stream or an embedded client. If the avatar is being driven by a voice agent, your orchestration layer needs to ensure that the speech stream and the avatar stream share the same conversational state. That is where many implementations get subtle bugs: the model answer is correct, but the rendered face is one turn behind because the session event was not propagated cleanly.

Latency also matters at the handoff points. If you wait for the full assistant response before initiating TTS or avatar playback, you create unnecessary delay. Better patterns are incremental: start synthesis as soon as you have enough text, stream chunks when available, and let the avatar service consume a live audio or speech signal rather than a completed transcript if the platform supports it.

Adding Protoface to a Python service

In a Python stack, the practical integration point is usually the REST API or the Python SDK. That gives you programmatic control over avatars and sessions, while FastAPI handles your app-specific routes and websocket coordination.

A typical flow is:

1. Create a session from your backend using an API key.

2. Return a session identifier or embed URL to the browser.

3. Stream conversation events over your websocket.

4. Close the session when the conversation ends.

The exact request fields depend on the current docs, but the shape looks like this:

import requests

print(session)

import requests

print(session)

import requests

print(session)

If you prefer an SDK-driven flow, the Python SDK gives you the same basic control without hand-crafting HTTP calls. The repository is useful for reading examples and checking the current surface area: GitHub repo.

For developers already building on LiveKit, the plugin route is even more direct because the avatar can be dropped into an existing voice agent. That is a different architecture than a bare FastAPI/WebSocket app, but it is a good fit when your agent already uses LiveKit for audio transport and you want a synchronized face without reworking the whole media layer. The relevant integration is the LiveKit plugin published on PyPI and its examples in the repo: PyPI package and plugin repo.

FastAPI implementation notes that save time later

A few things matter once you move beyond the first prototype:

• Use async end-to-end. Blocking calls in your websocket handler will show up as jank under load.

• Separate session creation from media streaming. You want to create, inspect, and tear down sessions independently of browser reconnects.

• Log conversation boundaries. Session start, user turn, assistant turn, avatar state, and disconnects are the minimum useful audit trail.

• Rate limit at the edges. If you expose session creation over HTTP, protect it like any other public control surface.

• Budget for backpressure. When the client falls behind, drop or coalesce nonessential events instead of buffering forever.

For local development, it helps to keep a fake avatar adapter around so your websocket logic can be tested without depending on live media infrastructure. That way you can validate ordering, retry behavior, and cleanup logic before you wire in the real provider.

Why this is simpler than doing everything in-browser

It can be tempting to push as much logic as possible into the browser. For a talking avatar, that usually becomes a security and operational problem quickly. If the browser needs direct credentials for avatar/session creation, you have to expose secrets or invent brittle token exchange logic. If the browser is responsible for media orchestration without a server-side control plane, you lose observability and make it harder to enforce usage limits or cleanup.

A backend with FastAPI gives you a stable place to hold secrets, validate inputs, and manage lifecycles. The websocket gives you a low-latency interactive channel. The avatar service handles the media-heavy part. That division of labor is the right default for most production systems.

Conclusion

The core pattern is straightforward: use FastAPI to manage sessions and WebSocket traffic, keep the session state explicit, and delegate synchronized avatar rendering to a service designed for realtime media. The hard parts are not the HTTP endpoints; they are latency, cleanup, and keeping the conversation state aligned across text, speech, and video.

If you want to build this out, start by defining your session events, then wire up a minimal websocket server, and only after that connect the avatar provider. The public documentation at docs.protoface.com is the right place to fill in exact request fields, SDK methods, and deployment details. If you want a working reference implementation, the quickstarts linked from the Protoface repository are the fastest way to see the pieces assembled end to end.