Introduction

Realtime avatar backends look simple from the outside: a browser opens a session, the model speaks, and a video face tracks the audio. In practice, the backend has to coordinate a WebRTC or media-streaming session, synthesize or proxy audio, keep lip-sync aligned, and survive the usual production problems: traffic spikes, cold starts, worker failures, and session routing across multiple app instances.

This post is about the scaling problem, not the avatar model itself. By the end, you should understand how to structure a FastAPI backend for realtime avatar sessions, where horizontal scaling breaks naïve designs, and what to put behind shared state so you can add more workers without losing live sessions. I’ll also show where Protoface fits when you want to avoid building the avatar layer from scratch.

What “horizontal scaling” actually means for realtime avatars

For a regular HTTP API, horizontal scaling is straightforward: any request can hit any stateless worker, and shared data lives in Redis, Postgres, or another backend. Realtime avatar systems are less forgiving because a session is a long-lived conversation with active media transport.

The key distinction is that a session usually has two separate planes:

• Control plane: create sessions, issue tokens, attach metadata, persist state, rotate credentials, and authorize the client.

• Media plane: transport audio/video frames over WebRTC or a similar streaming channel, keep timing aligned, and push updates with low latency.

You can horizontally scale the control plane very easily. The media plane is the part that needs deliberate design.

The most common failure mode is assuming that “session state in memory” is fine because the process is fast. It works until the browser reconnects to a different worker, or your autoscaler moves traffic, or your pod gets restarted mid-conversation. At that point, the new worker has no idea which avatar, voice, prompt, or timing state belonged to the session unless you externalize it.

Designing a FastAPI backend that can scale

A practical FastAPI service for realtime avatars should avoid treating one process as the owner of a session. Instead, make the session record the source of truth and let workers act as executors.

Keep session state external and explicit

Store the minimum durable state you need to resume or inspect a session:

• session ID

• user or tenant ID

• avatar configuration

• voice/model settings

• lifecycle status

• media endpoint or room assignment

In FastAPI, the session creation endpoint should create the record first, then return a short-lived connection payload. The worker that accepts the WebRTC or streaming connection can reconstruct the session from the store.

from fastapi import FastAPI, HTTPException

}

from fastapi import FastAPI, HTTPException

}

from fastapi import FastAPI, HTTPException

}

That shape matters more than the exact field names. The important part is that the app can regenerate the session context from storage instead of relying on process memory.

Use shared infrastructure for routing, not sticky process memory

Sticky sessions can hide scaling bugs, but they do not solve them. If a client is pinned to one pod, that pod still becomes a single point of failure for the live session, and autoscaling becomes awkward. A better approach is to use shared state plus deterministic routing.

In practice, this means one of the following:

• Route the session to a worker based on a shared mapping in Redis.

• Use a media server or room service that can reconnect clients independently of the API worker.

• Keep the avatar logic stateless enough that a reconnecting client can resume from the persisted session record.

For voice agents, the audio pipeline is usually the bottleneck before the web server is. Your FastAPI app should not try to own the full media loop unless you are very sure about concurrency, backpressure, and reconnect behavior. The safer pattern is: FastAPI creates and manages sessions, while a specialized media worker handles the realtime stream.

Model the realtime path as a state machine

Session lifecycle bugs become much easier to reason about if you model them explicitly. A simple state machine is often enough:

• creating — session record exists, media not yet attached

• ready — client can connect

• active — media is flowing and the avatar is speaking

• idle — connected but waiting for input

• ended — terminal state

• failed — terminal or recoverable depending on cause

When you scale horizontally, every worker must enforce these transitions the same way. Use optimistic concurrency or transactional updates so two workers do not both think they own the same live session.

Concurrency, backpressure, and latency budgets

Realtime avatar systems are sensitive to tail latency. A 200 ms API request is fine for ordinary CRUD; the same delay can make conversation feel laggy when it sits in front of a voice turn or frame update.

There are a few concrete rules that help:

1. Keep HTTP handlers short. Do validation, auth, and state transitions; push expensive work to background tasks or dedicated workers.

2. Bound queue growth. If audio frames or LLM responses pile up, drop or coalesce work rather than letting the queue drift indefinitely.

3. Track end-to-end latency. Measure from user audio input to avatar video output, not just server processing time.

4. Separate control-plane failures from media-plane failures. A transient database issue should not necessarily kill an active media session.

Also remember that autoscaling is not instant. If your first worker melts under load while new pods are coming up, the session layer needs to degrade gracefully. That usually means rejecting new sessions cleanly rather than overcommitting existing ones.

FastAPI implementation details that matter in production

FastAPI is a good fit here because it gives you async request handling and clean dependency injection, but you still need to be careful about how you structure the app.

Use async I/O for network-bound work, but do not assume async automatically means scalable. If your code calls a blocking SDK or performs CPU-heavy audio work in the request path, move that logic out of the event loop. For example:

from fastapi import BackgroundTasks

return {"ok": True}

from fastapi import BackgroundTasks

return {"ok": True}

from fastapi import BackgroundTasks

return {"ok": True}

For authentication, short-lived session tokens are better than handing the browser long-lived API credentials. The browser should only know enough to connect to one session, not enough to create arbitrary sessions.

If you expose a webhook or callback endpoint, treat it as part of the distributed system. Verify signatures, make handlers idempotent, and assume duplicates will happen. Realtime systems often retry on network flakiness, and retries are normal.

Where Protoface fits in this architecture

If your application needs the avatar layer itself rather than just the backend around it, Protoface gives you a few lower-friction integration points. For a Python-centric stack, the SDK is useful when you want to create or inspect avatars and sessions from your own service logic. For voice agents built on LiveKit, the livekit-plugins-protoface plugin can drop a synchronized talking face into the agent without you wiring the media path manually; the plugin and examples are in the repository linked from the quickstarts and docs.

The operationally important part is that the session API remains explicit. Your FastAPI app still owns authorization, tenant boundaries, and lifecycle decisions, while the avatar backend handles the synchronized video face. That keeps your control plane small and your scaling story sane.

For example, a server-side session creation flow might look like this, with the exact request fields defined in the docs:

import requests

session = resp.json()

import requests

session = resp.json()

import requests

session = resp.json()

If you are integrating from Python, the SDK is a cleaner option than raw HTTP, and the public docs cover the session and avatar flows in more detail: docs.protoface.com. For LiveKit agent work specifically, the plugin repository is the best place to inspect examples and update patterns: GitHub organization.

Common scaling pitfalls to avoid

There are a few mistakes I see repeatedly in realtime systems:

• Keeping session state in process memory. Fine for a demo, fragile in production.

• Treating reconnects as new sessions. That breaks continuity and wastes resources.

• Mixing control and media logic in one endpoint. Makes latency and failure domains harder to manage.

• Ignoring idempotency. Duplicate session creation or teardown requests happen under retry.

• Overusing sticky sessions. They mask the real routing problem and limit resilience.

If you need a quick sanity check, ask whether any one worker can disappear without permanently losing the session. If the answer is no, the system is not horizontally scalable yet.

Conclusion

Horizontal scaling for realtime avatar backends is mostly about discipline: keep the control plane stateless, externalize session state, separate media handling from ordinary HTTP request flow, and treat reconnects as a normal case. FastAPI is a solid foundation for that architecture, but the hard part is designing around long-lived realtime sessions rather than classic REST calls.

If you are building this yourself, start with a small state machine, persistent session records, and a clean separation between API workers and media workers. If you want a faster path to a working avatar layer, read the docs, try the Python SDK or LiveKit integration, and validate the session lifecycle before you scale traffic: docs.protoface.com.