Introduction

Horizontally scaling a realtime avatar service is mostly about controlling concurrency, latency, and state. The hard part is not generating pixels; it is coordinating WebRTC sessions, speech/animation timing, and backend capacity so that adding more users does not turn into queued sessions, jittery lip sync, or expensive idle workers.

This post walks through a practical architecture for a Python and FastAPI control plane that can create and manage avatar sessions, route work to stateless workers, and scale out cleanly behind a load balancer. By the end, you should be able to design a service that accepts a burst of avatar requests, provisions sessions predictably, and keeps realtime media flows isolated enough to scale on ordinary infrastructure.

What “horizontal scaling” means for realtime avatars

For a normal CRUD API, horizontal scaling is usually about making request handlers stateless. For a realtime avatar service, that is necessary but not sufficient. You have at least three different kinds of state to think about:

• Control-plane state: avatar definitions, session records, API keys, usage accounting, and rate limits.

• Realtime session state: an active voice interaction, its negotiated transport, and the currently assigned worker.

• Media state: the actual audio/video stream, which should stay on one worker or one tightly coupled media pipeline for the life of the session.

The main scaling rule is simple: keep the FastAPI layer stateless, keep durable state in an external datastore, and make the media/session worker the unit of realtime ownership. If you try to share live session state across multiple web servers, you will spend a lot of time rebuilding a distributed systems problem that WebRTC already made hard enough.

Separate the control plane from the media plane

A good first cut is to split the system into two services:

1. FastAPI control plane: authenticates API requests, validates inputs, creates session records, and assigns work.

2. Realtime worker pool: runs the avatar session, handles voice input/output, and maintains the media connection.

The control plane can scale horizontally with standard techniques: multiple Uvicorn or Gunicorn workers behind a load balancer, shared PostgreSQL or Redis for state, and short-lived request handlers. The worker pool scales separately based on active sessions and resource usage.

In practice, a session creation request should do as little as possible synchronously:

1. Authenticate the caller.

2. Validate the avatar/session parameters.

3. Create a durable session row with a pending status.

4. Reserve capacity or choose a worker.

5. Return connection details or a session token.

That keeps the HTTP path fast and predictable. The heavy lifting happens after the request is accepted.

Use FastAPI as a stateless orchestration layer

FastAPI is a good fit for this control plane because it makes it easy to express validation, dependency injection, and async request handling. The important part is not the framework itself; it is the discipline of not storing per-session runtime state in process memory.

A common shape is:

• API keys validated on every request.

• Session metadata stored in PostgreSQL.

• Short-lived coordination data in Redis.

• Worker assignment done through a queue, lease table, or atomic Redis claim.

A minimal session-creation endpoint might look like this:

from fastapi import FastAPI, Header, HTTPException

return {"status": "pending", "session_id": "sess_123"}

from fastapi import FastAPI, Header, HTTPException

return {"status": "pending", "session_id": "sess_123"}

from fastapi import FastAPI, Header, HTTPException

return {"status": "pending", "session_id": "sess_123"}

This is intentionally boring. Boring is good. The endpoint should not start a realtime stream itself; it should initiate a workflow that can survive retries and process restarts.

Make session assignment explicit

Scaling breaks down when session ownership is implicit. If a worker dies and another worker cannot deterministically claim the session, you get orphaned media state and stuck users.

There are a few reasonable patterns:

• Queue-based provisioning: the API enqueues a session request and a worker consumes it.

• Lease-based assignment: the API writes a session row with an expiring lease for a specific worker.

• Coordinator-based routing: a small service maps new sessions to healthy workers based on load.

For most teams, lease-based assignment is the simplest to reason about. A worker renews its lease while a session is active; if it fails, another worker can reclaim the session after the lease expires. This is much easier than trying to keep a websocket or RTP flow alive across app servers.

For WebRTC-style traffic, remember that the media path is sensitive to jitter and reconnection delays. You want fast failure detection, but you also want to avoid aggressive rebalancing. Once a session is live, let it stay where it is unless you have a clean handoff mechanism.

Model the scaling bottlenecks honestly

Realtime avatar services usually hit bottlenecks in a predictable order:

1. CPU/GPU on workers for video synthesis, lip sync, or model inference.

2. Concurrent session count per worker.

3. Network egress for video streams.

4. Control-plane churn from session setup and cleanup.

This matters because “just add more FastAPI pods” only solves the last item. A clean architecture exposes per-worker capacity, then uses that signal in the scheduler. For example, you might reject new sessions when a worker is at 90% of its target concurrent load instead of letting the queue build up and degrade every active stream.

For Python services, a few operational details are worth calling out:

• Use async I/O for network-bound orchestration.

• Keep long-running media work out of the request handler.

• Separate API workers from media workers so a spike in control traffic does not starve live sessions.

• Track p95 setup latency, active sessions per worker, reconnect rate, and media failure rate.

If you cannot measure those four things, you are basically guessing at capacity.

Rate limit where abuse is cheapest to stop

Realtime avatar systems attract two kinds of pressure: legitimate bursts and expensive abuse. You should rate limit as close to the ingress as possible, but the exact strategy depends on the surface.

For a REST API, that means API-key-based request limiting and per-account quotas. For embedded experiences, you usually want additional browser- and origin-level controls. For session APIs, limit by IP, key, and duration so one caller cannot create a large number of long-lived expensive sessions.

In FastAPI, the implementation can be simple if you already store counters in Redis. The important thing is that the limiter is shared across instances; in-memory counters will fail as soon as you run more than one pod.

Also make sure session creation is idempotent if your clients might retry. A duplicate request during a transient network failure should either return the original session or fail cleanly without provisioning two workers.

Keep media handling close to the worker

The media plane should be attached to the worker that owns the session. That worker should handle voice decoding, avatar timing, and outbound video generation without bouncing frames through multiple internal services. Every extra hop adds latency and failure modes.

That also means you should treat worker lifecycle carefully:

• Drain before shutdown so active sessions can finish or reconnect elsewhere.

• Advertise health only when the worker can accept new sessions.

• Separate readiness from liveness; a worker may be alive but temporarily full.

For deployment, it is often better to scale workers based on active session count or GPU utilization rather than raw CPU. A worker with one expensive session may be “full” long before its CPU hits 100%.

Where Protoface fits in

In a real integration, you do not want to build the avatar layer from scratch unless you have a strong reason. Protoface exposes the pieces you need: a REST API for avatar and session management, a Python SDK for programmatic control, and a LiveKit Agents plugin that drops a synchronized talking face into an existing voice agent.

That matters for scaling because it lets your FastAPI service stay focused on orchestration. A typical pattern is to create or look up a session over the API, then hand the session into the media stack rather than having your web app generate video itself. For Python, the SDK keeps the control plane code readable:

from protoface import ProtofaceClient

)

from protoface import ProtofaceClient

)

from protoface import ProtofaceClient

)

If your voice agent already runs on LiveKit, the plugin approach is even simpler because the avatar becomes part of the agent pipeline rather than a separate subsystem. The GitHub repo with examples is a useful reference: https://github.com/protoface-ai/protoface-quickstart-openai-realtime. For the API and SDK details, use the docs at https://docs.protoface.com.

Operational patterns that make scaling survivable

Three patterns are worth adopting early:

1. Persist every session transition: created, queued, assigned, active, ended, failed.

2. Use background reconciliation: if a worker dies, mark its sessions stale and recover them deliberately.

3. Separate billing events from realtime events: usage metering should be asynchronous so it does not sit on the critical path.

This is especially important if you bill by quality tier. The system needs to know not just that a session existed, but which tier it used, how long it ran, and whether it completed normally. Keep those events append-only if possible; it makes retries and audits much less painful.

Finally, write chaos tests for the boring failures: worker restarts, Redis timeouts, duplicate session creation, and network blips during session setup. Realtime systems usually fail in these edge cases, not in the happy path.

Conclusion

Horizontally scaling a realtime AI avatar service is mostly about enforcing clean boundaries: FastAPI as a stateless control plane, a durable store for session state, a separate worker pool for media, and shared coordination for capacity and rate limiting. Once those boundaries are in place, adding replicas becomes an operational exercise rather than a rewrite.

If you are building this kind of system, start by mapping your session lifecycle end to end, then choose a worker assignment model that survives retries and worker loss. From there, wire in observability, quotas, and draining behavior before you chase throughput.

For implementation details and integration examples, check the docs and the quickstarts linked from the public repository. If you are using LiveKit, the plugin path is a practical way to get a synchronized avatar into an existing agent without reworking your voice stack.