Introduction

Building a realtime avatar service is mostly a capacity-planning problem disguised as a media problem. The hard parts are not “can we render a talking face?” but “can we do it for many tenants, with bounded latency, predictable cost, and safe isolation between customers?” If you get those wrong, the system works fine in demos and fails under load or during a noisy neighbor incident.

This post walks through a practical TypeScript design for a multi-tenant avatar service: how to model sessions, how to size media workers, how to separate control-plane and data-plane concerns, and how to keep capacity predictable when customers create bursty interactive sessions. By the end, you should be able to design a service that accepts avatar/session requests, allocates media resources deterministically, and exposes a clean API to upstream voice agents or web apps.

Start with the right mental model

A realtime avatar stack usually has three distinct pieces:

• Control plane: authenticated APIs for creating avatars, sessions, policies, and billing records.

• Media plane: the realtime audio/video path that handles lip sync, speech playback, and outbound video transport.

• Tenant policy layer: per-customer limits, routing rules, quality tier mapping, and abuse controls.

The biggest mistake is to treat avatars as “just another REST resource.” The session lifecycle is stateful and time-sensitive. The control plane should issue intents, not directly perform media work. A good request flow is:

1. Tenant calls your API to create a session.

2. Your API validates auth, quota, and requested quality tier.

3. You reserve capacity before you hand out join credentials or embed tokens.

4. The media worker establishes the realtime pipeline and reports session state back.

That separation gives you a clean place to enforce limits and a clean place to scale. It also makes failure modes easier to reason about: if reservation fails, the request fails fast; if a worker dies, the control plane can mark the session unhealthy and reclaim the slot.

Model capacity as explicit reservation, not best effort

For predictable capacity, every session should consume a known budget at admission time. That budget can be simple:

• One session = one media slot for a given quality tier.

• Slots are tiered, because higher quality usually costs more CPU/GPU, bandwidth, or both.

• Slots have a TTL so abandoned reservations do not pin capacity forever.

In practice, keep a per-tier inventory in a datastore that supports atomic compare-and-swap or a transactional decrement. The shape looks like this:

type QualityTier = "standard" | "hd";

}

type QualityTier = "standard" | "hd";

}

type QualityTier = "standard" | "hd";

}

Admission logic should be deterministic:

async function reserveSlot(tenantId: string, tier: QualityTier) {

}

async function reserveSlot(tenantId: string, tier: QualityTier) {

}

async function reserveSlot(tenantId: string, tier: QualityTier) {

}

The point is not this exact schema; it is that “admit or reject” happens before any media session starts. If you do that consistently, you can answer a customer’s question about capacity with an actual number, not a hopeful guess.

Tenant isolation is mostly about the control plane

Multi-tenancy failures tend to happen in metadata, not in video frames. You need to isolate:

• API keys and auth scopes

• Session ownership

• Per-tenant rate limits

• Per-tenant configuration such as voice, instructions, and allowed origins

• Usage accounting for billing and overage enforcement

A useful pattern is to make tenant identity part of every persisted record and every cache key. Avoid “global session lookups” keyed only by session ID if those IDs ever cross trust boundaries. In TypeScript, enforce this by carrying tenant context through every service boundary:

interface TenantContext {

}

interface TenantContext {

}

interface TenantContext {

}

Two gotchas matter in production:

• Idempotency: session creation endpoints should tolerate retries. Use an idempotency key so a client timeout does not create two sessions.

• Cleanup: reservations and active sessions need heartbeats or expiry. If a worker crashes, reclaim the slot automatically.

For abuse control, rate limit both the creation path and the active-session path. Creation limits are per tenant and per API key; embed or browser-facing flows usually need per-IP limits as well. If you have a customer-managed iframe surface, make the browser talk only to the iframe origin and keep secrets server-side.

How the media path usually works in a voice-agent system

Realtime avatars are typically attached to a voice agent pipeline. The agent receives speech audio, runs ASR/LLM/TTS or some equivalent text/audio loop, and the avatar consumes the resulting speech stream plus timing signals to drive lip sync and expression. The avatar service should not try to own the whole agent stack; it should expose a media endpoint or plugin that can be inserted into an existing pipeline.

From an implementation point of view, keep the media worker stateless where possible. Any state that matters for billing or failover should live in the control plane or a durable session store. The worker’s job is to:

• join the realtime room or session,

• attach the avatar media track,

• sync to the incoming speech stream,

• emit health and usage events back to the API.

If you are using WebRTC under the hood, remember that transport constraints matter: NAT traversal, ICE retries, jitter, and packet loss all affect perceived responsiveness. Your capacity model should include headroom for those realities. In other words, if a worker “supports 20 sessions” in a lab, treat that as the ceiling only after you measure under realistic network conditions.

A concrete TypeScript service shape

A service like this tends to have four modules:

1. HTTP API for auth, session creation, and configuration.

2. Capacity service for inventory and reservation logic.

3. Worker coordinator for dispatching sessions to media hosts.

4. Usage pipeline for recording session minutes and tiered billing.

TypeScript is a good fit for the API and coordinator layer because the interfaces are explicit and the request/response contracts are easy to keep consistent. Keep the actual media processing in whatever runtime makes sense for your stack; the control plane does not need to be co-located with the media plane.

A minimal session creation endpoint might look like this:

app.post("/v1/sessions", async (req, res) => {

});

app.post("/v1/sessions", async (req, res) => {

});

app.post("/v1/sessions", async (req, res) => {

});

The exact fields will vary, but the invariant should not: a successful response means capacity has been reserved and a worker can be attached without another admission decision.

Where Protoface fits

This is the point where a developer platform earns its keep. With Protoface, you do not need to build the avatar rendering, lip-sync transport, and session orchestration layers from scratch just to prove out your app. The REST API at api.protoface.com is the control surface for avatars and realtime sessions, and the Python SDK and LiveKit plugin are the natural integration points if your stack already has a voice-agent runtime.

For example, if you are using a LiveKit agent, the plugin approach keeps the avatar attached to the agent session instead of inventing a custom bridge. The shape is intentionally lightweight:

# illustrative only; exact fields and setup are in the docs

)

# illustrative only; exact fields and setup are in the docs

)

# illustrative only; exact fields and setup are in the docs

)

If you are provisioning sessions from a backend, the REST API gives you the same control-plane boundary described above, with standard bearer authentication:

curl https://api.protoface.com/v1/sessions \
}'

curl https://api.protoface.com/v1/sessions \
}'

curl https://api.protoface.com/v1/sessions \
}'

Use the public docs for the exact request shapes and session lifecycle details: docs.protoface.com.

Operational details that save you later

Three details tend to separate a service that scales from one that merely works:

• Backpressure: when you are near capacity, fail fast with a clear error instead of queuing indefinitely.

• Observability: track reservation latency, active sessions by tier, worker health, and session teardown reasons.

• Graceful degradation: if you support multiple tiers, prefer rejecting a higher tier while still admitting a lower tier, rather than dropping all traffic.

Also, make billing follow actual session state, not just request intent. A created-but-never-attached session should not be billed the same way as an active realtime session. That requires clear state transitions: reserved, attached, active, draining, closed.

If you are exposing browser-facing embeds, keep the browser boundary narrow. The iframe pattern is safer than exposing API keys client-side, and it lets you enforce origin allowlists and per-embed limits without pushing trust into the page. That is usually the right choice when the customer just wants an avatar on a site, not a bespoke backend integration.

Conclusion

Predictable capacity for realtime avatars comes from treating sessions as reserved media resources, not loose API calls. Keep admission control in the control plane, keep the media plane stateless where possible, and make tenant identity, quotas, and expiration explicit in your data model. If you do that, the rest becomes a straightforward scaling problem instead of a guessing game.

For concrete implementation details, integration examples, and the exact request shapes, start with the docs at docs.protoface.com. If you are wiring an avatar into an existing voice agent, the LiveKit plugin path is the fastest way to validate the architecture before you build more infrastructure around it.