Introduction

Realtime AI avatars are not just “video rendering” attached to a chatbot. They’re a streaming system: audio is arriving continuously, speech synthesis is producing frames or audio chunks continuously, lip-sync has to stay aligned, and your application still needs to handle interrupts, cancellations, retries, and client disconnects without piling up latency.

If you’re building this in TypeScript, the hard part is usually not “how do I call the API?” It’s “how do I keep end-to-end latency low while multiple conversations, sessions, and media streams are active at once?” By the end of this post, you should have a practical mental model for concurrency, throughput, and backpressure in an avatar pipeline, plus a few implementation patterns you can use immediately.

Concurrency: model the pipeline, not just the request

For realtime avatars, concurrency is mostly about overlapping stages safely. A typical path looks like this:

user audio or text → agent reasoning → TTS or voice streaming → avatar synthesis / video face sync → transport to client

Each stage has its own rate and buffering behavior. If you treat the whole thing as “one async function,” you’ll often end up serializing work that should overlap. That wastes latency and increases tail jitter.

In TypeScript, the most reliable pattern is to isolate each stage behind a bounded queue or async iterator, then connect stages with cancellation-aware tasks. That gives you three benefits:

• Clear ownership: each stage owns one responsibility.

• Independent scaling: reasoning, synthesis, and transport can be tuned separately.

• Predictable shutdown: when a user hangs up, you can cancel the pipeline without leaking work.

Two practical rules help a lot:

1. Don’t spawn unbounded promises per incoming event.

2. Always propagate cancellation from the client session to the downstream work.

For example, if a new user utterance arrives before the previous one finishes, you usually want to abort the old synthesis and start the new turn. That’s not just a UX improvement; it keeps your queue from accumulating obsolete work.

Throughput: measure the bottleneck, then remove it

Throughput is not the same as concurrency. You can run many tasks concurrently and still have poor throughput if one part of the pipeline is single-threaded, chatty, or blocked on network round trips.

In realtime avatar systems, common bottlenecks are:

• TTS startup time: slow first audio chunk increases perceived lag.

• Frame generation or video compositing: expensive per-frame work can dominate CPU.

• WebRTC transport: network jitter and congestion can force you to buffer more than you want.

• Event fan-out: pushing every intermediate state to logs, dashboards, or the browser can become a hidden tax.

The right metric is usually end-to-end “turn start to first visible response,” not raw request count. Break that down into:

• time to first token / first speech chunk

• time to first avatar frame

• steady-state chunk cadence

• queue depth under load

When you profile, look for two classes of inefficiency:

1. Excess serialization. Example: awaiting every sub-step in sequence even when some steps can overlap. If avatar prewarm, prompt construction, and metadata fetch are independent, start them together with Promise.all, but only for truly independent work.

2. Excess copying. Streaming systems often move audio buffers, JSON event payloads, and image/frame references around. Avoid re-encoding or cloning data unless a stage genuinely needs a new representation.

Backpressure: the part most teams miss

Backpressure is how a system says “slow down” before it falls over. In a realtime avatar app, this matters because every stage can outrun the next one. The TTS layer may emit chunks faster than the avatar compositor can consume them, or your server may accept more sessions than the GPU or media worker pool can support.

Without backpressure, one of three things usually happens:

• latency climbs as buffers grow

• memory usage rises until the process is unstable

• your app starts dropping work in ad hoc ways, which is usually worse than controlled shedding

In TypeScript, bounded queues are the simplest useful tool. A queue with a fixed capacity forces you to choose a policy when it fills up:

• Block: wait until downstream catches up.

• Drop latest: discard new low-priority work.

• Drop oldest: keep the freshest state, which is often right for live conversations.

• Cancel and replace: ideal for user interruptions or barge-in.

For avatars, “cancel and replace” is often the right answer during a conversational turn. If the user speaks again, the old turn is stale. Don’t preserve it just because you can.

A simple TypeScript pattern for bounded streaming

Here’s a compact pattern for a bounded async queue you can use between pipeline stages. It is intentionally minimal; production code should add metrics, timeout handling, and graceful shutdown semantics.

type Item<T> = { value: T; resolve: () => void; reject: (e: unknown) => void };

}

type Item<T> = { value: T; resolve: () => void; reject: (e: unknown) => void };

}

type Item<T> = { value: T; resolve: () => void; reject: (e: unknown) => void };

}

The exact implementation is less important than the policy. A queue with capacity forces you to confront overload immediately instead of hiding it in memory.

Cancellation and turn-taking in practice

Realtime avatar apps are interactive, so the user can interrupt at any time. That means your pipeline should be turn-aware, not just session-aware. A good mental model is: one active turn per session, with a single cancellation token controlling all work spawned by that turn.

In TypeScript, use AbortController at the session boundary and pass its signal into every async operation that can block or stream. If a new turn starts, abort the old controller, flush or drain its queues, and create a fresh controller for the new turn.

const controller = new AbortController();

}

const controller = new AbortController();

}

const controller = new AbortController();

}

That pattern keeps the system responsive under barge-in. It also protects you from a subtle bug: if a previous turn keeps streaming after the user has moved on, you end up with avatar motion that no longer matches the conversation state.

Where Protoface fits: keep the avatar layer narrow

This is where Protoface is useful: it gives you a focused avatar surface so your app can treat avatar rendering as a streaming endpoint rather than a custom media subsystem. For a TypeScript service, that means you can keep your own concurrency and backpressure policy in the agent, while delegating avatar session creation and face synchronization to the platform.

If you’re wiring a LiveKit voice agent, the plugin path is the cleanest integration point. The general shape is:

// Python side, inside a LiveKit agent process

// Python side, inside a LiveKit agent process

// Python side, inside a LiveKit agent process

If you prefer direct API control, the REST API is a good fit for session orchestration and lifecycle management. A basic request pattern looks like this:

curl https://api.protoface.com/<endpoint> \
-d '{"avatar_id":"...","session_options":{}}'

curl https://api.protoface.com/<endpoint> \
-d '{"avatar_id":"...","session_options":{}}'

curl https://api.protoface.com/<endpoint> \
-d '{"avatar_id":"...","session_options":{}}'

Exact endpoints and fields are in the docs, but the architectural point is the same: your application should manage queueing, turn cancellation, and load shedding; the avatar service should handle the media-specific part of the problem.

Operational tips that matter in production

A few details make a big difference once you have real traffic:

• Cap session fan-out: don’t let one noisy tenant monopolize workers.

• Instrument queue depth: if it trends upward, you have a throughput issue before you have an outage.

• Separate hot and cold paths: setup, auth, and avatar provisioning should not sit on the latency-critical path of each utterance.

• Prefer bounded retries: retrying a stale turn is usually worse than failing fast and letting the agent re-ask or recover.

• Log turn IDs, not just session IDs: when a user interrupts, you need to see which turn produced which media.

If you expose avatars through the browser, keep the browser transport simple. An iframe embed is often the safest way to ship an interactive avatar without putting API keys in the client. That reduces your attack surface and removes a whole class of browser-side concurrency bugs from your app code.

Conclusion

For realtime AI avatars, performance work is mostly about respecting the shape of the stream: overlapping the right stages, measuring the right latency, and applying backpressure before your queues become your outage. In TypeScript, the practical tools are bounded queues, turn-scoped cancellation, and careful separation between reasoning, media generation, and transport.

Keep the avatar layer narrow, keep your pipeline cancellable, and treat overload as a design case rather than an exception. If you want implementation specifics, the docs are the next stop: docs.protoface.com. For code examples and integration surfaces, the relevant quickstarts and SDK/plugin repos linked there are usually the fastest way to get from concept to a working system.