How to Scale Realtime Talking Avatars in TypeScript Without Blowing Up Latency or Cost
TypeScript patterns for scalable realtime talking avatars: stream audio/video, bound session state, control latency and cost.
Introduction
Building a realtime talking avatar sounds straightforward until you put it under actual load: audio and video have to stay synchronized, token generation can’t stall the render loop, WebRTC sessions need to recover cleanly, and your per-user cost has to stay predictable. The failure modes are usually subtle. You don’t “crash”; you just add 200 ms here, 400 ms there, and the experience starts feeling uncanny.
This post is about the architecture and TypeScript-side implementation details that keep realtime avatars responsive and affordable. By the end, you should have a mental model for where latency comes from, how to keep session state bounded, and how to design your agent so you can scale to more concurrent conversations without scaling cost linearly.
Start with the real latency budget
For a talking avatar, end-to-end latency is the sum of several independent systems:
Speech input latency: microphone capture, VAD, streaming transcription.
Agent latency: intent routing, LLM latency, tool calls, response streaming.
Avatar latency: audio-to-lip-sync alignment, video frame generation, transport.
Network latency: client ↔ server and media transport overhead.
The important part is that these are not just “fast” or “slow” in isolation. They create queueing effects. If your agent waits for a full LLM completion before it emits any audio, the avatar can’t start speaking early. If your avatar renderer waits for long text chunks, mouth motion gets disconnected from the utterance. If your media path uses a separate channel for audio and video with weak synchronization, the user sees drift.
The practical target is not “minimum latency” in the abstract. It’s keeping time to first spoken motion low, then keeping audio/video drift bounded throughout the session. That means:
Stream the agent output.
Start audio and lip-sync on partial output where possible.
Keep per-session state minimal and isolated.
Avoid doing expensive work on the critical path of every turn.
Design the TypeScript service around session boundaries
If you are building a TypeScript backend that coordinates avatar sessions, treat each session as a short-lived, stateful media transaction rather than a general-purpose request. That sounds obvious, but many implementations accidentally turn every conversation into a long-lived object graph with caches, listeners, queues, and retries that are never reclaimed.
A scalable structure usually looks like this:
Control plane: HTTP endpoints to create avatars, mint sessions, and return connection metadata.
Session worker: handles one realtime conversation, streams events, tracks state, and tears down cleanly.
Media layer: WebRTC or an equivalent realtime transport for audio/video.
In TypeScript, keep the session object small and explicit. Store only what you need to resume or terminate the conversation: session ID, user ID, media connection handle, and a tiny amount of agent state. Do not attach the entire request context, user profile, prompt history, and tool clients to every session object. That turns memory pressure into latency pressure once concurrency rises.
Also make teardown deterministic. Media sessions leak cost when they remain connected after the user has left. Use explicit timeout and disconnect paths, and make them idempotent.
Stream early, not just often
The biggest latency win usually comes from changing your agent from “wait for answer, then speak” to “stream partial answer, then speak while thinking.” For a voice agent, that means producing incremental text or audio as soon as the model yields it, and feeding that stream into your avatar pipeline immediately.
There are two common anti-patterns:
Buffering the full LLM response before synthesis or lip-sync.
Chunking too aggressively, which creates unnatural pauses and tiny media bursts.
You want a stable intermediate chunk size: enough text to preserve prosody, small enough to keep the avatar moving. If your TTS or avatar service accepts streaming input, pass tokens or sentence fragments through as they arrive. If it only accepts text segments, flush on punctuation, clause boundaries, or short time windows.
Also pay attention to backpressure. When media generation lags behind token generation, the correct response is not to keep accumulating unlimited buffered text. Cap the queue, drop low-value intermediary updates, and favor the latest turn state. A realtime avatar is not a log consumer; it is a live surface.
Control cost by matching fidelity to the interaction
Realtime avatar systems become expensive when every session gets treated like a premium demo. Cost control is mostly about quality tiering and workload shaping:
Use the lowest visual quality that still fits the use case. A support bot embedded in a dashboard often does not need the same fidelity as a sales demo.
Keep conversations short-lived where possible. Idle sessions are hidden cost.
Bound reconnection and retry behavior. Aggressive retries can double or triple spend when a downstream media step is failing.
Separate interactive and non-interactive paths. Not every request needs a full realtime session; some can be handled with async voice generation or a static fallback.
Another useful pattern is to define a “session budget” up front. For example: if a session is idle for N seconds, close it; if a user exceeds a turn limit, degrade or end gracefully; if media setup takes too long, fall back rather than waiting indefinitely. Those rules are boring, but they prevent the worst-case tail from dominating the month.
In practice, this also means instrumenting three metrics per session:
Time to first audio
Time to first visible mouth motion
Session duration and active speaking time
If those metrics drift, you know whether the problem is model latency, media startup, or unnecessary idle time. Without that separation, you end up optimizing the wrong layer.
Where Protoface fits: session orchestration without exposing your media stack
This is the part where a dedicated avatar service earns its keep. With Protoface, you can create and manage realtime avatar sessions through the REST API, or integrate a synchronized talking face directly into a LiveKit voice agent using the livekit-plugins-protoface plugin. The important architectural point is that you can keep avatar-specific logic out of your application core and let the integration surface handle the media coordination.
For backend-controlled workflows, the REST API is the right fit when you want to mint sessions from your own TypeScript service and keep API keys off the client. A minimal request looks like this; the exact fields depend on the endpoint you’re using, so check the docs:
If you are already using LiveKit for the voice layer, the plugin path is usually the lowest-friction way to add the face. The agent continues to manage conversation logic; the plugin keeps the avatar synchronized with the spoken output. That means fewer custom queues in your TypeScript service and fewer chances to desynchronize audio and video.
For self-serve web embeds, the iframe model is even simpler: the browser never sees your API key, and the iframe can enforce allowlists and rate limits independently. That is useful when you want a lightweight deployment surface without turning your frontend into a media control plane.
If you want implementation details, use the docs rather than guessing at payloads or lifecycle events: docs.protoface.com.
TypeScript implementation gotchas that matter in production
A few things consistently cause trouble in realtime services:
1. Don’t tie media cleanup to request lifetime. HTTP requests end; sessions live longer. Use separate timers or worker tasks for session expiry.
2. Don’t let one slow session block the rest. Any shared queue, lock, or per-process event loop contention will show up as jitter. Keep session work isolated and async.
3. Don’t assume reconnects are free. A broken WebRTC connection may need a full session restart. Build your state machine so “restart” is a first-class path, not an edge case.
4. Don’t over-cache prompt or media state. Cache only what reduces repeat work across sessions. Per-session caches usually just increase memory footprint and GC churn.
As a rule, if a piece of state is not needed to send the next media frame or answer the next user turn, it probably doesn’t belong on the hot path.
Conclusion
Scaling realtime talking avatars is mostly an exercise in discipline: stream early, keep session state tight, bound retries, and make teardown explicit. The technical challenge is less “can we generate an avatar?” and more “can we keep the entire conversation path predictable as concurrency rises?”
If you’re building this in TypeScript, start by instrumenting latency at each hop, then simplify the session lifecycle until you can reason about it on a whiteboard. When you’re ready to wire in an avatar layer, use the integration surface that matches your architecture: REST for backend orchestration, the LiveKit plugin for voice-agent integration, or an iframe for browser embeds. The docs at docs.protoface.com and the quickstarts linked from the repo are the fastest way to get from concept to a production-shaped implementation.