Introduction

Routing thousands of live avatar sessions is mostly a systems problem, not an avatar problem. The failure modes are the same ones you see in any real-time media service: sticky state in the wrong place, slow control-plane operations on the call path, fan-out that grows with traffic, and cleanup that is too lazy when clients disconnect ungracefully. If you add Twilio into that mix, you also inherit PSTN timing constraints, webhook retries, and the need to keep media bridging stable even when your app layer is under load.

By the end of this post, you should have a concrete mental model for how to design Twilio-backed avatar sessions so they scale horizontally, survive spikes, and fail cleanly instead of dropping calls. I’ll focus on the call-routing architecture, the session lifecycle, and the operational details that matter when you’re handling thousands of concurrent conversations.

Start with the right split: signaling, media, and avatar state

The first mistake is treating “a call” as one thing. It is not. For a Twilio-to-avatar flow, there are at least three separate concerns:

• Telephony signaling: inbound webhook, call status callbacks, TwiML generation, retry handling.

• Media transport: the live audio stream, typically bridged into your voice agent runtime over WebRTC or a similar realtime transport.

• Avatar session state: which agent is speaking, current conversation context, and the synchronized video face rendering state.

If you keep these coupled in one process, you will eventually lose calls when that process blocks, restarts, or gets overloaded. The design goal is to make signaling stateless and fast, media handling horizontally scalable, and avatar session state reconstructible from durable metadata.

A good rule: the Twilio webhook should do as little work as possible. It should authenticate the request, decide where the call should go, return TwiML quickly, and hand off any heavier initialization to a worker or session service. Do not make the webhook wait on model loading, avatar creation, or any long-lived media setup.

Use call IDs as your routing primitive

At scale, you need a stable key that ties together all the moving parts. In practice, that key is usually the Twilio CallSid. Treat it as the primary identifier for the session and use it to map:

• the telephony leg from Twilio,

• the realtime agent connection,

• the avatar render session, and

• your own billing/usage record.

Keep that mapping in a fast shared store, not in process memory. Redis is the usual choice because it gives you low-latency lookups and TTL-based cleanup. The pattern is straightforward:

1. Twilio hits your webhook with a new CallSid.

2. Your app allocates a session record keyed by CallSid.

3. You select a worker or media node and store the assignment.

4. The worker joins the agent session and attaches the avatar stream.

5. Callbacks from Twilio use the same key to update state and drive cleanup.

That last point matters. If your app gets a duplicate webhook, a status callback arrives late, or the media leg reconnects, you must be able to rehydrate state deterministically from the session record. This is what prevents double-allocations and orphaned avatar instances.

Keep webhook handling idempotent and fast

Twilio will retry webhooks when it does not get a timely response. Under load, retries are normal. Your code must therefore be idempotent on session creation and cheap enough to respond within your latency budget even during a spike.

A practical pattern is:

• verify the Twilio signature,

• upsert the session row keyed by CallSid,

• return TwiML that connects the call to your media service, and

• enqueue any nontrivial work for asynchronous handling.

Here is a minimal webhook skeleton in Python. The exact TwiML and connection details depend on your media stack, but the structure is what matters:

from flask import Flask, request, Response

return Response(twiml, mimetype="text/xml")

from flask import Flask, request, Response

return Response(twiml, mimetype="text/xml")

from flask import Flask, request, Response

return Response(twiml, mimetype="text/xml")

Notice what is not here: avatar creation, model warmup, and any blocking call to a downstream service. Those should happen in a separate worker path so webhook latency stays predictable.

Scale the media path independently from the control plane

Once the call is connected, the bottleneck shifts to realtime media handling. The audio stream must be processed continuously; even short stalls become audible. The main scaling decision is whether session assignment is sticky and how you distribute live connections across workers.

For thousands of sessions, use a routing layer that can pick an available worker based on load and then keep that session pinned there for the life of the call. That worker becomes responsible for:

• terminating the streaming leg,

• forwarding audio into your agent runtime,

• receiving agent audio back,

• driving avatar lip sync / video rendering, and

• cleaning up when the call ends.

Don’t route mid-call unless you absolutely have to. Session migration is possible in theory, but in practice it increases the chance of audible glitches, duplicated audio, or avatar desynchronization. The more robust approach is to make workers disposable and sessions sticky.

Also, keep CPU-heavy work off the hot path. If you are doing VAD, transcription, TTS, and avatar rendering in the same service, isolate those stages with bounded queues. Backpressure should drop or slow nonessential work, not the telephony stream. If your worker is overloaded, it is better to reject new sessions than to let existing calls underrun.

Handle teardown like a first-class path

A surprising amount of “dropped call” behavior is actually bad cleanup. Calls end normally, but your infrastructure keeps the session open, leaks media sockets, or leaves avatar state around until a timeout fires. Under heavy traffic, those leaks accumulate and make the next wave of calls look unstable.

Use multiple signals to detect end-of-life:

• Twilio status callbacks indicating the call ended,

• media stream close events,

• agent runtime disconnects, and

• a TTL on the session record as a final safety net.

Each of those should converge on the same cleanup routine. That routine should be idempotent too. If you see the same CallSid twice, deleting the avatar session twice should be harmless.

Operationally, this is where a lot of systems fail under load: the create path is fast, but teardown depends on a best-effort callback. Build for the case where the callback is delayed or missing.

Where Protoface fits: attach the avatar after the call is already stable

This is the part that can simplify your stack: use Protoface as the avatar/session layer after you have a stable Twilio routing design. In other words, let Twilio handle the phone leg, let your media service handle the stream, and use the avatar API or SDK to create and manage the synchronized face for that live session.

If you are already building a Python-based voice pipeline, the Python SDK is the most direct way to create sessions and bind them to your worker logic. The exact fields are documented in the docs, but the shape is typically:

from protoface import ProtofaceClient

)

from protoface import ProtofaceClient

)

from protoface import ProtofaceClient

)

If you are using a LiveKit voice agent, the plugin path keeps the avatar synchronized with the agent without making you manage the video face as a separate service. The point is not the exact integration detail; the point is that the avatar should be attached to an already-routed, already-stable realtime session rather than being created inside your Twilio webhook.

That separation is what helps you scale cleanly. Twilio stays in the signaling lane, your media workers stay in the realtime lane, and the avatar layer becomes a managed dependency rather than another thing your webhook has to orchestrate under pressure.

Concrete trade-offs and gotchas

A few implementation details are worth calling out because they tend to bite teams once traffic grows:

• Do not store live session state only in process memory. You will lose it on deploys and autoscaling events.

• Do not do synchronous avatar creation in the webhook. It raises retry rates and creates duplicate work.

• Do not assume one worker can handle unlimited concurrent calls. Media processing is CPU- and bandwidth-sensitive.

• Do not rely on a single callback for teardown. Use status events plus TTL cleanup.

• Do not let queue growth hide overload. If workers are behind, shed load early and predictably.

If you need browser-based demos or customer-facing embeddings instead of phone calls, the same lifecycle discipline still applies, but the transport changes. In those cases, an iframe-based embed can keep the browser isolated from your API keys and avoid a lot of frontend plumbing. For Twilio call routing specifically, though, the main concern is still the control-plane/media split described above.

Conclusion

To route thousands of avatar sessions without dropping calls, keep the webhook thin, make session assignment idempotent, pin each call to a worker for its lifetime, and treat cleanup as a first-class path. The winning architecture is boring on purpose: stable IDs, shared state, bounded work per process, and no heavy lifting on the telephony callback.

If you are implementing this with a realtime avatar layer, keep the avatar session creation outside the Twilio hot path and attach it once the media leg is established. The public docs at docs.protoface.com cover the SDKs, API, and integration surfaces in more detail, and the quickstarts in the GitHub org are a good way to validate the flow end to end before you put production traffic on it.