Introduction

When you add a realtime avatar to a voice agent, the hard part is not lip sync or rendering a face. It is keeping audio, video, and conversational state aligned under load while preserving low end-to-end latency. If the system falls behind, users notice it immediately: the avatar talks over the agent, mouth movement lags the speech, or sessions fail when a single backend gets hot.

This post walks through a practical load-balanced architecture for a realtime avatar stack built around Twilio Voice for telephony ingress/egress and a WebRTC/video pipeline for avatar delivery. By the end, you should have a clear implementation model for: distributing sessions across workers, keeping per-call state isolated, handling backpressure, and deciding where the avatar service sits in the media path.

I’ll also show where Protoface fits as a developer-facing avatar layer when you want to drop synchronized talking video into an existing voice agent or conversational app without building the avatar backend yourself.

Start with the media path, not the app architecture

The most common mistake is to start with app servers, databases, and queues. For realtime avatars, start with the media graph:

• Twilio delivers the phone call audio to your application.

• Your voice agent streams audio to ASR / LLM / TTS components.

• The avatar renderer consumes the agent’s speech stream and outputs a lip-synced video track.

• The client receives audio and video as separate realtime tracks and must stay within a tight jitter budget.

The key constraint is that the avatar cannot be treated like a background job. It is part of the interactive loop. If your TTS returns text or audio late, the avatar will either idle awkwardly or animate with stale content. If you fan out work to multiple workers, you need a single source of truth per live session so audio and animation state do not diverge.

In practice, a good design keeps the control plane and media plane separate. The control plane creates sessions, assigns workers, stores metadata, and handles authentication. The media plane is a low-latency path that moves audio/video frames and state updates for active calls.

Load balancing realtime sessions: sticky by session, not by request

For HTTP workloads, round-robin is usually fine. For realtime avatar sessions, it is not. A single call or web session should be pinned to one worker for the duration of the interaction, or at least to one consistent session coordinator. That worker owns:

• conversation state

• media buffers

• timing information for speech and animation

• cleanup when the session ends

You can still scale horizontally, but the unit of load balancing is the live session, not individual messages. In a Twilio-backed voice agent, the typical pattern is:

1. Receive a Twilio webhook for the inbound call.

2. Create an internal session record.

3. Select a worker using consistent hashing, rendezvous hashing, or a lease from a shared coordinator.

4. Hand the call off to that worker and keep it pinned there until hangup.

That lets you bound cross-worker coordination. If you try to distribute a single call across multiple workers, you will end up reconstructing session state over a network boundary on every turn, which is usually slower and more failure-prone than just keeping the whole call local to one process.

For the actual balancing algorithm, prioritize stability over perfect evenness. A session moving mid-call is expensive. If one worker is near capacity, it is better to reject or queue new sessions than to migrate an active one. When a worker does fail, you can rehydrate from persisted state if your application supports it, but you should design for failover as an exception path, not the default path.

Keep the conversation state small and explicit

A realtime avatar system often grows a lot of hidden state: last assistant turn, current speaking segment, partial transcript, interim TTS audio, animation blend state, barge-in flags, and network timing. That state should be represented explicitly, not inferred from timestamps sprinkled through the codebase.

A useful mental model is a per-session state machine:

• Idle: no active speech, waiting for user input.

• Listening: audio arrives, ASR is buffering or decoding.

• Thinking: LLM or rules engine is preparing the response.

• Speaking: TTS and avatar playback are active.

• Interrupting: user barges in, current speech is cut off.

• Ending: teardown in progress.

That model matters because the avatar should not be driven directly by “whatever audio exists right now.” The avatar should reflect the authoritative conversational state. For example, if the user interrupts, stop rendering the current speech segment immediately and mark the response aborted; do not keep animating until the buffered audio drains.

In a distributed stack, persist only what you need to recover the session: session IDs, current state, selected worker, and minimal transcript or turn metadata. Avoid persisting raw media unless you have a separate archival requirement. Media is ephemeral; state is what you need to make the next turn coherent.

Backpressure, latency, and barge-in

Realtime avatar systems fail in subtle ways when downstream components slow down. A few rules help keep the experience stable:

• Bound queue depth for every stage. Unbounded queues turn latency spikes into visible lag.

• Drop or truncate stale intermediate results. Partial TTS output that arrives too late is worse than no output.

• Prefer short, incremental speech chunks so the avatar can begin animating before the entire response is synthesized.

• Support barge-in by treating user audio as higher priority than assistant playback.

Twilio Voice gives you the telephony ingress/egress, but it does not solve the internal timing problem for you. Once audio is inside your application, you are responsible for coordinating the call leg, the agent, and the avatar renderer.

On the video side, WebRTC is the right delivery mechanism for interactive avatars because it is designed for low latency and adapts to changing network conditions. But WebRTC is not magic: if you enqueue too much upstream, the client will still see stale frames or delayed speech. The goal is not maximum throughput; it is predictable end-to-end latency under load.

One practical debugging tactic is to measure stage-by-stage latency separately:

• Twilio ingress to app receive time

• ASR decode latency

• LLM response latency

• TTS generation latency

• avatar render start time

• client playout time

If you only record total latency, you will not know whether the bottleneck is your queueing strategy, your speech engine, or the avatar renderer.

Where Protoface fits: an avatar layer you can drop into an existing voice stack

If you already have a Twilio voice agent and you want to add a synchronized video face, the least invasive integration is usually at the avatar boundary: keep your telephony and agent logic as-is, and connect the assistant’s speech output to an avatar session through a dedicated API or plugin.

That is the role of Protoface in this kind of architecture. The REST API is useful when your backend wants to create and manage avatars or realtime sessions directly, authenticated with API keys. The Python SDK is the ergonomic option when your orchestration code already lives in Python. If your agent is built on LiveKit, the livekit-plugins-protoface plugin can attach a talking avatar to the agent so the video face stays synchronized with the voice pipeline. See the public docs at docs.protoface.com for the exact request and session fields.

A minimal REST-style flow looks like this:

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

And a Python-side integration might look like this at a high level:

from protoface import Client

)

from protoface import Client

)

from protoface import Client

)

The exact object model varies by endpoint and SDK version, so treat this as illustrative. The important part is the boundary: the worker that owns the live call should also own the avatar session for that call, so the video face can track the same speaking state as the agent.

Twilio Voice integration pattern that actually scales

A robust production pattern is to create one session controller per call, then attach the Twilio media stream and avatar session to that controller. That controller does four jobs:

1. accepts call events from Twilio

2. routes audio into ASR/TTS and speech policy logic

3. keeps the selected avatar session bound to the call

4. releases resources on hangup or timeout

If you need to scale this out, scale the controller tier horizontally and keep the live session pinned to a single instance. A shared store can hold lease information and non-media metadata, but avoid using it as the main transport for realtime audio or animation events. Network hops are where latency and ordering bugs accumulate.

In environments where calls are bursty, capacity planning should focus on concurrent sessions, not requests per second. One long-running session with bidirectional media can consume more resources than many ordinary API calls. Keep headroom for codec work, transport overhead, and GC pauses. If you run in Python, pay attention to blocking sections and isolate media processing from slow synchronous code.

Conclusion

A load-balanced realtime avatar stack works best when you treat each live session as a pinned, stateful unit; keep media on a low-latency path; and make backpressure visible instead of hiding it in queues. Twilio handles telephony ingress, WebRTC handles interactive delivery, and your application owns the session state machine and failover policy.

If you want to add a synchronized talking face to an existing voice agent rather than build the avatar layer yourself, start with the docs at docs.protoface.com and the relevant integration examples in the GitHub repo for your stack. Build one end-to-end call path first, measure latency at each stage, then scale the controller tier only after the session model is solid.