How to Preserve Session State When Migrating Realtime AI Avatars to Warm Pools
Preserve realtime AI avatar session state during warm pool migration with snapshot, handoff, and media continuity patterns.
Introduction
When a realtime AI avatar goes from a cold start to a live conversation, the obvious engineering problem is latency. The less obvious one is session continuity: if you move that avatar onto a warm pool worker, how do you keep the conversation state, voice settings, and media handshake intact without making the user notice?
This matters because realtime avatars are not stateless HTTP handlers. They sit in a live media session, usually over WebRTC or a similar streaming transport, while an agent is simultaneously generating text, speech, and avatar animation. If you migrate that session between workers, you are effectively moving a live call, not just reloading a page.
By the end of this post, you should be able to design a handoff path that preserves state across warm pool migration, understand what must be copied versus re-established, and avoid the common failure modes that cause dropped audio, mismatched turns, or avatars that briefly “forget” who they are.
What session state actually needs to survive
Before talking about warm pools, be precise about the state. In practice, a realtime avatar session has at least four categories of state:
Conversation state: transcript, turn order, pending tool calls, agent memory, and any system or custom instructions.
Media state: the live connection to the client, codec/session negotiation details, and any server-side stream identifiers.
Avatar configuration: which avatar is active, quality tier, voice, speaking style, and any session-level parameters.
Operational state: rate limits, timestamps, billing metadata, and routing information such as which worker currently owns the session.
These do not all move the same way. Conversation state is usually serializable and can be snapshotted. Media state is typically bound to a live transport and must be re-established or proxied. Avatar configuration may be mutable, but it should be treated as part of the session contract so the new worker can reproduce the exact experience. Operational state is often the glue that prevents duplicate ownership or double billing.
Warm pools are about amortizing startup, not making sessions stateless
A warm pool is a set of already-initialized workers ready to receive traffic. In a realtime avatar system, warm pools help by avoiding expensive cold starts: importing model clients, allocating GPU contexts, loading voice assets, or spinning up media pipelines. But warm pools do not eliminate session affinity. They just shift the handoff point.
The key design mistake is to treat a migration as “start a new session on a new worker and hope the user does not notice.” That usually breaks because:
the client may still be connected to the old worker’s media endpoint,
the new worker may miss in-flight user utterances or agent tool calls,
the avatar can briefly resynchronize at the wrong mouth pose or speech boundary, and
duplicate responses can be generated if the old and new workers both believe they own the turn.
Instead, you want a controlled ownership transfer: one worker remains authoritative until the new worker has enough state to continue seamlessly, and the media path is switched only when the new worker is ready.
Designing the handoff: snapshot, transfer, resume
A practical migration flow looks like this:
Snapshot the session state from the current worker.
Transfer that snapshot to a warm worker and mark it as the candidate owner.
Resume by reattaching or proxying the live media session, then flipping ownership.
The important part is to define the snapshot boundary carefully. The snapshot should include anything needed to reproduce the current conversational and avatar state, but not transient transport internals that cannot be safely replayed. A useful rule is:
Persist: transcript, turn index, conversation memory, agent settings, avatar/voice selection, pending tool state, and session IDs.
Do not persist: raw socket objects, in-memory encoder state, ephemeral DTLS/ICE details, and worker-local file handles.
If your agent supports tools or function calls, you also need idempotency. A worker may receive a user utterance, begin a tool call, and then get migrated before the result is delivered. The new worker must know whether that call already happened. The safest pattern is to assign a monotonically increasing turn or event sequence number and require the resume worker to ignore stale events.
Media continuity: what to reattach and what to renegotiate
For realtime avatars, the hardest part is usually the media plane. A browser or client is maintaining a live audio/video session, and the avatar’s face is synchronized to the agent’s speech output. If you move processing to a new worker, you have two broad options:
Proxy the live session through a stable session endpoint, so the client connection does not change even if the backing worker does.
Renegotiate on the new worker, which is simpler architecturally but more likely to create a visible or audible interruption.
For user-facing avatars, proxying or otherwise preserving the session endpoint is usually the better choice. The client keeps talking to the same logical session, while the backend ownership changes behind the scenes. If renegotiation is unavoidable, keep the old worker alive long enough to drain buffered audio and complete the current video frame sequence, then switch on a turn boundary rather than mid-utterance.
Also pay attention to state that is often forgotten: lip-sync timing. If the new worker starts generating speech output while the avatar renderer believes it is still in a previous phoneme window, the face will appear out of sync for a few hundred milliseconds. Carry the playback cursor, speech boundary markers, or equivalent timing metadata in your session snapshot if your pipeline exposes them.
Concurrency, ownership, and failure handling
Warm pool migration is mostly a distributed systems problem dressed up as media handling. You need a clear ownership model:
Exactly one worker is authoritative for a session at any instant.
Ownership transfers are atomic from the perspective of turn processing.
Every event is deduplicated using session IDs and sequence numbers.
That means your migration protocol should have explicit phases: pending-migration, handoff-ready, and active, for example. If the new worker fails to initialize, you should be able to fall back to the old worker without losing the user’s place in the conversation. If the old worker dies early, the new worker should reconstruct from the last committed snapshot and resume from the last acknowledged turn.
Two practical gotchas:
Do not snapshot on every token. Commit state at meaningful boundaries, such as end-of-turn or after a tool call completes. Token-level snapshots add noise and make recovery ambiguous.
Separate committed state from speculative state. The assistant may be halfway through generating a response when migration starts. Persist what the user has already heard or what the system has already accepted, not the speculative partial decode unless your replay logic can handle it cleanly.
Implementation sketch with a Python SDK
A straightforward way to model this is to store a serialized session blob in your own backend or cache, then have a warm worker reconstruct from that blob. The exact Protoface fields depend on the API surface, but the pattern is consistent:
The point is not the exact method names; check the docs for the real schema. The pattern is what matters: create a durable session record, store the minimum necessary state externally, and let a warm worker rehydrate from that record rather than reconstructing from scratch.
How this looks in a LiveKit agent
If you are embedding an avatar into a voice agent, the migration boundary often sits inside the agent process rather than in a frontend app. That is where the Pipecat integration and the LiveKit plugin-style architecture become relevant: the avatar renderer is one component in a larger media graph, and you want the graph to survive worker churn without losing the conversation context.
In practice, the agent should emit a compact session snapshot whenever a turn is committed. When a warm worker comes online, it receives the snapshot, reattaches to the same logical session, and continues generating both speech and video from the last committed turn. If you are using LiveKit Agents with the Protoface plugin, treat the avatar component as stateful middleware, not as a disposable UI effect.
If you want to see the plugin shape and examples, the relevant repository is the quickest reference. For protocol-level details, the docs are the source of truth.
Operational guardrails that save you later
A few things are worth building from the start:
Heartbeat and lease expiry: let workers periodically renew session ownership so abandoned sessions can be reclaimed.
Snapshot versioning: include a schema version in every serialized session so you can evolve state without breaking active calls.
Replay-safe events: make all state transitions idempotent, especially tool invocations and billing events.
Drain mode: allow a worker to stop accepting new sessions while it finishes active ones, which reduces mid-turn migrations.
These are boring pieces of infrastructure, but they are what keep warm pools from turning into a source of intermittent, hard-to-reproduce media bugs.
Conclusion
Preserving session state during warm pool migration is mostly about respecting the boundary between durable conversation state and transient media state. Snapshot the former, re-establish the latter carefully, and use explicit ownership transfer so only one worker speaks for the session at a time. If you do that, warm pools buy you lower latency without sacrificing continuity.
For implementation details, examples, and the exact session fields available in the API and SDK, start with https://docs.protoface.com and the relevant quickstart or integration repo for your stack. If you are wiring this into a LiveKit-based voice agent, the plugin examples are the most direct place to see the pattern end to end.