Introduction

When an avatar is driven by a realtime voice agent, there are really two state machines in play:

• the UI state your app thinks the avatar is in, and

• the session state the realtime system is actually in.

Those two drift apart more often than people expect. A frontend may still show “connected” after the underlying session has already ended. A backend may think it has an active avatar session, while the browser iframe has been refreshed and lost its transport. Or the agent may be speaking, but the UI is still rendering the previous “listening” state because it never received the latest event.

This post is about reducing that drift. By the end, you should be able to design avatar flows that keep the UI honest, recover cleanly from disconnects, and treat the realtime session as the source of truth instead of a decorative side channel.

Start with a single source of truth

The first mistake is to let UI state evolve independently from session state. In realtime systems, that almost always leads to stale indicators and awkward edge cases. The better model is:

1. the session is authoritative for connection and lifecycle state,

2. the UI renders a projection of that state, and

3. local UI state exists only for ephemeral concerns like hover, drag handles, or in-progress animations.

For an avatar app, the session usually has a small set of canonical states: created, connecting, connected, active/speaking, idle/listening, interrupted, ended, failed. The exact names vary, but the rule is the same: the UI should not invent new lifecycle states that the backend cannot confirm.

A practical pattern is to model the frontend as an event consumer. You receive session events, update a local store, and derive the visible UI from that store. If the transport drops, you mark the state as uncertain rather than pretending it is still live.

Use event-driven state, not optimistic assumptions

Realtime avatar sessions are typically long-lived WebRTC or streaming connections. They can disconnect for reasons outside your control: tab suspension, network changes, server-side timeouts, expired auth, or the agent process restarting. Because of that, optimistic UI is useful only for short-lived transitions. For anything lifecycle-related, wait for confirmation.

Two rules help a lot:

• Only transition to “connected” after the session handshake actually completes.

• Only transition to “ended” after you receive a terminal event or explicit cleanup confirmation.

That avoids the common bug where the UI flips to connected when a create request returns 200, even though media setup still has to complete. The create call means the session exists, not that the avatar is ready to talk.

If you are consuming events from a backend, keep the event payloads small and idempotent. A reconnect may replay state, and your reducer should be safe if it sees the same session snapshot twice.

Design for reconciliation after reconnect

Drift usually shows up after a reconnect. The user refreshes the page, the browser restarts a peer connection, or the worker process reconnects to the agent. When that happens, do not try to infer state from the last UI frame. Re-fetch or resubscribe to the authoritative session state.

A good reconciliation loop looks like this:

1. Persist the session identifier outside the live transport.

2. On reconnect, fetch the current session snapshot from the backend.

3. Merge it into local state using the session timestamp or version if available.

4. Reset any transient UI indicators that were tied to the lost transport.

For example, if the avatar was speaking when the socket died, the UI should not continue animating the mouth indefinitely. Instead, it should show “reconnecting” or “syncing” until a fresh state arrives. That may feel less polished in the moment, but it is much better than presenting false certainty.

This matters even more if you allow multiple surfaces to observe the same session. A dashboard, a live preview, and a customer-facing embed can all become inconsistent if each maintains its own guess about what the avatar is doing.

Keep transport state separate from domain state

A lot of state bugs come from collapsing everything into one enum. Connection status, agent intent, audio playback, and speaking activity are not the same thing.

At minimum, separate these dimensions:

• Transport state: disconnected, connecting, connected, reconnecting, failed.

• Session lifecycle: created, active, ended.

• Interaction state: idle, listening, speaking, interrupted.

This separation makes your UI easier to reason about. For example, “reconnecting” is a transport concern, not an avatar emotion. “Speaking” is a media state, not a lifecycle state. If you mix them, every reconnect becomes a special case.

It also makes your tests better. You can unit test reducers and state transitions without spinning up a WebRTC stack, and then separately test transport recovery behavior.

Handle race conditions explicitly

Realtime systems are full of races:

• The user clicks “stop” while a start request is still in flight.

• The browser receives an old event after the session has already ended.

• The agent emits a speaking update just as the transport disconnects.

• The frontend remounts and replays an outdated snapshot from cache.

The cure is not to avoid races; it is to make them harmless. Use a session-scoped monotonic counter, server timestamp, or revision field if your API provides one. If it does not, at least gate state updates by session ID and ignore messages that belong to a prior session.

When you send commands to the backend, make them idempotent where possible. “End session” should be safe to retry. “Create session” should either be deduplicated or clearly return a new session ID every time, so the UI can tell the difference between a retry and a new live session.

One practical trick: keep the current session ID in a store that outlives component mounts, but tie every transport listener to that ID. When the ID changes, tear down old listeners before attaching new ones. That prevents stale callbacks from mutating current UI state.

Example: syncing a session snapshot in Python

If you are driving sessions from a backend, the safest pattern is to create or fetch the session on the server, then expose only the minimal state the frontend needs. The exact fields depend on the API, but the shape is usually straightforward.

from protoface import Client  # illustrative; see docs for exact SDK surface

}

from protoface import Client  # illustrative; see docs for exact SDK surface

}

from protoface import Client  # illustrative; see docs for exact SDK surface

}

The important part is not the exact SDK call; it is the flow. Create on the backend, persist the session ID, and refresh from the authoritative source whenever the UI might be stale.

Example: consuming session lifecycle events in a LiveKit agent

If your avatar is attached to a LiveKit voice agent, the plugin should be treated as a media adapter, not as your state store. The agent can emit speaking and connection signals, but your app should still own the canonical session record. The plugin repo has examples worth skimming: https://github.com/protoface-ai/protoface-plugin-pipecat.

# Illustrative only; exact names may differ by plugin version.

ui_store.update({"transport_state": state})

# Illustrative only; exact names may differ by plugin version.

ui_store.update({"transport_state": state})

# Illustrative only; exact names may differ by plugin version.

ui_store.update({"transport_state": state})

The point here is to bind UI updates to explicit events, not to implicit assumptions about whether audio is “probably still playing.” If the event stream is interrupted, switch the UI to a reconnecting state and rehydrate from the backend.

Where Protoface fits

This is exactly the sort of problem Protoface is meant to make manageable: you create and manage avatars and realtime sessions through the REST API or Python SDK, then attach them to your agent or embed. The useful part for drift reduction is that the session is an addressable backend object, not just an in-memory browser widget. That gives you something concrete to reconcile against after reconnects, refreshes, or agent restarts. The public docs at https://docs.protoface.com are the right place to confirm the exact fields and lifecycle events for your integration.

If you are already using a voice-agent stack, the LiveKit plugin path is usually the cleanest place to wire this in because it keeps the media path and the avatar state close together, while still letting your application own the session record.

Operational safeguards that pay off

Once the basics are in place, a few guardrails make a big difference:

• Heartbeat or freshness checks: if you have not heard from the session for a while, mark the UI uncertain.

• Explicit cleanup: end sessions on tab close, route change, or task completion when appropriate.

• Server-side logging: log session ID, connection transitions, and terminal reasons so you can trace drift after the fact.

• Versioned updates: prefer ordered events or snapshot-plus-delta patterns over ad hoc booleans.

In practice, most “avatar stuck speaking” bugs are not media bugs at all. They are stale state bugs caused by missing reconciliation. Instrumenting the lifecycle usually fixes more than touching the rendering code.

Conclusion

Reducing drift is mostly about discipline: make the session authoritative, separate transport from domain state, and reconcile after any reconnection or reload. If you do that, the UI becomes a faithful view of the realtime system instead of an optimistic guess.

For implementation details, examples, and current API shapes, start with the docs and the relevant SDK or plugin repository. Then build your frontend around session IDs, explicit lifecycle events, and replay-safe state updates. That will save you a lot of time the first time a browser tab refreshes in the middle of a live avatar conversation.