Introduction

When you add a realtime avatar to a voice agent, you usually end up with two independent event streams:

• WebSocket events from your browser or backend session control plane.

• Agent events from the LLM/voice stack, such as ElevenLabs Agent lifecycle, transcript, tool, and audio timing events.

The hard part is not receiving both streams. The hard part is correlating them reliably enough to answer questions like: “Which avatar session belongs to this agent run?”, “Did the avatar start speaking before the agent emitted audio?”, or “Which user turn caused this lip-sync spike?”

By the end of this post, you should be able to design a correlation layer that:

• assigns stable IDs across browser, backend, and agent processes,

• merges WebSocket and agent events into one timeline,

• handles retries, reconnects, and duplicate events without corrupting state, and

• keeps your realtime avatar app debuggable when timing gets messy.

Start with a shared correlation model

The first mistake developers make is treating “connection,” “session,” “conversation,” and “agent run” as interchangeable. They are not. If you want deterministic debugging, you need a small set of IDs with explicit scope.

A practical model looks like this:

• avatar_session_id: the Protoface realtime avatar session. This is the thing your UI connects to over WebSocket/WebRTC semantics, depending on the surface you use.

• agent_run_id: one execution of your voice agent. For ElevenLabs, this is the lifecycle of a live agent conversation or turn sequence.

• conversation_id: the user-facing conversation thread, which may survive reconnects and multiple agent runs.

• event_id: unique identifier for an event payload so you can dedupe retries.

• correlation_id: your own join key, propagated through every layer if possible.

Do not rely on timestamps alone. They are useful for ordering, but not for identity. WebSocket delivery can be delayed, agent events can arrive out of sequence, and different components may timestamp in different clocks or time zones. Use IDs for identity and timestamps for ordering.

Use a join key that survives the whole request path

In practice, the easiest strategy is to mint a correlation_id at the first trust boundary you control, then thread it through:

1. the browser when you open the avatar session,

2. your backend when you create or attach the session,

3. the agent runtime when the ElevenLabs conversation starts, and

4. every event you emit into your logs or analytics pipeline.

If your agent stack supports metadata, put the join key there. If it does not, keep a server-side mapping table from conversation_id to correlation_id. The important part is that correlation happens at the edges, not later in a log scraper.

import uuid

import uuid

import uuid

A useful rule: the browser should never invent identity that the backend cannot verify. Let the backend mint or authorize the avatar session, then hand the browser a short-lived session token or embed URL. This keeps your correlation layer aligned with your security boundary.

Normalize both streams into one event envelope

Once the IDs are in place, normalize everything into a common envelope before doing any downstream processing. Whether the event came from a WebSocket message or an ElevenLabs Agent callback, represent it the same way in your app.

{
}

{
}

{
}

That envelope gives you three big wins:

• Deduplication: retrying a WebSocket message should not replay the same state transition twice.

• Ordering: you can sort by a monotonic ingest timestamp inside a single process, while preserving original source timestamps for debugging.

• Joinability: metrics, logs, traces, and user-facing playback can all be keyed off the same structure.

For realtime avatar apps, the event types that matter most are usually:

• session opened / closed,

• user speech started / stopped,

• transcript partial / final,

• agent turn started / completed,

• audio chunk queued / played,

• avatar speaking state changes, and

• error / retry / reconnect.

Not every source emits all of these. That is fine. The point is to map source-specific events into a smaller app-specific vocabulary.

Handle the ugly parts: duplication, reordering, and reconnects

Realtime systems fail in boring ways. The browser reconnects. The agent restarts. An upstream service retries a webhook-like callback. The avatar session is still alive, but the client thinks it is new. Correlation only works if you design for these cases up front.

Deduplicate at the ingestion boundary

Never assume a message is exactly-once. Use event_id when the source provides one. If it does not, build a deterministic fingerprint from the source, type, source timestamp, and stable payload fields. Store processed fingerprints in a short TTL cache or durable store depending on how expensive duplicate processing is.

Treat reconnects as continuation, not replacement

A reconnect should usually preserve the same conversation_id and correlation_id, while getting a new transport-level connection ID. If you create a brand-new app session on every reconnect, your timeline becomes fragmented and your metrics become misleading.

A good pattern is:

• conversation_id remains stable across a user interaction.

• agent_run_id changes when the voice agent is restarted or a new turn begins, depending on your architecture.

• transport_connection_id changes on every WebSocket reconnect.

Use state machines, not ad hoc booleans

Avatar apps often go sideways because developers track “isSpeaking” and “isConnected” as independent flags. That works until events arrive out of order. Prefer a small explicit state machine with allowed transitions. For example:

• idle → listening → thinking → speaking → idle

Then have both event streams drive the same state machine. A WebSocket “audio started” event and an agent “response_audio_ready” event may arrive in different orders, but both should converge on the same state after validation. If a transition is invalid, log it with the full envelope and do not silently accept it.

Make timestamps useful, not authoritative

For debugging, keep three time concepts separate:

• source_ts: the timestamp from the emitting system.

• ingest_ts: when your service received the event.

• render_ts: when the browser actually applied the state change, if applicable.

That distinction matters because the thing you are trying to understand in a lip-sync app is often not “what happened?” but “where did latency accumulate?” If the agent emitted audio quickly but the avatar started speaking late, your bottleneck is likely in transport or render, not generation.

Python-side correlation example

If you are orchestrating the agent and the avatar session from Python, keep the join key in a single context object and attach it to every event you log or persist. The exact Protoface session fields will depend on the docs, but the pattern stays the same.

from dataclasses import dataclass, field

from dataclasses import dataclass, field

from dataclasses import dataclass, field

In a real app, feed those envelopes into a queue, a DB table, or your tracing system. The point is that your browser events and agent events end up as rows in the same conceptual timeline.

Where Protoface fits

This is exactly the kind of problem a developer-facing avatar layer should make easier. With Protoface, you can keep the avatar session creation and management on the backend via the REST API or Python SDK, while your frontend only deals with the session it has been authorized to use. That separation makes it much easier to assign a stable correlation_id on the server and attach it to both the avatar session and the ElevenLabs agent run.

If you are using the LiveKit agent path, the ElevenLabs Agents quickstart is the right place to see how the avatar layer is inserted into the voice stack. The integration pattern is the same: initialize the agent, create or attach the avatar session, propagate your join key, then log every event in one place.

For example, a backend can create a session, return a short-lived handle to the browser, and keep the authoritative mapping server-side:

import os

import os

import os

And if you are managing avatars or sessions programmatically, the Python SDK is the cleanest place to centralize that logic; see the docs for the exact request and response shapes.

Practical debugging checklist

When events do not line up, work through the problem in this order:

1. Confirm the same correlation_id appears in browser, backend, and agent logs.

2. Check whether a reconnect created a new transport connection without changing the conversation identity.

3. Look for duplicate event fingerprints before assuming a state machine bug.

4. Compare source timestamps to ingest timestamps to isolate network delay.

5. Verify that the avatar session and agent run are attached to the same conversation thread.

If you cannot answer those five questions quickly, the event model is too loose.

Conclusion

Correlating WebSocket events and ElevenLabs Agent events is mostly an architecture problem, not a streaming problem. Keep a stable join key, normalize events into one envelope, dedupe aggressively, and model state transitions explicitly. That gives you a realtime avatar app that you can actually debug when latency or reconnect behavior gets weird.

If you are wiring this up now, start by defining your correlation model in the backend, then map both event streams into a single timeline. The public docs at docs.protoface.com cover the exact Protoface session and SDK shapes, and the quickstart repos show the integration patterns in runnable form.