Introduction

If you are building a realtime avatar app on top of ElevenLabs Agents, the hard part is usually not getting audio to play or video to render. The hard part is understanding what happened when the experience feels slow, broken, or inconsistent: did the agent stall on transcription, did the LLM take too long, did the avatar session fail to attach, or did WebRTC itself lose quality?

This post is about adding observability to that stack in a way that is actually useful in production. By the end, you should be able to trace a single user interaction across your agent runtime, your avatar session, and your transport layer; log the right identifiers; and build enough telemetry to debug latency, failures, and quality issues without guessing.

I’ll use Protoface as the avatar layer in the examples, because it provides a clean way to attach synchronized talking video to a live voice agent. The core ideas apply regardless of your exact video surface: you want correlation IDs, stage-level timings, and events that describe the lifecycle of each realtime session.

What to measure in a realtime avatar stack

Realtime systems are not like ordinary request/response APIs. One user turn can cross several async boundaries: microphone capture, VAD or STT, LLM inference, TTS, avatar synthesis or driving, and finally WebRTC transport to the browser. Observability needs to reflect that pipeline, not just the final user-visible result.

At minimum, log and aggregate these three categories:

• Lifecycle events: session created, avatar attached, stream connected, stream disconnected, agent turn started/ended.

• Latency breakdowns: time to first token, time to first audio, time to first video frame, end-to-end turn latency.

• Quality and failure signals: reconnects, ICE restarts, media packet loss/jitter if available, transport timeouts, 4xx/5xx API failures, and per-turn exceptions.

The useful metric is rarely “agent latency” in the abstract. It is usually “why did this user see a frozen face for 1.8 seconds after the model finished speaking?” That question requires timing both the agent and the avatar path.

Use a correlation ID for the entire interaction

The first thing to do is assign a stable interaction ID at the boundary of your application. In a browser app, that can be a server-issued conversation ID. In a backend agent, it can be the same value you thread through the agent state, your logs, and your metrics labels. If you create a new avatar session per conversation, use the same ID as the session reference in your app layer.

The important part is consistency. Your logs should answer: “For conversation X, what happened across all components?” If you do only one thing for observability, do that.

conversation_id = "conv_7f3b2c"
})

conversation_id = "conv_7f3b2c"
})

conversation_id = "conv_7f3b2c"
})

Then carry that ID into every downstream call where you can. If your agent framework lets you attach metadata to events, do it there. If not, include it in structured logs around those calls. The point is to make a single grep-able identifier visible everywhere.

Instrument the agent pipeline, not just the avatar

For ElevenLabs Agents, the avatar is only one stage in the path. You still need visibility into model and speech timings. A practical pattern is to wrap each stage with timing logs, then emit a summary event at turn completion.

from time import perf_counter

})

from time import perf_counter

})

from time import perf_counter

})

That may look basic, but it gives you a latency histogram per stage, which is more useful than a single “slow request” log. If you later add traces, these same stage boundaries become spans.

Also log the content-level facts that help explain behavior: the transcript length, whether the turn was interrupted, whether the user barged in, whether the agent fell back to a shorter response, and whether audio was regenerated. Those are often the difference between “infra bug” and “expected behavior under interruption.”

Track the avatar/session lifecycle separately

Avatar and WebRTC session events deserve their own timeline. A user may have a perfectly healthy agent turn while the video side is failing to attach, renegotiating, or running behind the audio. If you collapse all of that into one success/failure flag, you lose the ability to debug the actual system.

Useful session events include:

• avatar session created

• session attached to agent

• peer connection established

• first video frame rendered

• disconnect reason

• reconnect or ICE restart

If your frontend can report the first rendered frame, do that. “Connected” is not the same as “visually usable.” The delta between connection and first frame is one of the most valuable metrics in a realtime avatar app.

On the browser side, if you are using an iframe-based embed or a direct client, make sure you listen for the session-ready event and any error states that your integration exposes. The exact event names depend on the integration surface, but the general rule is the same: capture timestamps for each lifecycle transition and send them to your backend or analytics pipeline.

Protoface: use the session boundary as your observability anchor

The cleanest place to hook this up is usually where you create or attach the avatar session. With the REST API or Python SDK, you create a realtime session, store its ID alongside your conversation ID, and then log the moments that matter as the session progresses.

For example, a session creation call might look like this in curl form:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

And the same pattern in Python can be kept just as small:

from protoface import Client

})

from protoface import Client

})

from protoface import Client

})

The exact request/response fields are in the docs, but the observability pattern is the same: create the session once, store the session ID, and use it as the join key between application logs and avatar telemetry. If you need a broader reference for the API and SDK shape, start with the documentation at docs.protoface.com.

Make the logs answer debugging questions

Good logs are not just timestamps. They should answer the questions you will ask at 2 a.m. when a customer says the avatar is “slow.”

Here are the fields I would include in structured logs for each turn or session event:

• conversation_id: stable app-level identifier

• session_id: avatar or media session identifier

• stage: stt, llm, tts, attach, connect, first_frame

• duration_ms: timing for the stage

• status: success, timeout, error, interrupted

• error_code: normalized error bucket

• provider: elevenlabs, protoface, webrtc, browser

Once you have those, you can build practical dashboards: p50/p95 latency by stage, disconnect rate by browser, first-frame time by region, and error rate by avatar template or quality tier. For realtime systems, tail latency matters more than averages.

One subtle but important gotcha: avoid logging raw audio or transcripts unless you have a clear retention and privacy policy. Instead, log lengths, flags, hashes, and normalized error categories. You can always sample richer payloads in a controlled debug environment.

When to add traces, metrics, or both

If you are already running OpenTelemetry or a similar tracing stack, use it. Each user interaction should become a trace root, and each stage in the agent/video pipeline should become a child span. That gives you a timeline view for free, and it makes multi-service debugging much easier.

If you are not ready for distributed tracing yet, structured logs plus a handful of metrics is still a strong starting point. I would prioritize:

1. turn latency histogram

2. avatar first-frame latency histogram

3. disconnect/reconnect counter

4. API error counter by endpoint/status

5. active session gauge

The rule of thumb is simple: use logs for forensic detail, metrics for trends and alerting, and traces for end-to-end causality. In a realtime avatar app, you usually need all three eventually, but you do not need to boil the ocean on day one.

Practical trade-offs and gotchas

Two things tend to trip teams up.

First, WebRTC and media-session failures are often intermittent. A session can connect successfully and still have bad media quality. If you only alert on hard failures, you will miss the degradation mode that customers actually notice.

Second, your agent and avatar can fail independently. The model may continue generating while the video stream has dropped, or the browser may still show the last frame while audio is healthy. Keep separate status dimensions for each subsystem rather than one combined “online/offline” flag.

If you operate at scale, sample high-volume success logs, but do not sample away rare errors or disconnect reasons. Also make sure your correlation IDs are present in both sampled and unsampled paths, otherwise debugging gets awkward fast.

Conclusion

Observability for ElevenLabs Agents in a realtime avatar app is mostly about disciplined instrumentation: stable correlation IDs, stage-level timings, and separate lifecycle tracking for the agent and the media session. Once you have those, you can answer the questions that matter in production: where latency accumulates, where sessions fail, and whether the avatar layer or the agent layer is responsible.

Start small. Add structured logs around turn boundaries and avatar session events, then promote the recurring fields into metrics and traces. If you are using Protoface as the avatar layer, anchor your telemetry on the session ID and follow the examples in the documentation and the relevant quickstart repositories, including the ElevenLabs Agents quickstart. That will get you from “it seems slow” to actionable, stage-specific debugging very quickly.