Introduction

When you add a conversational avatar to an ElevenLabs voice agent, you usually get audio working first and observability later. That’s the wrong order if the agent is going into production. Once the avatar is live, you need to answer basic questions quickly: did the session start, how long did it run, did the model stream audio smoothly, did the video face keep up with the voice, and which failures belong to your app versus the avatar pipeline?

This post shows a practical way to add logging and metrics around an ElevenLabs conversational avatar agent in FastAPI. By the end, you should be able to instrument session lifecycle events, correlate requests with realtime sessions, measure latency and failure rates, and expose the right data for debugging and alerting without turning your codebase into a telemetry project.

What you should measure in a realtime avatar pipeline

A conversational avatar agent is not a single request/response flow. It is a stateful, realtime pipeline that usually includes:

• an HTTP request that starts or configures a session,

• a WebRTC or similar realtime transport for media,

• a voice stack that turns text into audio and audio into speech events,

• a rendering or sync layer that keeps the face aligned with the spoken audio,

• and your application logic, which may include tools, handoffs, or business rules.

Logging every line of code is not useful. Logging the wrong layer is worse. The goal is to capture enough information to reconstruct a session and identify which component caused a problem, while keeping logs structured and bounded.

Use structured logs with a stable correlation ID

The first thing to add is a request/session correlation ID that survives from the initial FastAPI request into the realtime session. If you already have a conversation ID in your database, use that. If not, generate one at the edge and pass it through every log line.

For HTTP endpoints, log at least:

• request ID

• session ID

• user ID or tenant ID, if applicable

• avatar or agent configuration version

• start/end timestamps

• result status and error class

Prefer JSON logs. They are easier to search, aggregate, and join with metrics. Avoid dumping entire prompts, transcript blobs, or raw audio payloads into logs unless you are explicitly in a debug environment and have a data retention policy for it.

import logging

raise

import logging

raise

import logging

raise

Two practical details matter here. First, emit the same correlation ID in every component that can see the session, including callbacks, background tasks, and any webhook handlers. Second, keep the log schema stable. If your dashboard queries expect session_id, don’t rename it to conversation_id next week.

Instrument the realtime lifecycle, not just the HTTP boundary

The important failures usually happen after the initial POST succeeds. In a voice-agent flow, that means you need lifecycle events for connect, media start, first audio packet, first transcript, interruption, disconnect, and error. You do not need a metric for every minor internal state transition. You do need enough events to compute latency and availability.

A simple event model looks like this:

• session_started: your app accepted the request and began setup

• media_connected: the realtime transport is up

• first_audio_out: the agent produced the first outbound audio frame

• first_avatar_frame: the avatar video is actually rendering/synced

• session_ended: normal stop, timeout, or user disconnect

• session_failed: setup or runtime failure

These events let you compute useful latency metrics such as:

• time to connect

• time to first audio

• time to first visible avatar response

• session duration

• failure rate by stage

For FastAPI, middleware is a reasonable place to measure HTTP latency, but not the whole story. Use endpoint-level or background-task instrumentation for session lifecycle, because realtime work outlives the request that created it.

import time

session_latency.labels(stage=stage).observe(time.time() - started_at)

import time

session_latency.labels(stage=stage).observe(time.time() - started_at)

import time

session_latency.labels(stage=stage).observe(time.time() - started_at)

In production, the most useful labels are the ones that help you slice by deployment and agent version. Keep label cardinality low. Good candidates are environment, model/provider version, avatar version, and tenant tier. Bad candidates are user IDs, session IDs, and prompt hashes.

Correlate voice, video, and application errors separately

Developers often flatten every failure into “agent error,” which makes debugging harder. A better approach is to classify failures by origin:

• application errors: invalid config, auth failure, bad request payload

• transport errors: websocket/WebRTC disconnect, ICE failure, network timeout

• voice errors: TTS failure, STT failure, provider timeout, audio underrun

• avatar sync errors: video stalled, lip-sync lag, frame generation failure

This distinction matters because the remediation is different for each class. A transport failure may be a client network issue. A voice error may be an upstream provider outage. An avatar sync problem may be a rendering or buffering issue in your own pipeline.

In logs, emit both a human-readable event name and machine-parsable fields. For example: error_type=transport, stage=connect, provider=elevenlabs, retryable=true. This makes it much easier to build dashboards and alerts that reflect actual operational behavior instead of generic exceptions.

Expose metrics that answer production questions

If you only export counters, you will know that something happened, but not whether users felt it. For realtime agents, histograms and gauges are usually more useful than raw counters alone.

Useful metrics include:

• session starts per minute

• active sessions gauge

• connect latency histogram

• first-audio latency histogram

• session duration histogram

• failure counters by stage and provider

• disconnect reason counts

For a conversational avatar, “first audio” is often the more important SLA than total response time. Users are forgiving if the agent takes a second to think, but not if the call feels dead. If your avatar video is supposed to lead the audio, then measure that too. If it is supposed to lag slightly behind to preserve lip sync, measure whether the lag stays within your target window.

Keep alerting simple. A good first set of alerts is:

1. failure rate above a threshold for the last 5–10 minutes

2. p95 connect latency above baseline

3. active sessions dropping to zero when traffic is expected

4. provider-specific error spikes

Avoid alerting on every per-session error. Alert on aggregate behavior that indicates user impact.

FastAPI implementation pattern that stays maintainable

The easiest way to keep observability code from spreading is to centralize it in a small session service layer. Your FastAPI route should validate input, create a context object, and delegate to a service that handles session creation, logging, and metric emission. That gives you one place to add lifecycle hooks later.

from dataclasses import dataclass

raise

from dataclasses import dataclass

raise

from dataclasses import dataclass

raise

If you have callbacks or webhooks from the voice layer, treat them as first-class observability inputs. They are often the only place you can see when the remote provider decided to disconnect or when a stream ended unexpectedly. Also, make sure webhook handlers are idempotent. Replayed events should not double-count your metrics.

Where Protoface fits in

If your stack already uses a LiveKit voice agent, the ElevenLabs Agents quickstart is a good reference point for how to attach a synchronized avatar to the agent and where to hook your logs and metrics. At a high level, the integration point is the same one you would use for any other realtime dependency: session creation, transport events, and teardown.

Protoface’s REST API and Python SDK are also useful when you want your FastAPI service to create and track avatar sessions directly rather than burying that logic inside the client. The exact request and response fields depend on the endpoint, so use the docs as the source of truth. A minimal request might look conceptually like this:

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","metadata":{"request_id":"req_456"}}'

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","metadata":{"request_id":"req_456"}}'

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","metadata":{"request_id":"req_456"}}'

In practice, the important part for observability is not the exact payload shape. It is that you carry your own request/session identifiers through the avatar session so you can join app logs, provider events, and metrics later.

Conclusion

Building good logging and metrics for a realtime avatar agent is mostly about discipline: use structured logs, propagate a correlation ID, instrument the realtime lifecycle instead of only HTTP, and separate transport, voice, and avatar-sync failures. Once you can answer “where did this session fail?” and “how long until the user saw and heard the agent?” you have enough observability to operate the system with confidence.

If you are implementing this now, start with one endpoint, one session ID, and three metrics: session starts, failures by stage, and first-audio latency. Then expand from there. For API details and the available integration surfaces, check docs.protoface.com.