Introduction

In a realtime voice agent, the hardest debugging problem is usually not “did the model answer?” It’s “what happened across the whole turn?” A single user utterance can fan out into speech-to-text (STT), LLM reasoning, text-to-speech (TTS), and avatar rendering, all running on different clocks and often on different services. When something feels off—late responses, cut-off audio, duplicated turns, lips out of sync, or a session that looks fine in one log but broken in another—you need a reliable way to correlate events across those layers.

This post shows a practical correlation strategy you can use in production: how to stitch together STT, TTS, and Agora session logs into one traceable conversation timeline. By the end, you should be able to answer questions like:

• Which STT result produced this response?

• Which TTS chunk belonged to that turn?

• What did the media session look like at the same time?

• Where did latency or failure enter the pipeline?

Use one turn ID everywhere

The core idea is simple: assign a single conversation-turn identifier at the edge of your agent pipeline, then propagate it through every hop. Don’t rely on timestamps alone. Clock skew, buffering, retries, and async fan-out will make timestamp-only correlation brittle. A turn ID gives you a stable join key.

At minimum, log these fields for every event you can influence:

• session_id: the voice session or call.

• turn_id: one user utterance and the agent response it triggers.

• event_type: STT final, LLM start, TTS start, media publish, etc.

• service: STT vendor, TTS vendor, Agora, agent runtime.

• timestamp_ms: server-side monotonic or UTC time.

• trace_id or request_id: if a provider returns one, store it too.

If you have control over the orchestration layer, generate the turn ID before you send audio to STT. Then attach that ID to downstream logs and metadata. In practice, this is the only stable way to correlate across asynchronous systems.

Define the event model before you debug

Most teams start by collecting logs from each system independently, then discover they cannot actually reconstruct the flow. Fix that by standardizing a minimal event schema. You do not need a giant tracing platform to begin with; you need consistent records.

A useful event shape looks like this:

{
}

{
}

{
}

Use the same pattern for TTS:

{
}

{
}

{
}

And for Agora session events:

{
}

{
}

{
}

Two important details:

• Log start and end events, not just success/failure. Duration is often the clue.

• Preserve raw provider IDs inside the payload. If you need to open a vendor ticket, those IDs matter.

Correlate by turn, then reconstruct the timeline

Once every event carries session_id and turn_id, correlation becomes a data problem instead of a forensic one. For each session, group all events by turn, sort by timestamp, and compute durations between adjacent stages.

That gives you a timeline like this:

1. User audio starts.

2. STT partials arrive.

3. STT final transcript is emitted.

4. LLM starts generating a response.

5. TTS starts for the first token or sentence.

6. Audio is published into the media session.

7. Avatar video is synchronized to the outgoing speech.

From that sequence, a few latency metrics become obvious:

• STT latency: first user audio to final transcript.

• Think time: final transcript to first LLM output.

• Time to first audio: final transcript to first TTS byte or chunk.

• Media egress delay: TTS chunk to published audio track.

This is also how you catch bugs that are invisible in one service’s logs. Example: STT may look healthy, and TTS may look healthy, but the gap between them might be growing because your agent is serializing too much work on the hot path. Or Agora logs might show audio publication is fine, while your application logs show you started TTS 800 ms late because you waited for a full sentence boundary.

Watch out for the three common failure modes

1. Partial STT churn. Realtime STT often emits multiple interim hypotheses before a final transcript. If you log all partials as separate turns, you will overcount. Treat partials as updates to the same turn_id, and only mark a transcript as final when the STT service says it is final.

2. TTS chunk boundaries. TTS may stream audio in chunks. A single agent response can therefore produce several media events. That is normal. The correlation key should be the turn ID, not a one-to-one mapping between response text and audio packets.

3. Media session offsets. Agora session logs tell you about channel participation, publishing, and connection state, but they do not automatically explain application-level intent. A healthy session can still carry the wrong audio if you published the wrong track or started playback before the agent had real text. Combine media logs with application logs, not instead of them.

Practical instrumentation in Python

If your orchestration is in Python, a lightweight logger around your agent pipeline is often enough. The key is to emit structured events at the handoff points between STT, LLM, TTS, and media.

import time

emit(session_id, turn_id, "tts.started", "elevenlabs", {"voice": "alloy"})

import time

emit(session_id, turn_id, "tts.started", "elevenlabs", {"voice": "alloy"})

import time

emit(session_id, turn_id, "tts.started", "elevenlabs", {"voice": "alloy"})

If you are using a voice-agent framework, put this logging in the callback or middleware layer rather than inside business logic. That keeps the correlation consistent even when you swap models or vendors.

How this maps onto Agora session logs

Agora gives you the media-plane view: who joined, when tracks were published, connection state, and often room-level events that help explain delivery issues. That is useful, but it is not enough by itself, because Agora is only seeing the media transport. Your application still needs to say which turn produced which media.

For debugging, I recommend joining logs in this order:

1. Filter by session_id to isolate one call or browser session.

2. Group application logs by turn_id.

3. Overlay Agora timestamps for publish, subscribe, reconnect, and track-state events.

4. Compute per-turn latency deltas and look for gaps.

This approach makes root-cause analysis much faster. If the media session reconnects in the middle of a turn, you will see a gap between TTS start and audio publication. If the network is fine but the agent is slow, you will see the gap earlier between STT final and TTS start. If the problem is lip-sync drift, compare the publish time of audio with the avatar render time, not just the overall session duration.

Where Protoface fits

In a setup that includes an avatar layer, the same turn ID should flow into the avatar session alongside your voice pipeline. That way the speaking face is correlated with the exact STT and TTS turn that drove it. Protoface is useful here because the avatar is part of the realtime session, not an afterthought, so you can keep the conversational timeline aligned across the agent, media transport, and visual output.

If you are integrating through the LiveKit Agents plugin, the same principle applies: attach your correlation metadata at the agent boundary and carry it through the turn lifecycle. The plugin examples in the GitHub repo are a good reference for where to hook logging around the media and speech events. For lower-level management of avatars and sessions, the docs are the right place to confirm the exact request and response fields for your setup.

A small but important operational note: if you expose sessions through embeds or API-created sessions, make sure your own application logs store the external session identifier returned by the platform. That ID is the bridge between your backend records and the realtime media logs.

Conclusion

Correlating STT, TTS, and Agora logs is mostly about discipline: generate one turn ID early, propagate it everywhere, and log structured start/end events at each boundary. Once you do that, the “why was this response late?” question becomes a timeline query instead of a guess.

If you are building or debugging a realtime voice agent, start by instrumenting the orchestration layer, then add media-plane correlation, then verify avatar sync against the same turn IDs. For implementation details, session management, and integration examples, see the documentation at docs.protoface.com.