Introduction

If you’re adding a realtime avatar to a voice agent, latency is not one number; it is a chain. Audio has to be captured, encoded, transported, transcribed or inferred, turned into speech, synchronized to video mouth movement, and rendered in the browser. If any hop adds jitter or buffering, the avatar stops feeling live even if the underlying model is “fast enough.”

This post shows a practical way to measure codec impact on end-to-end avatar latency in Python and TypeScript. By the end, you should be able to isolate the cost of audio/video encoding choices, distinguish transport delay from model delay, and build a measurement harness that gives you actionable numbers instead of vague “it feels slow” feedback.

What you should measure first

When developers talk about avatar latency, they often collapse several different delays into one bucket. That makes optimization hard. Start by separating the path into stages:

• Capture latency: time from microphone or synthesized audio availability to the first encoded packet.

• Codec latency: time spent encoding/decoding audio and video frames.

• Network latency: RTT, packet loss, jitter, and buffering between client and server.

• Agent latency: ASR, LLM, TTS, and avatar synthesis/animation delay.

• Render latency: time until the browser actually paints the next visible frame.

For codec work, you want to measure the first, second, and last stages directly. If you only time “user spoke” to “avatar moved,” you’ll conflate transport, model latency, and browser scheduling with the codec under test.

A useful metric set looks like this:

• First packet latency: timestamp when the first encoded audio/video packet leaves the client.

• First decoded frame latency: timestamp when the receiver can decode the first frame.

• Time to visible motion: timestamp when the avatar visibly reacts in the browser.

• Steady-state jitter: variance of frame spacing after the stream is warm.

• Bitrate at target quality: actual bandwidth cost for a given subjective quality tier.

Build a repeatable measurement harness

The main requirement for meaningful codec comparison is repeatability. Keep the media source fixed, record timestamps at each boundary, and avoid human interaction in the loop.

For audio, use a deterministic sample, such as a short WAV file or a generated tone sequence. For video, drive a fixed set of avatar frames or a known animation source. Your goal is not to test the model; it is to test what the codec does to the transport path.

In practice, you’ll want to compare at least these conditions:

• opus vs. raw PCM for audio transport

• different sample rates and frame sizes

• H.264 profile/bitrate settings where applicable

• browser-side buffering thresholds

Keep the network constant if possible. If you can, run the server and client in the same region and test over a local or low-jitter path first. Once you understand the codec baseline, re-run over a real WAN connection to see how much of the result is actually network variance.

The simplest measurement pattern is to stamp every boundary with a monotonic clock and emit structured logs. In Python, that usually means time.perf_counter_ns(); in TypeScript, use performance.now() or process.hrtime.bigint() on the server side. Don’t mix wall-clock timestamps with monotonic timing if you care about sub-100 ms differences.

Python: instrumenting the client path

If your avatar pipeline is driven from Python, instrument the point where media is handed off to the transport and the point where you receive the first remote response. The exact integration will depend on your stack, but the pattern is the same.

import time

print({"event": "first_packet_queued_ms", "value": (t2 - t0) / 1e6})

import time

print({"event": "first_packet_queued_ms", "value": (t2 - t0) / 1e6})

import time

print({"event": "first_packet_queued_ms", "value": (t2 - t0) / 1e6})

What matters is not the SDK call itself, but where the call sits relative to your codec boundary. If your encoder buffers 20–60 ms of audio before it emits anything, you should see that immediately in the delta between “frame captured” and “first packet queued.”

For comparisons, keep one variable at a time:

• same audio source, different codec

• same codec, different packetization interval

• same settings, different bitrate caps

When you plot results, look at both median and tail latency. A codec that is 10 ms faster on average but occasionally spikes by 100 ms may feel worse than a slightly slower but stable option, especially for lip-synced avatars where bursty delay creates visible desynchronization.

TypeScript: browser-side and server-side timing

TypeScript is useful when you’re measuring from the browser, because the browser is where the avatar is finally perceived. The transport may look good from the backend, but a decoder queue or render stall can still add a visible delay.

const t0 = performance.now();

}

const t0 = performance.now();

}

const t0 = performance.now();

}

In the browser, instrument three points if you can:

1. message arrival or first packet receipt

2. decoder output

3. first paint or animation frame that reflects the new state

That lets you tell whether the codec is slowing down decode or whether your UI is just painting late. For video avatars, the latter is easy to miss. A WebRTC track can be decoded in time but still appear late if your render loop batches DOM updates, misses a frame, or waits on layout.

If you are comparing browser codecs, use the same test harness with the same media source and the same tab state. Browser background throttling, autoplay restrictions, and tab visibility all affect timing. Keep the tab active and disable unrelated extensions while measuring.

How codec choice shows up in realtime avatars

For conversational avatars, codec decisions usually matter in two places. First, they affect how quickly audio reaches the agent or client. Second, they affect how much buffer the receiver needs before it can safely decode and render.

Audio codecs such as Opus are generally a good fit for realtime conversation because they balance bitrate and delay well at low packet sizes. But you still pay for encode/decode, packetization, and jitter buffering. Smaller frames reduce algorithmic delay but increase overhead and can make you more sensitive to packet loss. Larger frames improve compression efficiency but increase startup latency.

Video has similar trade-offs. If your avatar stream is high-motion or high-detail, you may need higher bitrate or a more efficient codec to avoid macroblocking. But if your main goal is lip sync and expressive facial motion, pushing too much visual fidelity can be counterproductive if it adds delay. A slightly softer image that updates on time often feels better than a sharper image that lags behind speech.

Practical gotchas:

• Buffering hides codec latency: a player that buffers heavily can make the network look stable while increasing startup time.

• Jitter buffers are adaptive: they can change during the session, so measure both cold start and steady state.

• Model latency can mask codec wins: if TTS takes 300 ms, shaving 15 ms from encoding may not move the user-visible number much until you also optimize the agent path.

• Latency and sync are different: a stream can be low-latency but visually off if audio/video clocks drift or frames are dropped.

Using Protoface without losing visibility into the path

Protoface is useful here because it gives you a developer-facing realtime avatar surface while keeping the integration points familiar: a REST API, a Python SDK, and a LiveKit plugin for voice agents. For codec work, that means you can treat the avatar as part of your realtime stack and instrument the same boundaries you would for any other media service.

For example, if you are attaching an avatar to a LiveKit voice agent, the plugin path is usually the fastest way to test whether a particular media configuration changes the perceived time to first face motion. The plugin lives in the Python ecosystem, so you can keep your timing code next to the agent and log the deltas around session setup, audio handoff, and first visible response.

There is also a straightforward REST path for controlled experiments. You can create and manage sessions from a backend, then compare different client codec settings against the same avatar/session configuration. Exact request fields vary, so use the docs for the current schema, but the shape looks like this:

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

The point of using a managed avatar surface is not to avoid measurement; it is to make measurement easier. You still want timestamps at capture, encode, transport, decode, and render. But once the avatar session is a stable dependency, you can vary codecs and network conditions without rebuilding the whole system each time.

Analysis strategy that actually helps

Once you have logs, do not stop at averages. Group results by codec, packet size, and network condition, then compare distributions. A few practical rules:

• Use median for the “typical” case.

• Use p95 or p99 for tail behavior.

• Compare startup separately from steady state.

• Correlate spikes with packet loss, retransmits, or GC pauses.

If you see a codec that lowers bandwidth but increases p95 startup latency, ask whether that trade-off is acceptable for your product. In a support bot, a 20 ms increase may be fine if it reduces infra cost. In a live sales demo or game NPC, visible delay may be a worse trade than extra bandwidth.

Also measure in the context of the full agent stack. A good test matrix looks like this:

1. codec only, local loopback or same-region test

2. codec plus network, real WAN conditions

3. codec plus agent, including ASR/TTS/model response time

4. codec plus browser render, which is what users actually feel

Conclusion

Measuring codec impact on realtime avatar latency is mostly about discipline: isolate one layer at a time, timestamp every boundary with monotonic clocks, and compare distributions instead of anecdotes. Once you do that, you can tell whether your bottleneck is encode/decode cost, buffering, transport, or the agent itself.

If you want a practical starting point, use your existing Python or TypeScript stack, wire in a deterministic media source, and log the timestamps around session setup, first packet emission, first frame receipt, and first visible render. Then compare codecs under the same network conditions and with the same avatar/session configuration.

For integration details, the docs at docs.protoface.com are the right place to look, and the relevant plugin or SDK repository will usually have the shortest path from measurement to a working harness.