Introduction

In realtime avatar systems, codec latency is often the difference between a conversation that feels natural and one that feels like a turn-based bot with a face. When you stream a voice agent into a talking avatar, the audio codec sits on the critical path: it adds framing delay, algorithmic lookahead, packetization overhead, and sometimes jitter-buffer growth on top of the network and model latency you already have.

This post is about reducing that codec-induced delay without breaking sync. By the end, you should be able to reason about where audio latency comes from, choose codec settings that fit realtime avatar streams, and avoid a few common mistakes that quietly add hundreds of milliseconds.

Where codec latency actually comes from

People often treat “codec latency” as a single number, but in practice it is the sum of several smaller delays:

• Frame duration: Most codecs operate on fixed-size frames, commonly 20 ms. If the encoder waits to collect a full frame before producing output, that is already 20 ms of buffering.

• Algorithmic delay: Some codecs need lookahead or internal analysis windows. That delay is built into the codec design and cannot be removed by tuning alone.

• Packetization delay: If you aggregate too many frames per packet to “save overhead,” you increase end-to-end latency.

• Jitter buffering: The receiver may intentionally hold packets to smooth network variance. A larger jitter buffer makes audio more stable, but less interactive.

• Resampling and format conversion: Mismatched sample rates, channel counts, or sample formats can force extra processing steps in the pipeline.

For avatar streams, there is a second constraint: the video mouth animation has to track the audio closely enough that users do not notice desync. If the audio path is slow, the avatar can either lag behind the user’s expectation or be forced to predict movement from incomplete audio. Both are bad.

The practical takeaway is simple: optimize the entire chain, not just the codec choice. A low-delay codec with a large frame size can still feel slow if your pipeline batches data or your transport buffers aggressively.

Use shorter frames, but understand the trade-off

The easiest lever is frame size. Smaller audio frames reduce the amount of audio that must be collected before encoding and sent onward. In realtime voice systems, 20 ms is a common baseline; 10 ms can feel more responsive when the rest of the pipeline is well tuned. The downside is overhead: more packets, more headers, more CPU wakeups, and more sensitivity to network jitter.

For most avatar use cases, the rule is:

• Start at 20 ms if you want a good balance of efficiency and responsiveness.

• Drop to 10 ms if latency is the primary goal and your network path is stable.

• Avoid batching multiple frames unless you have measured that packet overhead is the bottleneck.

If you are using an RTC stack, the codec frame size has to fit with the media engine’s packetization strategy. The transport may already be tuned for low-delay audio, so you get the most benefit by keeping the audio pipeline continuous: capture, encode, send. Any queue in the middle is a latency multiplier.

Prefer low-delay codecs and avoid unnecessary transcoding

The codec itself matters. A codec with a small or zero lookahead is naturally better suited to interactive voice than one optimized for compression efficiency or music fidelity. In realtime agents, the goal is intelligibility and stability, not archival quality.

That said, codec choice is often less important than codec consistency. If your input audio arrives as PCM, is transcoded to a compressed format, then decoded again before being rendered or analyzed, you are paying latency and CPU for no user-visible benefit. Each conversion adds buffering and scheduling points where time slips away.

Good practice:

1. Keep the audio path in one sample rate and one channel configuration as long as possible.

2. Avoid transcoding unless the downstream system requires it.

3. Keep the encoder and decoder on the edge of the pipeline, not in the middle.

This is especially important when the avatar renderer is downstream of a voice agent. If your agent already generates the spoken response as streaming audio, feed that stream directly into the avatar layer rather than rebuffering it into larger chunks for convenience.

Latency tuning in practice: buffer sizes, jitter, and sync

Once you have a low-delay codec, the next gains come from the buffers around it. This is where most realtime systems accidentally lose the advantage they just gained.

Three knobs matter most:

• Encoder queue depth: Keep it near zero. If you can encode on arrival or on the next frame boundary, do that. Avoid “helpful” queues that wait for more audio.

• Receiver jitter buffer: Use the smallest buffer that still survives your target network conditions. On a controlled LAN, you can run very small. On the public internet, leave some room for variance.

• Video sync strategy: Drive the avatar mouth from the same audio timing reference that the client hears. If video and audio use different clocks, desync will appear even when both streams are individually low-latency.

A useful mental model is that low-latency audio is not just “fast audio”; it is “predictable audio.” The avatar renderer needs a consistent relationship between audio timestamps and mouth movement. If the audio pipeline is variable, the video pipeline either jitters or lags to compensate.

That is why end-to-end measurement matters. Measure capture-to-playout latency, not just encode time. A codec that saves 8 ms in the encoder but adds 30 ms in buffering is still a regression.

Measuring the real bottleneck

If your avatar feels slow, do not guess. Instrument the pipeline:

• Capture timestamp: when the audio frame is collected.

• Encode timestamp: when the codec output is produced.

• Network send/receive timestamps: when packets leave and arrive.

• Playout timestamp: when audio becomes audible or the avatar consumes it.

• Render timestamp: when the mouth movement is displayed.

Once you have those numbers, the bottleneck usually shows up immediately. Common patterns:

• Large encode gaps mean your frame size is too big or the codec is too expensive for the device.

• Large receive-to-playout gaps mean the jitter buffer is too conservative.

• Large render gaps mean the avatar layer is waiting on synchronized audio timestamps or batching frames internally.

On the server side, avoid hiding latency behind queues. A queue makes throughput look good and latency look bad. For conversational agents, you want the opposite: stable, bounded latency with enough headroom for network variance.

How this fits Protoface in a LiveKit voice agent

If you are embedding an avatar into a LiveKit voice agent, the cleanest approach is to keep the media path simple and let the avatar follow the agent’s audio stream rather than rebuilding it. The quickstart examples and the LiveKit plugin documented in the repo show the intended shape: your agent produces audio, and the avatar consumes that audio with synchronized lip motion.

That matters for latency because you are not introducing another arbitrary buffering layer. The plugin is designed to drop an avatar into the agent pipeline, so you can focus on audio timing instead of wiring up a separate video service. For the exact integration details, installation steps, and configuration fields, use the docs at docs.protoface.com and the relevant repository examples.

If you want to inspect or automate sessions directly, the REST API and Python SDK are also useful for building latency tests. For example, a small harness can create a session, attach an avatar, and then compare stream timing across different audio settings.

import os

print(resp.json())

import os

print(resp.json())

import os

print(resp.json())

The exact request shape depends on the endpoint and session fields in the docs, but the pattern is the same: create a controlled session, vary one audio parameter at a time, and measure end-to-end latency before and after.

Concrete tactics that usually help

If you need a practical checklist, this is the one I would use before blaming the network:

• Keep frame sizes at 20 ms or below for interactive speech.

• Do not batch multiple frames unless you have a specific transport reason.

• Avoid codec changes midstream.

• Eliminate resampling and channel conversion when possible.

• Minimize encoder and decoder queue depth.

• Trim jitter buffers until they barely tolerate your real network conditions.

• Drive avatar animation from the same audio timeline as playback.

And one anti-pattern to call out explicitly: do not optimize for compression ratio first. A few extra kilobits per second are usually cheaper than extra latency in a conversational product. Users will notice 150 ms of lag far more than they will notice modest audio overhead.

Conclusion

Reducing audio codec latency in realtime avatar streams is mostly about removing buffering and avoiding unnecessary transformation. Choose short frames, keep the pipeline continuous, minimize transcoding, and tune jitter buffering to the actual network rather than a worst-case fantasy. Then verify with timestamps instead of intuition.

If you are building a voice agent with a synchronized face, start with the docs, instrument the path, and make one change at a time. The quickest way to better latency is usually not a new model or a more aggressive codec preset; it is a simpler media pipeline. For integration specifics and examples, see docs.protoface.com.