Introduction

If you’re building a realtime avatar that lip-syncs to an incoming audio stream, the hard part is usually not the face render itself. It’s the boundary between transport and playback: packets arrive out of order, in bursts, with jitter, and at a cadence that is almost never the same as the render loop. If you play audio or drive mouth cues directly from arrival time, the avatar will look unstable, even if the underlying model is fine.

A jitter buffer is the small piece of systems glue that makes the whole experience feel “locked.” It trades a tiny amount of latency for continuity and timing stability. In this post, we’ll look at how to design one for low-latency avatar lip-sync in WebSocket and WebRTC apps, what metrics matter, and where the common failure modes are. By the end, you should be able to reason about buffer sizing, playout policy, and sync strategy for a realtime avatar pipeline.

What the jitter buffer is actually doing

A jitter buffer is not a generic queue. It is a time-aware reorder-and-delay stage. Its job is to absorb variation in packet or chunk arrival time so downstream consumers can operate on a steadier cadence.

In a lip-sync pipeline, the buffer usually sits between transport and one or both of these consumers:

• the audio playout device, which needs continuous, evenly spaced samples;

• the avatar animator, which needs reliable timing signals for visemes, phoneme boundaries, or speech activity.

For WebRTC, the transport already includes congestion control, packet sequencing, and often an RTP jitter buffer in the media stack. You still need to think about application-level timing if you are consuming a mixed stream of audio plus animation metadata. For WebSocket, you own the full problem: message framing, ordering, backpressure, and how much to buffer before starting playback.

The core trade-off is simple:

• larger buffer = fewer glitches, but more end-to-end latency;

• smaller buffer = lower latency, but more risk of underflow and visible mouth jitter.

Design the buffer around timestamps, not arrival order

The first mistake is to treat “received first” as “should play first.” Network arrival time is noisy, and WebSocket message delivery may bunch frames together after a GC pause, a browser scheduling hiccup, or a transient network delay.

Every chunk that enters the buffer should carry a media timestamp from the producer side. That timestamp can be a presentation time, sample index, sequence number with a fixed frame duration, or an RTP timestamp if you are in a WebRTC path. Your playout logic should key off that timestamp, not the wall-clock receive time.

A practical buffer object needs three things:

• ordered storage by timestamp or sequence number;

• a playout clock that advances independently of arrival;

• a policy for missing, late, or duplicated frames.

# Minimal shape of a timestamped jitter buffer

return out

# Minimal shape of a timestamped jitter buffer

return out

# Minimal shape of a timestamped jitter buffer

return out

This is intentionally simplified, but the shape matters. You want a stable playout clock and a small, bounded delay. The buffer is not “full” or “empty” in the abstract; it is ahead of or behind the current playout deadline.

How to choose buffer depth

Buffer depth is a latency budget decision, not an implementation detail. A common starting point for low-latency lip-sync is 40–120 ms of buffered audio or timing metadata, depending on network quality and how aggressively the sender batches frames. If you are streaming very small chunks at high frequency, you can often run closer to the low end. If the network is variable, a little more delay is worth it.

A useful way to size it is to measure jitter as the variance of inter-arrival spacing relative to expected frame cadence. If your sender emits 20 ms audio frames and you regularly see 10–30 ms spacing on the wire, a 40 ms buffer is optimistic. If the same stream is stable within a few milliseconds, 40 ms may be enough.

In practice, a good initial policy is:

1. Start playback after you have accumulated N frames or M milliseconds, whichever comes first.

2. Keep a moving estimate of network jitter using recent inter-arrival deltas.

3. Expand the buffer slowly when underflow occurs; shrink it conservatively when the stream is stable.

This “adaptive but biased toward stability” approach avoids a common failure mode: aggressive shrinking after a brief stable period, followed by immediate underflow when the network oscillates again.

Underflow, late frames, and catch-up behavior

You need explicit behavior for three cases:

• Underflow: nothing is ready when playout needs the next frame.

• Late arrival: a frame arrives after its playout deadline.

• Burst arrival: several frames arrive at once after a stall.

For avatar lip-sync, underflow is worse visually than a slight delay. If audio continues but animation lags, the face can appear “slippery” or detached from the voice. If both audio and visemes underflow together and you insert a short concealment strategy, the artifact is usually less noticeable than a partial desync.

Common strategies:

• repeat the last stable mouth shape for one frame or one audio quantum;

• hold the current viseme while waiting for the next timestamped update;

• drop late animation frames rather than trying to render them out of order;

• if audio underflows, insert silence or comfort noise, depending on the media stack.

For lip-sync metadata, “late but still useful” is a narrow band. Once a viseme cue is late enough that it misses its intended phonetic window, it is usually better to discard it than to render it with stale timing. The avatar should stay temporally coherent, even if that means sacrificing some micro-accuracy.

WebSocket versus WebRTC: the buffer lives in a different place

With WebSocket, you are typically streaming application messages: audio chunks, phoneme timing, or speech events. That means the jitter buffer is part of your app server or browser client. You control the buffering policy entirely, which is good for flexibility and bad for mistakes. If you multiplex audio and animation on the same channel, make sure each message type has its own timestamp semantics and its own queueing policy.

With WebRTC, audio transport is usually handled by the media engine, which already performs packet reordering and playout buffering. The application-level concern shifts to synchronization across media tracks and metadata. For example, if your voice agent produces speech audio plus separate lip-sync events, the buffer must align those event timestamps with the audio timeline that the client actually hears.

The important design implication is this: in WebRTC, don’t fight the media stack by inventing a second audio playout clock. Instead, anchor animation to the media timing model you already trust. In WebSocket systems, you may need to build both the media buffer and the sync buffer yourself.

Practical implementation details that matter

A few details tend to make or break low-latency behavior:

• Monotonic clocks: use a monotonic time source for playout decisions; wall clock jumps are poison.

• Bounded queues: cap buffer growth, or a stalled consumer will turn latency into seconds.

• Sequence validation: reject duplicates and detect gaps early.

• Separate control and media paths: don’t let slow metadata processing block audio intake.

• Observable metrics: track buffer occupancy, underrun count, late-frame count, and effective latency.

For debugging, the best single graph is buffer occupancy over time with underrun markers. If occupancy sawtooths between empty and full, your target delay is wrong or your input cadence is too bursty. If it hovers high and stable but latency feels bad, the buffer is probably over-provisioned.

Where Protoface fits

This is exactly the kind of problem Protoface is meant to hide from application developers: you send a realtime voice stream or session events, and the avatar side is responsible for producing a synchronized talking face without making you hand-roll all the media plumbing. For developers integrating a voice agent, the LiveKit plugin is the most direct path; it drops a Protoface avatar into the agent so you get synchronized video output alongside the conversation. The plugin and examples are documented in the integration repo, and the broader API behavior is covered in the docs.

That doesn’t eliminate buffering concerns entirely; it just moves them to the right layer. Your app still needs clean timestamps, sensible chunking, and a realistic expectation of end-to-end latency. But you no longer need to build avatar playback logic from scratch just to get a talking face on screen.

Conclusion

A good jitter buffer for avatar lip-sync is small, timestamp-driven, bounded, and observable. It should smooth transport variance without hiding latency inside an ever-growing queue. For WebSocket apps, you own the whole timing path. For WebRTC apps, you usually sync to the media stack rather than reinventing it. In both cases, the goal is the same: preserve temporal coherence so the avatar looks attached to the voice, not merely adjacent to it.

If you’re implementing this yourself, start with a conservative target delay, measure underruns and occupancy, then tune from real network traces rather than intuition. If you’d rather focus on the agent and product logic, take a look at the docs, the quickstarts, and the LiveKit integration path on GitHub; they’re the shortest route to a working realtime avatar pipeline.