Testing Lip-Sync and Audio-Video Sync in Realtime Avatar Pipelines with Rust
Rust harness for measuring lip-sync, audio-video skew, drift, and jitter in realtime avatar pipelines under load.
Introduction
Realtime avatar pipelines fail in subtle ways. The face can be “right” visually but still feel off if mouth motion lags audio by 80 ms, if buffering causes occasional jumps, or if video cadence drifts relative to the speech clock. In practice, lip-sync and audio-video sync are not one problem; they are a set of timing problems across capture, inference, encoding, transport, and playback.
This post is about how to test those problems like an engineer, not like a demo viewer. By the end, you should be able to define measurable sync checks, build a repeatable test harness in Rust, and reason about where latency and drift come from in a realtime avatar pipeline. I’ll also show where Protoface fits when you want to plug a lip-synced avatar into a voice agent or a web experience without rebuilding the entire media stack.
What you should measure, not just watch
If you only watch a few clips manually, you’ll miss the failure modes that matter in production. For realtime avatars, the key metrics are usually:
End-to-end latency: from audio emission to visible mouth motion on the client.
Audio-video skew: how far the rendered video lags or leads the audio timeline.
Drift: whether skew grows over time under steady load.
Jitter: short-term variation in frame arrival or frame presentation times.
Freeze rate: periods where audio keeps playing but video stops updating.
The important distinction is between pipeline latency and sync quality. A pipeline can be slow but still tightly synced. Another can be fast on average but produce small timing errors that are perceptually worse because the mouth is consistently out of phase with the phonemes.
For lip-sync testing, I like to split the problem into two layers:
Transport correctness: are audio and video frames arriving with plausible timestamps and monotonic ordering?
Perceptual alignment: do mouth movements align with voiced segments, plosives, and pauses within an acceptable tolerance?
Transport checks can be automated entirely. Perceptual checks can be automated too, but only if you define a measurement strategy rather than eyeballing the result.
Build a deterministic test harness
The easiest way to test realtime sync is to remove as much variability as possible. Feed the pipeline a known audio track, record the resulting video, and compare timestamps on both sides. You want a harness that does three things:
Uses a fixed input utterance with clearly identifiable speech events.
Records audio and video timestamps at ingress and egress.
Computes skew and drift over the whole session, not just at the end.
A practical approach is to generate an input clip with alternating silence and short, sharp consonants. Consonants like p, b, and m tend to create visible mouth closure/opening events, which makes them useful markers. If you control the TTS, you can also insert short pauses so you get clean boundaries.
Rust test scaffolding
Rust is a good fit here because you can keep timing code precise and the data structures explicit. The core of the harness is just a timeline accumulator and a checker that compares expected event times against observed frame times.
That example is intentionally simple. In a real harness, you’d emit one sample per detectable speech event or per rendered video frame, then compute statistics such as max skew, p95 skew, and drift slope. The point is to make sync a numeric regression test.
Measuring audio-video skew correctly
There are a few common mistakes when measuring sync:
Using wall-clock time alone: system clock adjustments and scheduling jitter can corrupt your measurements. Prefer monotonic clocks for local timing.
Mixing capture time and presentation time: be explicit about whether a timestamp marks frame generation, network send, network receive, decode, or actual render.
Ignoring buffering: a frame that arrives late may still render on time if the player has enough buffer. That’s good for playback but changes the interpretation of “network latency.”
A useful pattern is to attach an event ID to each speech segment and propagate it through the pipeline. For example, if your speech generator emits segment 17 at time T0, and the video frame corresponding to that segment becomes visible at T1, then skew is T1 - T0 as observed at the client. If you can’t propagate IDs, a fallback is signal correlation: compute audio energy envelopes and compare them to mouth-open probabilities derived from the video.
Signal correlation is less precise, but it catches real regressions. A simple version is:
Compute a short-time energy envelope for the audio.
Compute a mouth-motion score for each video frame.
Slide one series against the other and find the lag that maximizes correlation.
That lag is not “truth,” but it is a strong proxy for whether the avatar is visibly behind the voice. If the best lag changes over time, you have drift. If the lag is stable but large, you have a systematic pipeline delay.
Test under load, not just in the happy path
Realtime avatar pipelines often pass single-session tests and fail when you run many sessions concurrently. The failure mode is usually not raw bandwidth; it’s scheduling and queueing. Video encoding may stall, TTS may burst, or a WebRTC sender may adapt its bitrate in ways that change frame pacing.
So your test matrix should include:
Single session with an ideal network.
Single session with induced packet loss and latency.
Many concurrent sessions on the same host.
Long-running session, ideally several minutes, to expose drift.
For long-running tests, track the difference between the expected speech event time and the observed mouth event time over the entire run. If the difference trends upward, you likely have a buffering or clock-domain issue. If it oscillates, you may be seeing adaptive jitter buffering or inconsistent frame pacing.
Practical regression thresholds
You need thresholds that are strict enough to catch regressions but loose enough to account for the realities of WAN transport and client rendering. Reasonable starting points for interactive avatars are often:
Max absolute skew: keep it within a small number of frame intervals for your chosen quality tier.
p95 skew: much more important than the median, because users notice repeated lag.
Drift over time: should be near zero for stable sessions.
Don’t hard-code thresholds without checking the actual product constraints. A conversational avatar in a support workflow can tolerate slightly more latency than an avatar in a fast back-and-forth game interaction. The right threshold depends on whether you optimize for conversational naturalness, responsiveness, or visual fidelity.
Also, test at the quality tier you intend to ship. Higher visual quality usually means more compute, larger frames, and potentially more buffering. That can improve perceived realism while also increasing sync pressure. Billing by quality tier is relevant here because the operational profile changes with the tier you pick.
Where Protoface fits
Once you have a sync harness, the integration point is usually whichever surface your product already uses. For a LiveKit voice agent, the cleanest path is the livekit-plugins-protoface plugin: drop in a Protoface avatar and let the agent gain a synchronized talking face without wiring a separate media stack. The plugin is published on PyPI, and the repo examples are a good starting point if you want to inspect how frames and audio are handed off. If you are integrating at the API layer or building tests around session creation, the REST API and Python SDK are the relevant surfaces; the exact request/response fields are documented in the docs.
A minimal REST call to create or manage a session will look like this shape:
And a Python SDK flow is similarly straightforward in structure:
If you are working in the LiveKit stack, the plugin and its examples are documented here: https://pypi.org/project/pipecat-protoface/ and https://github.com/protoface-ai/protoface-plugin-pipecat. For API and SDK details, use https://docs.protoface.com rather than guessing at field names.
Conclusion
Testing lip-sync is mostly about turning “looks okay” into measurable timing data. Build a deterministic harness, track skew and drift with monotonic timestamps, test under load, and treat sync as a regression surface like any other media bug. That gives you a way to catch the problems users actually notice: delayed mouth motion, inconsistent pacing, and subtle desynchronization that makes the avatar feel synthetic.
If you’re integrating realtime avatars into a voice agent or web app, start with the smallest viable test loop, then expand it until it exercises the same transport and buffering behavior you ship. The docs at docs.protoface.com are the right next stop for the supported surfaces, and the quickstarts are useful when you want a concrete baseline to compare against.