A Kotlin Developer’s Guide to Testing Lip-Sync Accuracy in Streaming AI Avatars
Kotlin guide to measuring streaming avatar lip-sync: capture A/V, estimate offset, and catch CI regressions.
Introduction
If you are shipping a voice agent with a talking face, “does it speak?” is the wrong test. The useful question is whether the video face stays visually aligned with the audio under real streaming conditions: network jitter, chunked TTS, variable frame cadence, and occasional latency spikes. In practice, lip-sync failures are often subtle enough to pass manual review and still be bad enough to erode trust.
This post shows how to test lip-sync accuracy in a streaming avatar system from a Kotlin developer’s perspective. By the end, you should be able to define a measurable sync metric, build a small test harness, and set up checks that catch regressions before they reach users.
What “lip-sync accuracy” actually means in streaming systems
In a prerecorded pipeline, you can compare audio and video against a fixed timeline. In a realtime avatar, that timeline is negotiated live: audio may be generated in chunks, video frames may arrive later than expected, and playback buffering can move the effective start time. So the thing you want to test is not absolute timestamp equality, but relative alignment over time.
There are three useful notions here:
Audio-video offset: how far the face motion lags or leads the spoken audio at any moment.
Stability: how much that offset varies over a session.
Degradation under load: how the offset changes when bandwidth drops, CPU is constrained, or the agent is under conversational pressure.
For avatars driven by TTS, the main failure mode is usually video lag: the audio begins, but the mouth shapes arrive a few hundred milliseconds too late. Even if that sounds small, it is visible. Once you get past roughly 150–200 ms of sustained offset, most users can notice the mismatch.
Build a test harness around observable signals
You cannot measure lip-sync by staring at pixels in a single screenshot. You need a harness that captures synchronized audio and video output from the same session and computes a timeline.
A practical setup has these pieces:
Deterministic stimulus: a fixed script or phoneme-rich utterance that produces clear mouth motion changes.
Session capture: record the avatar’s audio and video streams during the test.
Alignment algorithm: estimate offset from the recorded streams.
Thresholds: fail the test if median offset, max offset, or jitter exceeds your budget.
For the stimulus, use text that creates distinct plosive and vowel transitions. Phrases with lots of bilabials and open vowels are helpful because the mouth shape changes are easier to detect. A boring but effective test string is something like:
The important part is not the exact words; it is that they create visible mouth changes and are consistent between runs.
Measure the offset instead of guessing it
There are two broad approaches to estimating audio-video alignment:
Manual annotation: review the recording and mark when spoken sounds and mouth shapes begin. This is slow and fine for debugging, but poor for automation.
Signal-based analysis: derive an audio energy envelope and a video motion or mouth-openness trace, then correlate them over time. This is the right approach for CI.
For automation, you do not need perfect phoneme recognition. A coarse audio envelope often works well enough to detect gross sync drift. On the video side, you can track mouth region motion frame-to-frame or use a lightweight landmark detector if your test environment supports it.
The core idea is to compute two time series sampled on the same clock: one from audio amplitude, one from mouth activity. Then find the lag that maximizes correlation. That lag is your estimated offset.
A minimal Kotlin test shape
In Kotlin, keep the test code boring. Spin up the session, play the stimulus, capture the result, and hand the media off to an analyzer. The test should assert on numbers, not on visual intuition.
The exact thresholds depend on your product. A customer-support bot viewed in a browser can tolerate slightly more than a game NPC where visual responsiveness matters more. The useful part is to make the budget explicit and enforce it continuously.
Practical gotchas that break sync tests
Most lip-sync test failures are not model failures; they are pipeline failures. A few common ones:
Buffered playback: if your test starts measuring before the audio playout buffer is primed, the video may appear to lag even when generation is fine.
Variable frame rate: if your capture path drops frames, the motion trace becomes noisy and correlation becomes less reliable.
Codec delay: encode/decode latency can add a fixed offset that should be accounted for in the harness, not blamed on the avatar.
Session warmup: first-utterance behavior is often worse than steady state because streams, models, and caches are still initializing.
Two tactics help a lot in CI:
Measure steady-state offset after the first second or two of playback, not from time zero.
Store a baseline recording and compare new runs against it so you can spot regressions even when absolute numbers look acceptable.
If you are testing at the level of “avatar face is present in a WebRTC session,” that is not enough. You want to know whether the synchronization budget is preserved end-to-end.
Make the test resilient to streaming reality
Streaming systems are noisy by design, so the test should tolerate small fluctuations while still catching meaningful regressions. A few guidelines:
Test multiple runs: one good run can hide intermittent jitter. Use a small batch and aggregate.
Prefer distributions to single values: median, p95, and max are more informative than one average.
Separate generation from transport: if your model is fine but the transport layer is unstable, you need to know which layer regressed.
Version your fixtures: keep the stimulus text, the recording pipeline, and the thresholds under source control.
For Kotlin projects, I usually keep the analyzer as a pure function over captured media metadata where possible. That makes it easier to run locally, in CI, and on historical recordings. If you need to inspect raw streams, do it in a separate diagnostic tool rather than in the test itself.
Where Protoface fits
If your avatar is backed by Protoface, you can apply the same testing approach to a real realtime session instead of a mock. The most direct path is to create or drive a session through the REST API, run your stimulus, and then record the resulting audio/video for analysis. The public docs at docs.protoface.com cover the exact request shapes and session fields.
For example, you can create a session with a simple authenticated request and then use the returned session details in your test harness:
The same idea works if you are integrating through a LiveKit voice agent: use the Pipecat integration or the LiveKit plugin so the avatar participates in the same realtime pipeline your product uses. That matters because lip-sync regressions often show up only when the full agent stack is involved, not in isolated unit tests.
Conclusion
Testing lip-sync accuracy is mostly a measurement problem. Define an acceptable sync budget, capture real sessions, estimate audio-video offset, and fail when the distribution drifts beyond your threshold. In a streaming avatar system, that is much more reliable than subjective review, and it gives you a regression test that reflects actual user experience.
For the next step, wire one end-to-end case into CI: start a session, drive a fixed utterance, record the output, and compute median and p95 offset. Once that is stable, expand to jitter and first-utterance checks. If you want the exact API shapes or quickstart code, the docs at docs.protoface.com are the right place to start.