Measuring Latency and Lip-Sync Drift in CI for Conversational Video Agents
Measure end-to-end latency and lip-sync drift in CI for conversational video agents with deterministic timing, thresholds, and logs.
Introduction
When a conversational video agent feels “off,” the problem is usually not one thing. It may be network latency, jitter in the media pipeline, slow text-to-speech, avatar rendering delay, or plain lip-sync drift between audio and video. In realtime systems, those failure modes compound quickly: a 150 ms stall in one stage can become a noticeable conversational lag, and a small A/V offset can make the avatar look broken even if the speech itself is correct.
This post shows a practical way to measure both end-to-end latency and lip-sync drift in CI so regressions are caught before they ship. The goal is not to build a research-grade perceptual model. It is to define repeatable metrics, instrument the pipeline, and establish thresholds that reflect what your users can actually perceive.
What to measure, exactly
For conversational video agents, “latency” is usually overloaded. Break it into measurable segments:
Turn-start latency: time from user input completion to first audible agent response.
Avatar start latency: time from response start to first rendered video frame of the avatar speaking.
End-to-end latency: time from user utterance end to a synced audio/video response becoming visible.
Lip-sync drift: the temporal offset between audio phonemes and mouth motion over time.
The first three are pipeline metrics. The last one is a synchronization metric. You need both, because a system can be “fast” and still look wrong if mouth movements lag the audio by 80–120 ms, especially on plosives and fricatives.
In CI, define the measurement boundary up front. For example:
Start the timer when the test client sends a transcript chunk or audio input end marker.
Stop the timer when the first decoded audio sample is available, or when the first compositor frame containing the speaking avatar is received.
For drift, compare the timestamps of audio events to the corresponding visual mouth-open/close events over a fixed utterance.
That scope keeps the test reproducible. If you include browser startup, DNS, or arbitrary render warm-up, your numbers will vary too much to be useful in a PR gate.
Build a deterministic test harness
CI latency tests fail when they depend on live user behavior or unstable internet paths. The harness should be boring:
Use a fixed script or canned audio sample as input.
Pin the environment: same browser version, same region, same CPU class when possible.
Run multiple iterations and report percentile-based results, not just the mean.
Capture raw timing events so regressions can be debugged after the fact.
A simple pattern is to embed monotonic timestamps in your test runner and instrument each stage in the request path. If you control the voice agent, emit events for:
input received
LLM response started
TTS audio started
avatar session started
first video frame rendered
Then compute deltas in the harness. A minimal Python example might look like this:
This is intentionally simplistic. The useful part is the discipline: measure from one stable clock, record intermediate markers, and store enough context to explain outliers.
Measuring lip-sync drift without overfitting the metric
Drift is the difference between when audio should influence the mouth and when the mouth actually moves. You can measure it several ways, but the most practical CI approach is to use a fixed phrase with strong phonetic transitions and compare audio landmarks to visual landmarks.
For audio, use either forced alignment or a known script with timed phonemes. For video, detect changes in the mouth region over time. You do not need perfect phoneme-to-viseme mapping to catch regressions; you just need a consistent proxy that breaks when sync gets bad.
A workable strategy:
Use a short phrase with alternating closed and open mouth shapes.
Record the agent output as raw media or an encoded stream.
Extract audio energy peaks or forced-alignment phoneme timestamps.
Extract frame timestamps and a mouth openness signal from the avatar region.
Estimate the lag that maximizes correlation between the two signals.
If the correlation peak shifts by more than your tolerance, fail the test. In practice, tolerances are product-specific, but once drift is noticeable to humans, it is usually on the order of tens of milliseconds, not hundreds.
Be careful not to turn this into a brittle pixel test. Compression artifacts, rate control, and codec differences can move the signal around a little. The test should detect meaningful sync regressions, not fail because a browser updated its video pipeline.
Practical thresholds and failure modes
Good thresholds are empirical. Start by running the test on a known-good baseline across several days, then set limits around the 95th or 99th percentile. Typical failure modes in conversational video agents include:
Slow TTS startup: audio starts late, but the avatar may still animate on time, creating a pre-roll gap.
Video pipeline lag: audio is correct, but frame delivery or compositor scheduling delays the mouth motion.
Clock skew between services: independently timestamped events are compared without a shared monotonic clock.
Codec or buffering changes: audio is decoded earlier than video, which looks like drift even if generation was aligned.
Two useful guardrails:
Track both the absolute latency and the relative A/V gap. A system can regress in total latency while keeping sync, or keep latency flat while losing sync.
Log raw event timelines for every failing run. A single timeline with 6-10 timestamps is usually enough to pinpoint the stage that changed.
If your CI runs on GitHub Actions or similar shared infrastructure, expect some noise. Use repeated runs and compare against a baseline window rather than a single hard cutoff for every metric. For example, a PR can fail if median avatar-start latency increases by more than 20% or if lip-sync drift exceeds a fixed maximum for two consecutive runs.
How Protoface fits into this
For teams using Protoface as the avatar layer, the most useful integration point for this kind of testing is the LiveKit agent path, because it lets you treat the avatar as part of the realtime media stack rather than a separate UI concern. The plugin on PyPI, pipecat-protoface, and the related Pipecat integration are the natural place to instrument session start, audio onset, and first-frame timing in a voice-agent pipeline.
A minimal shape of the integration looks like this:
If you are wiring the avatar into a LiveKit agent, the plugin path is even better for CI because it keeps your test close to the real production code path. The relevant examples in the plugin repo are a better starting point than inventing a custom harness from scratch; see the plugin repository and the Pipecat service guide for the current integration shape.
For direct API-driven tests, use the REST API to create sessions from CI, then record the returned media stream and analyze it offline. A curl-style request is enough to bootstrap the session side; keep credentials in CI secrets and never ship them to the browser:
The exact fields depend on the endpoint shape in the docs, but the testing pattern is the same: create a known session, drive it with a known input, and measure the output timeline deterministically.
CI implementation details that matter
A few details tend to decide whether these tests are useful or just noisy:
Use monotonic time, not wall-clock time.
Run enough samples to separate random variation from regression.
Store artifacts: raw event logs, audio snippets, and a representative video clip.
Measure the same scenario every time: same utterance length, same voice, same avatar quality tier.
Keep the threshold tied to user experience, not an arbitrary engineering target.
Also remember that quality tiers can change latency. If your platform bills by quality tier, make sure CI locks the tier you actually intend to ship, otherwise you will compare apples to oranges and end up chasing phantom regressions.
Conclusion
Latency and lip-sync drift are separate failure modes, and both are worth testing in CI for conversational video agents. The practical approach is to measure stable pipeline milestones, run a repeatable scripted interaction, compare audio and visual timing against a known baseline, and fail builds only when the regression is real enough to matter to users.
If you are building on a realtime avatar stack, start with a deterministic smoke test, add raw event logging, then tighten thresholds as you learn what “good” looks like in your own environment. The docs are the right place to confirm the exact API shapes and integration details, and the quickstart repos are useful references when you want to move from theory to a working test harness quickly.