A CI Pipeline for Realtime AI Avatar QA: Audio-to-Face Sync Checks in GitHub Actions
CI workflow for testing realtime avatar audio-to-face sync in GitHub Actions with lag thresholds and artifacts
Introduction
When you ship a realtime avatar, the hard part is usually not “can I generate video?” It’s whether the video face stays synchronized with the audio stream under the conditions your users actually hit: variable network jitter, backend latency spikes, TTS chunking, and agent turn-taking. If you’re building on top of a voice agent stack, you need a way to catch regressions before they reach production, and manual spot-checking will not scale.
This post shows a practical CI pattern for testing audio-to-face sync in GitHub Actions. By the end, you should be able to spin up a repeatable pipeline that runs a voice-agent conversation, captures the avatar video and audio, computes sync signals, and fails the build when lip movement drifts beyond a threshold.
The examples are deliberately lightweight. The goal is not to create a perfect perceptual test; it’s to build a reliable automated gate that catches the classes of bugs developers actually introduce: wrong buffering, timestamp mishandling, broken media plumbing, and unintended latency regressions.
What to measure in CI
“Lip sync” is not a single metric. For CI, you want metrics that are stable, cheap to compute, and sensitive to the failures you care about. In practice, the simplest useful setup is:
Audio onset vs. mouth motion onset: detect when speech starts and when the mouth opens.
Per-utterance lag: estimate how many milliseconds the visual response trails the audio.
Regression threshold: compare the current run against a baseline or an absolute budget.
This works because realtime avatars are fundamentally a streaming system. Audio frames arrive incrementally, the agent may emit partial text, TTS converts it to audio in chunks, and the avatar renderer schedules facial motion from that stream. A test that only checks “did we get a video file?” misses the failure mode. A test that checks timing across the entire pipeline catches it.
Build a deterministic test conversation
The key to useful CI is determinism. Avoid open-ended prompts and avoid any test content that depends on model creativity. Instead, use a short script with fixed utterances and a fixed speaking rate. You want a conversation that produces a few clean mouth-open / mouth-close cycles, not a wide-ranging agent exchange.
At a minimum, your test harness should:
Start a realtime session.
Connect a synthetic or stubbed voice agent.
Send a fixed phrase like “one two three four five.”
Record the resulting audio/video stream for a few seconds.
Run sync analysis offline.
If your pipeline includes multiple services, keep the test focused on the avatar boundary. You are not trying to validate the model’s intelligence in CI; you are validating the media path and the avatar rendering contract.
How to compute sync from the recording
There are several ways to infer mouth motion. The most robust CI-friendly option is to inspect frame-level facial landmarks or a coarse mouth-open ratio over time. You do not need an ML-heavy perceptual model to catch obvious regressions.
A simple offline approach is:
Extract the audio track and detect speech onset with an energy threshold or VAD.
Sample frames at a fixed rate, e.g. 10–15 FPS for analysis.
Estimate mouth openness per frame using landmark distance or a bounding-box heuristic.
Align the two time series and compute lag at the first utterance and over the utterance window.
What matters is consistency. Use the same frame sampling rate and the same detector in every CI run. If you change the detector, expect your baseline to shift.
A minimal pseudo-implementation looks like this:
That threshold is an example, not a universal recommendation. Pick a number that matches your avatar quality tier, your network path, and the amount of buffering your stack intentionally introduces. The point is to make the budget explicit.
A GitHub Actions workflow that actually fails on drift
In CI, the test runner should be isolated from the build machine as much as possible. Your workflow usually needs three pieces:
A job that launches the test session and records media.
A job step that runs sync analysis.
A hard failure when the lag exceeds the acceptable range.
Here is a compact example using Python for orchestration and analysis:
The test script should emit artifacts even on failure so you can inspect the offending run. In GitHub Actions, that usually means uploading the recording and any debug plots as artifacts. The most useful debugging output is a simple timeline plot showing audio energy and mouth openness over time.
Two practical gotchas matter here:
Clock drift: if audio and video timestamps come from different clocks, normalize to the recording timeline rather than wall time.
Warm-up artifacts: ignore the first few hundred milliseconds if the session has connection setup or initial buffering.
Example orchestration with the Python SDK
If you prefer programmatic control, the Python SDK is the cleanest place to create sessions and manage test runs. The exact request fields depend on the API shape in the docs, but the flow is straightforward: authenticate, create a session, connect the agent, and record the stream for later analysis.
If you’re wiring this into a voice-agent stack, keep the agent prompt and playback configuration fixed across runs. That makes regressions attributable to media changes instead of prompt drift.
Where the realtime avatar integration fits
For teams already using LiveKit voice agents, the fastest way to get a meaningful CI signal is to test the avatar boundary directly through the plugin rather than mocking everything around it. The LiveKit plugin attaches a Protoface avatar to the agent so the agent’s spoken output is rendered as a synchronized talking face. That means your test exercises the same streaming path your application uses in production, which is exactly what you want in a regression test.
In practice, the test does not need to be complicated. A short agent response through the plugin is enough to verify whether audio packets, frame timing, and mouth animation remain aligned. If you are using the plugin examples as a starting point, the repository is the right place to look: https://github.com/protoface-ai/protoface-quickstart-openai-realtime is useful for understanding the shape of a realtime voice-to-avatar path, and the public docs at https://docs.protoface.com cover the API details you’ll need to make the test real.
The important architectural point is that this is a streaming integration test, not a unit test. You want the live media path, the agent loop, and the avatar renderer all in play, because that is where sync bugs emerge.
Practical CI hardening tips
A few things make these tests much more stable:
Use fixed utterances with repeated plosives and vowels; they create visible mouth motion.
Record short sessions; 5–10 seconds is often enough.
Fail on large regressions, not tiny noise; lip sync is naturally a little variable.
Keep artifacts so you can inspect every failure without rerunning locally.
Run in a controlled environment; avoid shared runners if you need tighter timing.
If you’re testing a web embed rather than a server-side agent, the same principle applies: drive the avatar through the same transport you ship to customers. For customer-managed iframe embeds, that means testing the embed surface and its media path, not a mocked version of it. The failure you want to catch is “the UI looks fine but the face is a beat behind the voice,” which is usually a transport or buffering problem, not a rendering bug.
Conclusion
A good CI check for realtime avatar sync is small, deterministic, and unforgiving in the right places. Record a known phrase, measure audio onset against mouth motion, and fail the build when the lag exceeds your budget. That gives you a concrete guardrail against the kind of regressions that are easy to miss in manual QA and painful to debug after release.
If you want to implement this with less scaffolding, start from the integration docs at docs.protoface.com and the relevant quickstart or plugin repository for your stack. Once the basic test is in place, you can expand it with more utterances, additional quality thresholds, and artifact uploads for easier debugging. The main thing is to make sync a machine-checkable contract, not a human eyeball test.