Automated Regression Testing for Lip-Sync, STT, and TTS in Webflow Avatar Flows
Automated regression testing for realtime avatars: measure lip-sync, STT accuracy, and TTS latency in Webflow avatar flows.
Introduction
When you ship a product with realtime avatars, the failure modes are different from standard web UI regressions. A face can be visually correct but out of sync with audio, speech recognition can silently drift behind the actual utterance, and a “working” session can still feel broken because latency crosses a human-perception threshold. Those issues are easy to miss in unit tests and hard to catch with manual QA once you have multiple providers, multiple browsers, and a few quality tiers in production.
This post shows a practical way to regression test the three things that matter most in an avatar flow: lip-sync alignment, STT correctness, and TTS playback behavior. By the end, you should be able to build a small automated harness that runs the same conversation repeatedly, captures timestamps and transcripts, and flags regressions before they reach users.
What to test in an avatar flow
A realtime avatar pipeline usually has three distinct subsystems:
TTS: text is turned into audio, usually as a stream of chunks rather than a single file.
Lip-sync / video rendering: visual frames are generated or selected to match the audio stream and phoneme timing.
STT: user audio is transcribed, often incrementally, with partial hypotheses that may be revised before finalization.
For regression testing, the question is not “did the feature work once?” but “did the system preserve timing and semantic quality across releases?” That means measuring:
End-to-end latency: time from user utterance start to first meaningful avatar response.
Audio/video alignment: whether mouth motion tracks the spoken content closely enough to avoid uncanny drift.
Transcript fidelity: whether STT preserves the user’s intent and key tokens.
Stability across runs: whether the same prompt yields consistent latency and transcription outcomes.
A useful mental model is to test the system at two layers: “protocol” and “perception.” Protocol tests verify events, payloads, and timestamps. Perception tests verify whether the resulting experience looks and sounds right to a human, which usually means defining tolerances rather than exact matches.
Build a deterministic test harness around a scripted conversation
The best regression tests start with a reproducible input. Instead of free-form user chatter, define a short script with known utterances, expected transcripts, and timing assertions. For example:
Start a fresh session.
Send a short prompt such as “Say the word banjo and then count to three.”
Capture the assistant audio start, the first visible lip movement, and the final transcript from STT.
Assert that the transcript contains the expected content and that lip motion begins within an acceptable window after audio onset.
The exact transport depends on your stack, but the test logic is the same whether you are driving a browser, a WebRTC client, or a voice agent framework.
Measure timing explicitly, not by eyeballing video
In realtime systems, “feels delayed” often corresponds to a measurable timing bug. You want timestamps at the moment each event occurs, not just a final recording. A minimal harness should record:
When the user audio was injected.
When STT first emitted a partial transcript.
When the assistant first emitted audio.
When the first non-static avatar frame arrived.
When the final transcript and final audio completed.
For lip-sync, a coarse but useful metric is the delay between first audio sample and first visible mouth movement. If that delta changes materially between builds, you likely have a regression in streaming, buffering, or frame scheduling. For STT, compare final transcripts against a golden set and allow for expected variability in punctuation or casing while still flagging lexical changes that alter meaning.
Keep the thresholds narrow enough to catch regressions, but not so narrow that normal network jitter creates noise. For browser-based tests, I usually prefer percentile-based thresholds over single absolute numbers. For example, “p95 time to first frame stays below 700 ms” is more robust than “every run must be below 500 ms.”
Regression testing STT: compare meaning, not just exact strings
STT regressions are often subtle. A model update might preserve the gist of a sentence but lose a proper noun, a number, or a negation. Those are the cases that matter in voice agents.
A practical strategy is to define a small corpus of scripted prompts that include:
numbers, dates, and email addresses;
named entities and product names;
negative phrasing such as “don’t cancel”;
short commands and longer conversational turns.
Then validate the final transcript using token-level checks plus a semantic fallback. For example, exact matching can be required for identifiers and numbers, while fuzzy matching is acceptable for filler words and punctuation. If your pipeline emits partial transcripts, you can also test that the final transcript is stable enough not to oscillate excessively.
One important gotcha: if you evaluate STT only on final text, you can miss a bad user experience where the model repeatedly revises partial hypotheses. In conversational systems, those revisions can confuse downstream turn-taking even if the final text looks fine. So capture both partial and final outputs, and watch for churn.
Regression testing lip-sync: use perceptual checks and timing tolerances
Lip-sync is harder to validate with a single numeric score because the output is visual, but it is still testable. You do not need to solve phoneme-to-viseme alignment in your test suite; you need to detect when alignment drifts enough for humans to notice.
A good regression test for lip-sync usually combines:
Frame timing: verify that mouth movement starts promptly after the first audio chunk.
Motion continuity: check that frames change smoothly rather than freezing or jumping.
Audio/video sync: compare the timeline of major speech peaks with visible articulation changes.
In practice, this can be as simple as capturing a short session recording and running a frame-difference metric over the mouth region, or as involved as using a human-reviewed golden video for release gating. The right choice depends on how sensitive your product is to visual fidelity. For customer-facing avatars, I recommend at least a deterministic smoke test plus a periodic golden-run review.
Also watch the trade-off between streaming latency and visual quality. Lower buffering can reduce delay, but if the video side is starved for audio context, mouth shapes may become less stable. Regression testing should therefore include “good enough” thresholds for both latency and smoothness, not just one or the other.
How Protoface fits into the test loop
This is where Protoface is useful: it gives you a controlled avatar surface you can drive from your existing voice-agent or session harness, so you can test the avatar layer without rebuilding the entire stack. If you already run a LiveKit-based agent, the LiveKit plugin and examples let you drop in a synchronized talking face and then exercise it with scripted conversations. If you prefer direct API-driven tests, the REST API is a clean way to create sessions and inspect behavior programmatically; exact request fields are documented in the docs.
Here is a minimal example of creating a session via the API in a test setup. The payload shape is illustrative; check the docs for the exact fields you need:
If you are writing tests in Python, the SDK can make session setup and teardown less brittle. Again, treat this as a shape example and confirm the concrete method names in the SDK docs:
For regression testing, the main advantage of a dedicated avatar surface is consistency. You can hold the avatar, voice, and instruction set constant while varying only the part under test: your STT model, your agent orchestration, or your WebRTC client behavior. That makes failures much easier to localize.
Practical test design: what usually breaks
Most failures I see in avatar flows come from one of four places:
Buffering changes that shift audio or video start time by a few hundred milliseconds.
Provider model updates that alter transcription of names, numbers, or punctuation.
Browser or WebRTC regressions that affect frame scheduling, autoplay, or packet timing.
Agent orchestration bugs where partial transcripts trigger the wrong turn-taking behavior.
Your regression suite should include at least one test for each category. Keep the corpus small enough that it runs on every PR, then add a slower nightly job with longer conversations and a few browser variants. If you have multiple quality tiers, run the same scripts against the tier you expect to ship so you catch tier-specific latency or rendering issues.
Conclusion
Automated regression testing for realtime avatars is mostly about making implicit experience constraints explicit. Define scripted conversations, capture event timestamps, compare transcripts with intent-aware rules, and measure lip-sync with timing thresholds rather than ad hoc review. Once those checks are in place, you can catch the regressions that users actually notice: delayed faces, drifting audio, and transcripts that lose meaning.
If you are building this kind of pipeline now, start by wiring one deterministic smoke test into your CI, then expand to a small golden corpus over time. The docs at docs.protoface.com are the best place to map these ideas onto the available API and SDK surfaces, and the GitHub examples are useful if you want to adapt the test harness to an existing voice-agent stack.