Introduction

Testing lip-sync in a realtime avatar system is not the same as testing ordinary video playback. You are validating a streaming pipeline: text or audio generation, speech timing, phoneme-to-viseme alignment, rendering latency, frame delivery, and browser playback all interact. If the avatar’s mouth is consistently early, late, or drifting on specific phonemes, the user experience degrades quickly even if the voice itself sounds fine.

This post shows a practical way to measure multi-language lip-sync quality from Python using Playwright. By the end, you should be able to build a repeatable test harness that:

• drives a realtime avatar session with controlled prompts or speech input,

• captures video frames from the browser with timestamps,

• checks mouth motion against expected speech timing across languages, and

• flags regressions before they reach users.

I’ll keep this focused on the mechanics of testing. The same approach works whether your avatar is embedded in a web app, driven by a voice agent, or exposed through a session API.

What “lip-sync accuracy” actually means in a realtime system

In a realtime avatar, the observable output is usually a browser-rendered video surface fed by a WebRTC or similar streaming path. The source of timing truth is rarely the same as the display clock. That means you should not test lip-sync by looking at the transcript alone, and you should not rely on subjective eyeballing for regressions.

For automation, break the problem into three measurable parts:

1. Speech timing: when the synthesized audio begins and how phonemes are distributed over time.

2. Visual mouth activity: when the avatar’s mouth opens, closes, and transitions between visemes.

3. Alignment error: the offset between the expected speech events and the observed visual events.

For English, a crude but useful approximation is to correlate mouth-open intensity with vowel-heavy regions and plosive closures. For multilingual testing, the exact phoneme inventory changes, but the core idea does not: you want to compare the motion envelope of the mouth against the known timing of the spoken audio, not just the text.

That distinction matters because a “correct” looking English clip can fail in another language if the system handles tokenization, text normalization, or TTS timing differently. Languages with different syllable density, elongation, or consonant clusters can expose timing drift that never showed up in English-only tests.

Build a repeatable test harness with Playwright

Playwright is a good fit because you control the browser, can wait on network and DOM state, and can capture screenshots or video from a known viewport. For lip-sync tests, you usually want two layers of automation:

• Session control: create or join an avatar session, inject a prompt or speech sample, and wait for playback.

• Frame sampling: grab frames at a fixed cadence and analyze mouth motion over time.

At a minimum, keep your browser viewport fixed, disable unrelated animations, and run the same test on a consistent machine or CI runner. If the avatar is rendered in an iframe or canvas, test the actual embedded surface, not a mocked component.

Capturing frames and measuring mouth motion

There are two practical ways to inspect motion in automation: computer vision on screenshots, or video export and offline analysis. Screenshots are usually simpler and good enough for regressions. The rough approach is to sample the avatar region at 5–10 fps during speech, then compute a mouth-open score for each frame.

If your avatar face is consistent, a simple region-of-interest method can work surprisingly well:

1. Crop a rectangle around the lower half of the face.

2. Convert to grayscale.

3. Compute edge density or pixel variance in a mouth subregion.

4. Track peaks and troughs over time.

That is not a perfect viseme detector, but it is often enough to detect large regressions such as “mouth stops moving,” “motion is delayed by 400 ms,” or “non-English speech is being rendered with the wrong cadence.”

from playwright.sync_api import sync_playwright

print(samples)

from playwright.sync_api import sync_playwright

print(samples)

from playwright.sync_api import sync_playwright

print(samples)

In practice, you will usually want a more robust signal than variance alone. If you have access to landmarks or segmentation, use them. If not, a narrow mouth-region crop plus temporal smoothing is often sufficient for regression testing.

Aligning the visual trace with the audio trace

The important metric is offset, not raw motion. A test should estimate when speech starts and compare that to when the mouth starts moving. For a healthy realtime pipeline, those events should be close, with a small lead or lag depending on how the renderer buffers frames.

One practical method is:

1. Record the audio start time from your agent or playback event.

2. Sample the mouth score every N milliseconds.

3. Detect the first sustained rise above baseline.

4. Compute lip_sync_delay_ms = mouth_motion_start - audio_start.

Do this per utterance, then aggregate by language. You want a distribution, not one number. Median delay tells you the typical case; p95 catches buffering or scheduling problems. If you compare languages, keep the prompts structurally similar in length and syllable complexity so you are measuring timing behavior rather than content differences.

A useful regression threshold is often relative, not absolute. For example: “Spanish and Japanese should not be more than 120 ms worse than English on the same rendering build.” The exact threshold depends on your avatar motion style and the quality tier you are using, but the key is to compare like-for-like across builds.

Language-specific gotchas that usually show up in testing

Multi-language lip-sync issues are often caused by layers outside the renderer:

• Text normalization: numbers, dates, and abbreviations may expand differently and change timing.

• Grapheme-to-phoneme behavior: some TTS stacks depend heavily on language metadata, and the wrong locale can distort timing.

• Syllable density: some languages compress more speech into the same wall-clock time, exposing latency in the motion pipeline.

• Frame pacing: a stable 30 fps render can still look wrong if viseme state updates arrive irregularly.

• Client buffering: the browser may be the source of the delay, not the avatar service itself.

When a test fails, try to localize the defect: is the audio delayed, is the first viseme late, or does the avatar keep moving but not match the phonetic rhythm? That distinction tells you whether to debug the model, the transport, or the frontend.

Automating the test across languages

For a useful suite, define a small set of prompts that are intentionally different across languages but similar in duration. Keep them short enough to run in CI and long enough to contain multiple mouth transitions. For each language, store:

• the prompt text,

• expected approximate duration,

• the acceptable delay window, and

• a baseline sample for visual comparison.

Then run the same browser harness for each language, capture the mouth score trace, and compare against your baseline. If you want a stronger signal, compute a correlation between the audio envelope and mouth score over time. Even a simple cross-correlation can catch drift that is hard to see by eye.

Here is the kind of test logic I would keep in a repo:

LANG_TESTS = [

pass

LANG_TESTS = [

pass

LANG_TESTS = [

pass

Keep the baseline data versioned with the avatar build. If you change the rendering model, frame rate, or voice provider, rerun the suite and update the thresholds deliberately rather than letting drift accumulate.

Where Protoface fits

This is exactly the sort of pipeline where a developer-facing avatar platform helps. With Protoface, you can create and manage avatars and realtime sessions via the REST API or use the Python SDK to drive tests programmatically, then open the session in a browser and inspect the actual streamed result. That lets you test the same production surface your users will see, instead of approximating it with a local mock.

If you are building around the Python SDK, the flow is straightforward: create a session, point Playwright at the session URL or embedded page, and collect frames while the agent speaks. Exact request and response fields are in the docs, but the shape is familiar:

import requests

print(session)

import requests

print(session)

import requests

print(session)

If your avatar is attached to a voice agent, the same test harness can be driven through the agent layer rather than manually pushing audio. For LiveKit-based agents, the plugin path is useful because it exercises the integrated streaming path end to end. If you want to see the implementation style, the examples in the public quickstarts are a good reference, and the API details are in the docs.

For teams using a browser embed, Playwright is especially valuable because it tests the real iframe or page integration, including timing issues introduced by the host app. The browser is often where lip-sync bugs become visible, even when upstream audio generation looks correct.

Conclusion

Multi-language lip-sync testing is mostly a systems problem: you are validating timing across the speech stack, the transport, and the browser renderer. A good test harness samples the avatar’s mouth motion over time, aligns it with speech start, and compares the result across languages and builds.

If you keep the viewport stable, capture frames consistently, and measure delay instead of relying on subjective review, you can catch regressions early and with very little manual effort. Start with a handful of representative prompts, add thresholds per language, and gate releases on the metrics that matter.

For implementation details, session APIs, and SDK usage, see the documentation. If you want a concrete integration path, the Python SDK repo and the quickstarts linked from the public docs and GitHub organization are the fastest way to wire a test harness into your own avatar pipeline.