Debugging Lip-Sync and Audio Drift in a Next.js Realtime Avatar Assistant
Debugging lip-sync and audio drift in a Next.js realtime avatar assistant: measure media timing, isolate clock skew, and fix browser playback sync.
Introduction
When a realtime avatar looks “almost right” but the mouth is consistently late, the audio arrives before the face, or everything drifts out of sync after a minute, the bug is usually not in the avatar model itself. It’s in the timing model around it: capture, buffering, transcoding, transport jitter, playback latency, and how your app stitches those pieces together.
This post is for developers building a Next.js realtime avatar assistant and trying to debug lip-sync and audio drift in the browser. By the end, you should be able to isolate whether the problem is in generation, transport, or playback; measure A/V skew with enough precision to be useful; and apply the right fix instead of tuning blindly.
Start with the pipeline, not the symptom
Realtime avatars are usually a chain of independently timed systems:
User audio is captured in the browser or received from a voice stack.
Speech is decoded or transcribed, then sent to an agent or media pipeline.
The agent generates audio and avatar motion, often with timestamps or frame pacing constraints.
Video frames and audio packets are delivered over WebRTC or another streaming transport.
The browser decodes and schedules both streams for playback.
Lip-sync errors come from mismatches between those stages. “Audio drift” usually means two clocks are not actually aligned: the audio clock, the video clock, and sometimes a server-side generation clock are all drifting relative to each other.
The debugging rule is simple: identify where the skew first appears. If the avatar is correct in a server-side preview but wrong in the browser, don’t waste time tuning the model. If the delay grows linearly over time, you are likely accumulating buffer imbalance or timestamp error. If it jumps after reconnects or tab throttling, you likely have a scheduling or latency-resync issue.
Measure the problem before changing code
Most teams try to eyeball the issue from a screen recording. That is useful for confirmation, not diagnosis. You want timestamps.
In practice, track at least these values:
Speech onset time: when the first audible sample for a turn is actually played.
First mouth movement time: when the first visible change appears in the avatar.
Frame-to-audio offset: the difference between the video frame timestamp and the audio playback time.
Drift over time: whether the offset stays bounded or increases.
For browser-side instrumentation, use the Web Audio clock for audio playback timing and compare it to video frame presentation. If you render video into a <video> element, the media element’s clock is not the same thing as your JavaScript event loop. That distinction matters: timers in the main thread are not precise enough for sync debugging.
A minimal pattern is to log both media time and wall time whenever the browser receives a chunk or frame:
If you are using WebRTC, also inspect jitter buffers, packet loss, and RTT in the browser stats API. A steadily growing audio delay often correlates with the receiver trying to absorb unstable delivery by buffering more aggressively.
Common causes of lip-sync errors in Next.js apps
Next.js is usually not the source of the timing bug, but it can make the bug easier to trigger. The most common failure modes I see are:
Double buffering: you buffer audio in the app, then the browser buffers again, and the video path uses a different buffer depth.
Unsynchronized clocks: one stream is scheduled from server timestamps, the other from client arrival time.
Main-thread contention: rendering, state updates, and heavy React work delay frame handling while audio keeps playing.
Autoplay and resume issues: audio starts only after a user gesture, while video may already be advancing.
Reconnect resets: the session resyncs on one channel but not the other.
Watch for browser scheduling traps
In a Next.js client component, it’s easy to accidentally build a sync scheme around setState, setInterval, or “arrived first, played first.” Those are not reliable clocks. If you need mouth motion to track audio, the video side should be driven by explicit media timestamps or a dedicated sync layer, not by React renders.
Two practical constraints matter a lot:
Do not infer playback start from network arrival. A frame arriving first does not mean it should render first.
Do not align avatar frames to animation frames alone. requestAnimationFrame is tied to paint cadence, not media timing.
Also check whether your browser tab is being background-throttled. If your app queues a burst of UI work after the tab returns to the foreground, the avatar can appear to “jump” while audio continues with less interruption.
Fix drift by choosing one timing authority
Drift usually happens when your system mixes timing authorities. Pick one source of truth and make the other stream follow it.
There are two common approaches:
Audio is authoritative: schedule video frames to match audio playback time, dropping or duplicating frames as needed.
Server timestamps are authoritative: both audio and video are emitted against the same turn timeline, and the client honors those timestamps during playback.
For realtime avatars, the second model is usually cleaner because speech synthesis and facial motion are generated together. But whichever model you use, the rule is the same: do not let client arrival order determine sync.
If drift accumulates gradually, inspect whether your pipeline is resampling audio inconsistently. A mismatch between capture sample rate, synthesis sample rate, and playback sample rate can create subtle offset growth or periodic corrections. In browser code, the common symptom is a stream that sounds fine at the beginning and progressively separates from the face.
What to test when the bug seems random
Timing bugs often look random because they depend on network jitter, device load, and tab state. Reproduce under controlled conditions:
Throttle the network and see whether the offset is stable or growing.
Open DevTools performance profiling and check for long tasks on the main thread.
Test in a fresh profile with hardware acceleration both enabled and disabled.
Try a simple page without your app shell to remove unrelated React work.
Record one short turn and one long turn; long turns expose drift more clearly.
When you change something, change one variable at a time. If you tweak buffer sizes, autoplay logic, and transport settings simultaneously, you will not know which change fixed the sync.
How Protoface helps in this specific part of the stack
If you are using a live voice agent stack, the cleanest way to avoid ad hoc avatar timing logic is to attach the face where the voice agent already exists. The LiveKit Agents plugin in the Protoface ecosystem is built for that use case, so the avatar stays coupled to the agent’s realtime turn flow instead of being driven by a separate browser-side animation loop. The plugin repo has the integration surface and examples: GitHub repository.
For browser embedding, the customer-managed iframe model is even stricter: the avatar session is isolated, the API key never enters the browser, and the sync logic stays inside the embed rather than in your Next.js app. That removes a whole class of timing mistakes caused by local scheduling code.
If you need to create or inspect sessions directly, use the REST API from a backend and keep the browser as a thin player. A typical call looks like this:
The exact request fields depend on the object you are creating, so check the docs for the current shape and quality-tier behavior: documentation.
A pragmatic debugging checklist
When the avatar is out of sync, I usually work through this order:
Confirm whether the offset is constant or grows over time.
Determine whether the skew appears before transport, after transport, or only after browser playback.
Log media timestamps instead of relying on render timing.
Remove React re-renders and UI work from the critical path.
Make one stream follow the other, rather than letting them free-run.
If you can answer “where does the offset begin?” and “which clock is authoritative?” you will usually find the bug quickly.
Conclusion
Lip-sync and audio drift are almost always clock and buffering problems, not mysterious avatar bugs. In a Next.js realtime assistant, the fix is to measure actual media timing, remove assumptions about arrival order, and keep the browser out of the job of inventing sync.
If you want a cleaner integration path for realtime avatars, start with the relevant Protoface surface for your stack, then verify behavior with a small, instrumented test page. The docs at docs.protoface.com are the best place to confirm the current API shapes, SDK usage, and integration details before you wire it into production.