Debugging Audio, Video, and Lip-Sync Issues in a TypeScript Tutor Avatar
TypeScript guide to debugging avatar audio, video, and lip-sync drift: timing, buffering, sample rates, and session state.
Introduction
When a tutor avatar starts talking but the video lags behind, the lip movements look “almost right” but never quite line up, or the audio drifts after a minute of conversation, the bug is usually not in one place. Realtime avatar systems couple three separate pipelines: text or token generation, speech synthesis, and video render or facial animation. Any mismatch in timing, buffering, sampling rate, or state management will show up as an audio/video synchronization problem.
This post is a practical debugging guide for TypeScript teams building a tutor avatar. By the end, you should be able to isolate whether the issue is in the agent, the transport, the media pipeline, or the avatar session itself, and then apply the right fix instead of guessing.
Start by classifying the failure mode
Before changing code, decide which class of bug you are seeing. The fix for each is different.
No audio, video visible. Often a permissions, device, codec, autoplay, or track-subscription issue.
Audio plays, face is frozen. Usually the video track is not attached, session state is stale, or the avatar stream is not being consumed.
Lip sync is consistently late or early. Timing mismatch between speech and video, often due to buffering, TTS chunking, or timestamp drift.
Sync starts correct but degrades over time. Clock drift, accumulated queue latency, or a backlog in the rendering pipeline.
That classification matters because “fix the delay” can mean anything from changing a sample rate to reducing end-to-end buffering by 200 ms.
Understand where sync is actually established
In a realtime avatar stack, lip sync is not magically inferred from the browser. It is established upstream by coordinating the speech waveform or phoneme stream with the video generation pipeline. If your agent produces text, then TTS, then avatar video, the avatar service needs a deterministic notion of what audio segment corresponds to what mouth motion.
The common places for timing to break are:
Text chunking. If you stream partial assistant text too aggressively, the TTS engine may emit tiny audio fragments that create visible micro-pauses.
Audio resampling. Mismatched sample rates or implicit transcoding can shift timing enough to produce apparent desync.
Transport buffering. WebRTC, WebSocket, or media relays may buffer audio and video differently.
Client render latency. The browser may receive both streams on time but paint the video late because the main thread is busy.
A useful mental model: lip sync is only as good as the slowest stage that participates in timestamping. If you cannot trust the timestamps, you cannot trust the visual alignment.
Debug audio first, then video, then sync
When troubleshooting, separate media transport from avatar logic. Do not assume a single “avatar bug” if the underlying audio stream is already broken.
1) Verify the audio track independently
Confirm that the browser is actually receiving and playing the assistant audio. Check for these issues:
Autoplay policy blocks audio until user interaction.
Muted output element or disabled audio sink.
Incorrect output device selection in the browser.
Silent audio due to upstream TTS failure or empty utterances.
If you are using WebRTC, inspect the remote track state, packet loss, jitter, and concealment. If packet loss is high, the avatar may look “wrong” simply because the audio reference is degraded.
2) Verify the avatar video track independently
If the face is frozen but audio continues, confirm that the video track is subscribed and rendering. In many cases, the track exists but is not attached to the DOM, or the component unmounted and remounted without re-subscribing.
For browser-side debugging, keep an eye on:
Track mute/unmute events.
Lifecycle changes when switching conversations or tabs.
Visibility changes that pause rendering.
Any “optimization” that detaches hidden video elements.
If you’re rendering in a React component, the most common failure is stale state: the track object changed, but the effect that attaches it did not rerun.
3) Measure the actual end-to-end delay
To debug lip sync, measure from a single user-visible event to the corresponding mouth movement. For example, log the moment an assistant token is emitted, the moment TTS begins streaming, and the moment the avatar frame changes. If those timestamps are only available on the client, you can still estimate:
If the gap between audio start and visible mouth movement is stable, you likely have a fixed pipeline delay. If the gap grows over time, look for buffering or a queue that is not being drained fast enough.
Common timing bugs in TypeScript clients
TypeScript itself is rarely the root cause, but client architecture often is.
Stale closures and repeated subscriptions
A React or Node event handler may capture an old session object. That leads to audio being published to one session while the UI listens to another. The symptom is maddeningly inconsistent: reconnecting “fixes” it for a few minutes.
Make sure you unsubscribe and re-subscribe on session identity changes, not just component mount. Log the session ID everywhere you route media.
Queueing too much before playback
For a tutor avatar, latency matters more than perfect sentence batching. If you wait for an entire assistant response before starting TTS, you add avoidable delay and make the avatar look robotic. If you stream too tiny a chunk, you increase overhead and may fragment the audio into unnatural bursts.
The practical sweet spot is usually to stream at phrase boundaries, not token boundaries. If your agent framework emits tokens, aggregate them into short, semantically coherent chunks before sending them downstream.
Sample-rate and format mismatches
If the speech engine emits PCM at one rate and the media pipeline expects another, resampling can introduce drift or latency. This is especially relevant if you mix browser audio, server-side TTS, and a WebRTC transport. Be explicit about sample rate, channel count, and encoding at every boundary.
Do not assume the browser or SDK will silently normalize everything in the exact way you want. “Works on my machine” often means “my browser happened to accept the format.”
Network and transport issues that look like lip sync bugs
In realtime systems, transport problems often masquerade as media bugs. The avatar may be perfectly synchronized at the sender but arrive late or unevenly at the client.
Jitter: uneven arrival times force buffering.
Packet loss: missing audio causes concealment and perceived drift.
Congestion: the video stream gets deprioritized relative to audio, or vice versa.
Reconnects: stale frames from an old session can appear briefly after the new session starts.
If the problem only appears on poor networks, the avatar is not the bug; your buffering strategy is. Keep the UX responsive by reducing unnecessary client-side buffering and handling reconnects as first-class state transitions.
A practical debugging checklist
When a tutor avatar goes off the rails, work through this sequence:
Confirm the assistant actually generated output.
Confirm audio is arriving, unmuted, and decodable.
Confirm the video track is present and attached to the UI.
Log timestamps at each boundary: text, TTS start, audio packet, video packet, render.
Check sample rates and codecs across the whole path.
Look for buffering, stale subscriptions, and reconnect behavior.
Test with a single short utterance before testing long conversations.
Short utterances are especially valuable because they minimize queueing noise. If one sentence is synced and the next is not, you are probably dealing with accumulated latency rather than a fundamental media incompatibility.
Using the LiveKit plugin to isolate the avatar layer
If your app already uses LiveKit for the voice agent, the cleanest way to isolate avatar issues is to drop in the Protoface LiveKit plugin and remove unrelated UI code from the equation. The plugin gives the agent a synchronized talking video face without forcing you to build a separate media bridge in TypeScript. The point is not to hide the problem; it is to narrow the surface area while you debug.
For example, once the LiveKit side is stable, you can focus on whether the issue lives in your agent logic or in your browser rendering. The plugin is published as livekit-plugins-protoface on PyPI, and the repository includes examples you can compare against your integration. See the repo at https://github.com/protoface-ai/protoface-plugin-pipecat and the Pipecat guide at https://docs.pipecat.ai/api-reference/server/services/video/protoface if your stack uses Pipecat.
That sort of session creation is useful for isolating problems because you can reproduce with a controlled avatar, controlled voice, and controlled instruction set. The exact SDK fields are documented, but the principle is the same: reduce variables until the bug becomes obvious.
When to use the REST API directly
If you suspect the issue is not media but session lifecycle, use the REST API to create, inspect, and tear down sessions in a deterministic way. That helps rule out stale frontend state, duplicated sessions, or orphaned resources. The API is also the right place to verify that the avatar you think you launched is the avatar actually active.
Use the docs at https://docs.protoface.com for the exact request and response shapes. For debugging, the important part is not the specific payload; it is the ability to create a known-good session and compare it against the one that misbehaves.
Conclusion
Audio, video, and lip-sync problems in realtime avatars are usually timing and state bugs, not “AI” bugs. Debug them by isolating the media path, measuring latency at each boundary, checking for buffering and resampling issues, and verifying session lifecycle before you blame the renderer.
If you want a controlled way to reproduce and narrow these issues, start with the relevant integration surface for your stack, then compare your implementation against the documented quickstarts and examples. From there, the remaining work is usually straightforward instrumentation and queue management. The docs at https://docs.protoface.com are the right place to confirm the exact API shapes and integration details.