Debugging WebRTC Audio and Video Issues in an Unreal Engine Triage Avatar
Debugging Unreal Engine WebRTC avatars: isolate signaling, ICE/DTLS, audio/video decode, render timing, and lip-sync drift.
Introduction
When a realtime avatar looks “broken,” the symptom is usually not the real bug. A frozen mouth can be a media track that never negotiated. Out-of-sync lips can be packet jitter, mismatched sample rates, or a video pipeline that is rendering late frames. A silent session can be an SDP or ICE problem that masquerades as an application issue. In an Unreal Engine triage avatar, you are dealing with all of that at once: WebRTC transport, audio capture, video decode/render, and your own engine loop timing.
This post is a practical debugging guide. By the end, you should be able to isolate whether your issue is in signaling, transport, audio capture/playback, video ingestion/rendering, or the avatar generation layer itself, and then verify the fix with a repeatable checklist.
Start by separating signaling, transport, and media
The first mistake is treating “no audio/video” as one problem. In WebRTC, those failures come from different layers:
Signaling: Offer/answer exchange, auth, session setup, track metadata.
Transport: ICE candidate gathering, connectivity checks, TURN fallback, DTLS/SRTP.
Media: Actual audio frames and video frames arriving, being decoded, and being rendered.
For triage, ask three separate questions:
Did the session establish at all?
Did audio and video tracks get negotiated?
Did frames continue flowing after negotiation?
If you can, inspect the WebRTC peer connection state transitions directly. A session stuck in checking or failed is a transport problem. A session in connected with no audio is usually codec, track, or rendering. A session that starts well and degrades over time is often buffering, jitter, or engine-side starvation.
Read the logs like a transport engineer
For WebRTC audio/video issues, the fastest path is to collect the browser or native client logs plus the remote side logs at the same time. In an Unreal integration, also capture the game thread and render thread timing. You are looking for the boundary where media stops being healthy.
Useful signals include:
ICE state: if it never reaches connected/completed, your issue is before media.
DTLS state: if DTLS fails, SRTP never starts.
Inbound RTP stats: packets received, packets lost, jitter, concealed samples, frames decoded.
Renderer timing: frame queue depth, dropped frames, texture update latency.
For audio, distinguish “no packets” from “packets but no sound.” The first means transport or remote sender problems. The second means decode, sample-rate conversion, output device selection, or engine routing. For video, distinguish “no frames” from “frames but black screen.” The second often points to texture upload, pixel format mismatch, or rendering on the wrong thread.
Audio: verify source, sample rate, and output path
Audio bugs in realtime avatars often come from mismatched assumptions about the stream. One system emits 48 kHz Opus packets, another assumes 44.1 kHz PCM, and the result is drift, distortion, or silence.
When debugging audio in Unreal:
Confirm that the remote stream is actually arriving as audio packets.
Confirm the codec and sample rate expected by your decoder.
Confirm that decoded PCM reaches Unreal’s audio mixer or your custom output path.
Check whether audio is being consumed on the correct thread and not blocked by game-thread work.
Two common gotchas:
Clock drift: If your avatar lip-sync is driven by a different clock than audio playback, the mouth will slowly lead or lag. Use a single master clock for timestamp alignment when possible.
Buffering too aggressively: A large audio buffer hides jitter but increases latency. A tiny buffer reduces latency but makes glitches more likely. For triage, start with a moderate buffer and measure end-to-end delay before tuning.
If you are consuming audio in a voice-agent pipeline, make sure the agent’s TTS or STT layer is not the bottleneck. A “silent” avatar can simply be waiting on upstream text generation or blocked on downstream playback.
Video: check decode, frame pacing, and Unreal texture updates
Video issues in an avatar pipeline are usually not about “video” in the abstract; they are about a specific stage failing. The frame may be decoded correctly but never uploaded to a texture, uploaded but not sampled in the material, or rendered but not visible because the widget or actor is offscreen.
Debug in this order:
Track presence: confirm the remote video track exists.
Frame arrival: confirm decoded frames are arriving at a steady cadence.
Pixel format: confirm the frame format matches what your Unreal texture update code expects.
Render path: confirm the texture/material/mesh is actually used by the avatar actor.
For Unreal specifically, watch out for thread boundaries. WebRTC callbacks may arrive on a network or worker thread, while texture updates and many render-facing operations need to be marshaled to the appropriate engine thread. If you update a texture from the wrong thread, you may get intermittent glitches instead of an obvious crash.
Another subtle failure mode is frame pacing. If the avatar face renders at 30 fps while audio is steady and the game is running at 120 fps, the visual can look “laggy” even though nothing is technically broken. The fix may be to decouple frame arrival from render rate and only upload the latest decoded frame, not every queued frame.
Lip sync is usually a timestamp problem, not a “face” problem
When people say the avatar is “not lip synced,” the underlying issue is often that audio and video timestamps are not aligned closely enough. The avatar renderer may be using one stream of phoneme or viseme timing while the displayed video is driven by another timeline.
A good mental model is:
Audio gives you the authoritative playback timeline.
Viseme or facial animation events should be scheduled relative to that timeline.
Rendered video is a presentation of those events, not the source of truth.
If lip motion consistently leads audio, your animation is being scheduled too early or your audio buffer is too large. If it lags, the inverse is usually true. If the mouth opens and closes in the right rhythm but the timing is off by a variable amount, you likely have jitter or queue backlog.
In practice, the best debugging move is to print or log the timestamp of the audio packet, the frame presentation timestamp, and the local render time in the same units. Without that, you end up guessing at “latency” when the real issue is a one-line offset or clock mismatch.
Use reproducible session setup to isolate the bug
For a triage avatar, you want a minimal test case that you can run repeatedly with the same parameters. That means fixed voice, fixed instructions, controlled network conditions, and a known-good client. If the bug only appears in one environment, do not start by changing the avatar model or prompt. First verify that the session itself is healthy.
A simple workflow is:
Create a fresh session.
Connect one client with logging enabled.
Record the first 30 seconds of ICE state changes, inbound stats, and frame timing.
Compare against a known-good run.
For programmatic session setup, the REST API is useful because it lets you create the exact same conditions every time. Exact fields will depend on the current docs, but the shape is straightforward:
If you are building the avatar into a Python-driven workflow, the SDK is a better fit for repeatable triage scripts. A minimal example might look like this:
Use the SDK or API to eliminate UI variables. Once you can create and connect a controlled session reliably, it becomes much easier to tell whether your issue is in Unreal, the browser, the network, or the avatar session itself. See the docs for the current request and response shapes.
Where Protoface fits in this debugging loop
For a LiveKit-based voice agent, the relevant integration point is the Protoface plugin in the agent pipeline. The value of that plugin in triage is not magic; it is that it gives your agent a synchronized talking video face without forcing you to hand-wire the avatar session logic yourself. That reduces the number of places where audio/video state can drift.
If you are debugging an agent built with LiveKit, keep the avatar integration narrow and observable: log when the avatar session starts, when tracks are attached, and when the remote media states change. The plugin source and examples in the repository are useful for understanding where tracks are created and how the avatar is attached to the agent flow: GitHub repository.
That said, the same core debugging rules still apply. Whether the avatar is coming from a plugin, a direct REST-created session, or a local Unreal playback path, the failure modes are the same: negotiation, transport, decode, render, timing. The benefit of using a managed avatar layer is that you can separate those concerns more cleanly and test them independently.
Conclusion
Most WebRTC avatar bugs are not mysterious. They are layered failures that look similar from the outside. If you separate signaling, transport, audio decode/playback, video decode/rendering, and lip-sync timing, the problem usually becomes obvious quickly.
For Unreal Engine triage avatars, the practical checklist is simple: verify ICE/DTLS first, inspect inbound stats second, confirm audio and video frames are actually arriving, then validate the engine-side render path and timestamp alignment. Use a reproducible session setup so you can compare a broken run against a known-good one.
If you want the exact API shapes, SDK methods, and integration details, start with docs.protoface.com. If you are wiring this into an agent pipeline, keep the integration minimal, instrumented, and easy to isolate.