How to Sync Lip Movement, Audio, and Facial Animation in a Realtime Unreal Engine Avatar
Learn Unreal Engine avatar lip sync with timestamped audio, visemes, buffering, and facial animation timing.
Introduction
Syncing a realtime avatar is mostly a timing problem. You need the audio stream, the facial animation stream, and the transport layer to agree on when each phoneme, jaw movement, eye blink, and head motion should happen. If those signals drift even a little, the result feels uncanny: lips arrive early, the face “snaps” between expressions, or audio and video disagree under jittery network conditions.
This post explains the practical architecture for keeping a realtime Unreal Engine avatar aligned with streaming audio and facial animation. By the end, you should have a clear mental model for how to structure the pipeline, what to timestamp, where to buffer, and how to avoid the common failure modes that show up in production.
Start with one clock, not three
The first rule is simple: pick a single timeline and map everything to it. In realtime avatar systems, audio is usually the master clock because human perception is most sensitive to lip sync. Facial animation and metadata should be derived from the same presentation timeline, not generated independently and “best-effort” synchronized after the fact.
In practice that means your pipeline should carry timestamps for:
audio capture or synthesis start time
frame presentation time for video or facial animation updates
segment boundaries for phonemes, visemes, or expression events
For an Unreal avatar, the main decision is whether you are driving a skeletal face rig, a morph-target rig, or a MetaHuman-style facial setup. The rendering tech changes, but the synchronization principle does not: animation should be sampled from time, not from arrival order.
How realtime lip sync actually works
There are three common ways to drive mouth movement:
Audio-driven visemes. Analyze the audio stream and estimate mouth shapes from amplitude and spectral cues.
Transcript-driven timing. Use the generated text and align it to speech timing, usually with TTS metadata or forced alignment.
Hybrid animation. Combine TTS timing, visemes, and procedural motion for jaw, tongue, eyes, and brows.
For realtime agents, hybrid is usually best. Audio-driven lip sync alone can work for rough conversation, but it struggles with plosives, fast speech, and low-latency streaming where you don’t yet have the full utterance. Transcript-driven timing gives better mouth shapes, but only if your speech system exposes usable timing metadata. If not, you end up approximating with buffered audio and heuristics.
The Unreal side should consume a compact animation payload, not raw audio analysis. A useful payload typically includes:
timestamped viseme weights
emotion or expression state
idle motion parameters
an utterance or segment id for replay/debugging
Then Unreal interpolates locally between keyframes. That keeps the face moving smoothly even if packets arrive slightly late.
Buffering is not latency; it is damage control
If you try to render every packet immediately, network jitter will show up as visible facial jitter. The fix is a small playout buffer. For avatar video and face animation, a buffer of a few frames is often enough to smooth transient jitter without making the interaction feel detached.
The usual trade-off is:
smaller buffer = lower latency, more risk of stutter
larger buffer = smoother motion, more end-to-end delay
For conversational agents, you generally want low hundreds of milliseconds end-to-end, but you can afford a little more latency in facial animation than in audio because users are more tolerant of slight visual delay than speech delay. Still, if the mouth is visibly behind the audio, the interaction feels broken fast.
Two implementation details matter:
Clock skew correction. The client clock and server clock will drift. Use timestamps relative to the media stream and periodically resync.
Late packet handling. If a viseme frame misses its display deadline, skip it or blend it into the next one instead of applying it out of order.
Unreal Engine integration: separate transport from animation
In Unreal, keep transport concerns away from the animation graph. The network layer should ingest session events, audio chunks, and animation metadata. A separate component should convert those events into a normalized state that the face rig can evaluate every tick.
A practical architecture looks like this:
Open a realtime session and receive audio plus face metadata.
Decode or stream the audio into Unreal’s playback path.
Store incoming viseme/expression keyframes in a time-ordered queue.
On each frame, sample the queue at the current playback time.
Apply the sampled values to morph targets, joints, or control rig parameters.
The key point is that the animation tick should read from a buffered timeline, not from the raw socket callback. Socket callbacks are about transport. The tick is about presentation.
Example: a minimal session-driven animation loop
The exact session schema depends on your service, but the shape is usually the same. You create a session, receive media/control events, and feed them into your runtime. Here is an illustrative Python example using the developer SDK to create a session and inspect the returned data:
On the Unreal side, the corresponding logic is usually:
That sampling step is what keeps the mouth aligned even when events arrive unevenly. If you are debugging sync, log both the media timestamp and the render timestamp. Most “lip sync bugs” are really timestamp bugs.
Handling facial motion beyond the mouth
Users notice lip sync first, but a convincing avatar also needs blink timing, gaze shifts, brow motion, and small head movements. These should not be random noise layered on top. They should be correlated with speech state.
Some practical heuristics:
Increase blink rate during pauses, not during high-articulation segments.
Drive subtle head motion from phrase boundaries, not individual phonemes.
Keep gaze motion slow enough to avoid looking like a tracking error.
Clamp facial amplitude so enthusiasm does not flatten into constant over-animation.
In other words, animate at multiple time scales. Lip movement is frame-level. Gestures are phrase-level. Eye motion is conversational and probabilistic. If you collapse everything into the same update frequency, the face looks mechanical.
Where Protoface fits
If you do not want to build the session orchestration, media transport, and timing layer yourself, Protoface provides a developer-facing realtime avatar API plus a LiveKit plugin that drops a synchronized talking face into a voice agent. That is useful when your core product is the agent logic, not the media plumbing.
For a LiveKit-based stack, the plugin is the cleanest integration point because it keeps the avatar in the same realtime session as the agent. The plugin package is published on PyPI as livekit-plugins-protoface, and the corresponding repo with examples is available on GitHub. The workflow is straightforward: create or select an avatar, connect the agent session, and let the plugin handle synchronized face playback alongside the audio stream.
If you prefer to manage sessions directly, the REST API at api.protoface.com is the control plane. A typical request looks like this:
The important part is not the exact payload shape, which is documented elsewhere, but the model: treat avatar creation and session management as separate from presentation. That separation makes it easier to swap transports, add retries, and keep your Unreal client focused on rendering.
Common gotchas
Three bugs show up repeatedly in production:
Out-of-order events. If you apply animation packets as they arrive, a late packet can visibly rewind the mouth. Always sort by timestamp or sequence number.
Audio/video drift. If the audio clock and the face clock are not derived from the same session timeline, the error accumulates over time. Resync periodically.
Over-buffering. Adding too much delay to “fix” stutter makes the avatar feel disconnected. Keep the buffer small and the interpolation smooth.
Also remember that Unreal’s frame rate is not your media clock. A 30 FPS render loop and a 48 kHz audio stream are different systems. Bridge them explicitly.
Conclusion
Realtime avatar sync is mostly about disciplined timing: one authoritative timeline, timestamped animation data, a small playout buffer, and local interpolation in the renderer. Once you structure the system that way, lip movement, audio, and facial animation become predictable instead of fragile.
If you are implementing this yourself, start by instrumenting your timestamps and writing the animation loop so it samples from audio time. If you want a faster path into a production-ready stack, review the docs at docs.protoface.com and the related quickstarts in the repository. The quickest way to spot sync issues is still the old-fashioned way: log everything, render a test utterance, and watch where the timestamps drift.