Reducing Latency in Unreal Engine Talking Avatars: End-to-End System Design for STT, LLM, TTS, and Animation
Unreal Engine talking avatars: low-latency STT, LLM, TTS, and lip-sync pipeline design with streaming, buffering, and instrumentation.
Introduction
Reducing latency in a talking-avatar stack is mostly a systems problem, not a model problem. Once you add speech-to-text (STT), an LLM, text-to-speech (TTS), and video lip-sync/animation, the user experience is bounded by the slowest stage and by how much of the pipeline you force to run sequentially. A good realtime avatar feels responsive because it starts producing useful output early, keeps the pipeline streaming, and avoids unnecessary buffering between stages.
This post walks through an end-to-end design for a low-latency avatar pipeline in Unreal Engine: how to budget latency, how to overlap STT/LLM/TTS/animation work, what to cache, what to stream, and where the common bottlenecks hide. By the end, you should be able to reason about latency at each hop, instrument the right metrics, and make sensible trade-offs between responsiveness, realism, and cost.
Think in latency budgets, not features
For interactive voice avatars, “fast” is usually perceived in the 200-800 ms range for first visible response, with continued audio/video streaming after that. You do not need the whole answer before you start animating. You need:
low input capture latency from microphone to STT
partial transcript delivery early enough to start intent detection
incremental LLM token emission
streaming TTS that can begin audio before the full response is synthesized
avatar animation that starts on the first audio frames, not after the full clip
The critical mistake is serializing the whole chain. A naive implementation waits for complete STT, then complete LLM text, then complete TTS audio, then a full animation packet. That guarantees high latency even when each individual service is “fast.”
A better mental model is a streaming DAG:
Each arrow should be as asynchronous as possible. If your architecture forces the Unreal game thread to wait on network round trips, you will feel that latency immediately in the avatar.
Stage 1: capture and STT without blocking the rest of the pipeline
Start with the microphone. The audio path should be continuous, jitter-tolerant, and isolated from frame timing. In Unreal Engine, do not tie capture, resampling, and upload to the render thread. Put audio acquisition on a dedicated worker, normalize to the format your STT provider expects, and stream chunks as soon as they are available.
Voice activity detection (VAD) matters because it reduces useless traffic and gives you a clean notion of utterance boundaries. But VAD should be conservative. If you wait too long for end-of-utterance silence, you add latency. A common compromise is:
stream audio continuously
use VAD to mark likely turn boundaries
allow the STT service to emit partial hypotheses during speech
finalize the utterance only when confidence is high enough
The key metric is not just transcription accuracy. Measure:
time from last spoken phoneme to first partial transcript
time from first user audio to final transcript
percentage of utterances that require correction after the LLM has already started responding
That last metric matters because if you let downstream stages consume unstable transcripts too aggressively, you create awkward self-corrections. For voice agents, it is often better to start the LLM on a partial transcript plus a small confidence threshold, then revise only when the transcript stabilizes.
Stage 2: make the LLM stream tokens that are actually useful
Once you have partial or final STT, the fastest path is to generate response tokens incrementally and hand them downstream immediately. The important distinction is between “token streaming” and “response buffering.” Streaming only helps if you propagate each chunk as soon as it is semantically usable.
In practice:
keep the system prompt short and stable
avoid inserting large conversation history unless you truly need it
trim tool-call latency by moving expensive external calls off the critical path
prefer short, direct responses for the first speaking turn
If the model is going to call tools, decide whether the avatar should speak before the tool result arrives. For example, “Let me check that” can be emitted immediately, while the actual answer can follow after the tool returns. This reduces perceived latency even when total backend work is unchanged.
For Unreal-based products, the practical implication is that the avatar should not wait for a full semantic sentence. It can begin speaking and animating on the first phrase, then continue as the LLM streams more text. That means your speech controller needs to accept incremental text deltas and support interruption if the user barges in.
Stage 3: streaming TTS and phoneme timing are the real latency lever
TTS latency is often where avatar stacks get painful. If you synthesize a full utterance before playback, you sacrifice responsiveness. Use a TTS service that supports streaming audio output and, ideally, timing metadata for phonemes, visemes, or word boundaries.
There are two separate problems here:
audio start latency: how soon after text arrives can the first audio frame play?
animation timing quality: how precisely can the mouth shapes follow the speech?
For the first one, the goal is to start playback on the earliest stable audio chunk. For the second, you need a timing signal. You can drive the face from:
phoneme timestamps from TTS, if available
real-time viseme classification from audio
a fallback rhythm-based mouth-open/close heuristic when timing metadata is absent
Phoneme-accurate lip sync looks best, but it is not always necessary to get a convincing result. For a realtime system, a small amount of approximation can be better than waiting for perfect data. If your animation arrives 300 ms late, the user notices the delay more than they notice a slightly imperfect viseme.
One practical pattern is to generate an audio timeline in advance while the audio buffer is already playing. That lets Unreal interpolate jaw and mouth shapes locally without blocking on every network event. If your face rig includes blend shapes, keep a cached mapping from phoneme classes to morph targets so you are not solving that mapping per frame.
Stage 4: keep animation off the critical path
Unreal Engine should consume timing data, not create it. Your animation layer should be deterministic and lightweight: ingest audio or viseme events, update a small set of morph targets, and let the renderer do the rest. If you are doing facial animation on the game thread every frame with heavy parsing or expensive state transitions, you are reintroducing latency and jitter.
Design for asynchronous arrival of data. Network jitter and service variability mean you will receive:
audio packets out of phase with frame boundaries
partial text before final text
late viseme updates that need to be dropped or blended
interruptions when the user speaks over the agent
That implies a small local state machine for the avatar:
Idle — waiting for a turn
Listening — capturing input and running VAD
Thinking — STT/LLM in progress, subtle idle motion only
Speaking — audio playback plus lip sync
Interrupted — abort current speech and return to listening
That state machine should be explicit. It is the simplest way to keep animation, audio, and dialog policy from fighting each other. If the user interrupts the avatar, stop the TTS stream, cancel any queued viseme events, and move immediately back to listening. Half-duplex behavior feels broken in conversational UI.
Where the bottlenecks usually are
Most latency surprises come from plumbing, not ML inference. Watch out for:
buffering at every hop — audio buffers, HTTP chunks, WebSocket queues, and render-thread queues can each add tens of milliseconds
overly large frame sizes — bigger audio chunks reduce network overhead but increase first-byte latency
blocking serialization — turning streaming events into full JSON blobs before forwarding them
thread contention in Unreal — work that touches the game thread during playback or capture
cold starts — model warmup, container spin-up, or lazy asset loading at the wrong moment
A useful rule: every stage should be able to start producing something before the previous stage has completely finished. If that is not true, you probably have an avoidable synchronization point.
Instrumentation is not optional. Log timestamps for:
audio capture start
first audio chunk sent
first STT partial
final STT
first LLM token
first TTS audio chunk
first rendered mouth movement
first audible playback
Once you have those numbers, the slowest link becomes obvious.
How Protoface fits into this pipeline
If you do not want to build the avatar transport, session management, and lip-sync plumbing yourself, Protoface provides the avatar side of the stack as a developer-facing realtime API. For Unreal Engine teams, the practical integration point is usually not the engine itself but the voice-agent layer that feeds the avatar.
For example, the LiveKit Agents plugin can drop a synchronized talking face into an existing agent so the audio stream and avatar stay aligned without you hand-rolling a separate timing channel. If you are already building your dialog system in Python, the SDK is the direct way to create or manage avatars and sessions. If you need a web surface instead of a native client, customer-managed iframe embeds isolate the browser from your API key and let you control per-embed instructions and limits.
When you are wiring this up, the main benefit is that you can focus on optimizing your STT/LLM/TTS latency budget rather than spending time on session orchestration and sync edge cases. The implementation details live in the docs, and the quickstarts are a good way to validate end-to-end behavior quickly: docs.protoface.com and github.com/protoface-ai.
Illustratively, creating a session through the REST API looks like this pattern:
The exact fields and response shape are documented in the API reference, but the point is the same: keep your app logic thin, and let the avatar service handle the realtime session lifecycle.
Conclusion
Low-latency talking avatars are built by removing unnecessary serialization, streaming aggressively, and keeping animation state local and lightweight. In practice, the big wins come from partial STT, token-streaming LLMs, streaming TTS, and a face rig that can animate from incremental timing data without waiting for a perfect full utterance.
If you are building this in Unreal Engine, start by instrumenting each stage and making one improvement at a time. Get first-audio latency down, then improve lip-sync timing, then reduce interruption lag. Once the pipeline is stable, you can decide whether to own every component or plug into an existing avatar layer. For implementation details, see the docs at docs.protoface.com.