Guide to Syncing STT, TTS, and Lip-Sync for Conversational Game NPCs
How to sync STT, TTS, and lip-sync for game NPCs with streaming text, audio timing, interruption handling, and low-latency avatars.
Introduction
Building a conversational NPC that speaks, reacts, and visibly “keeps up” with the player is mostly a synchronization problem. You have three independent streams to coordinate:
text generation from the LLM,
audio generation from TTS, and
mouth/face motion from lip-sync or avatar video rendering.
If any one of those lags behind, the illusion breaks. The player notices when the mouth starts too early, speech continues after the response has logically ended, or the face keeps moving during silence. By the end of this post, you should be able to reason about the pipeline, choose a latency strategy, and wire a game NPC so text, audio, and facial motion stay aligned under real-world streaming conditions.
Start with the actual timing model
For conversational NPCs, the core unit is not a “message”; it is a sequence of timed events. In a typical turn:
The LLM starts streaming text tokens.
You decide when a segment is stable enough to send to TTS.
TTS begins streaming audio chunks.
The avatar or lip-sync engine consumes audio timing to drive visemes or video frames.
The important detail is that these steps are partially overlapped. You do not want to wait for a full paragraph before starting speech, but you also cannot speak every token as it arrives. In practice, you work with speech chunks or utterance segments that are short enough to feel responsive and long enough to avoid constant restarts.
Use a segmentation strategy, not raw token forwarding
The LLM stream usually needs a boundary detector. That detector can be as simple as punctuation plus a minimum token count, or as sophisticated as a prosody-aware segmenter. The goal is to send TTS content when it is unlikely to be revised.
A practical policy for NPC dialogue is:
buffer tokens until you see a clause boundary or sentence boundary,
enforce a minimum buffer duration, typically a few hundred milliseconds of text,
flush early if the agent is clearly yielding the floor or answering a short query,
avoid sending text that the LLM may still retract or rewrite.
This matters because once TTS begins, lip-sync systems often assume the text/audio pair is stable. If you keep mutating the text mid-utterance, the avatar will fall out of sync even if the audio is still technically correct.
Separate “generation latency” from “presentation latency”
Developers often optimize the wrong thing. The user experience is not determined only by how fast the model responds; it is determined by how quickly the NPC appears to respond. That means you should distinguish between:
Generation latency: time spent waiting on the LLM or TTS service.
Presentation latency: time until the player sees mouth movement and hears the first syllable.
For a game NPC, presentation latency is what matters. A good system usually starts the face animation on the first audio frame rather than waiting for the full sentence. That means your renderer should treat audio as the clock source for mouth movement, not the text stream.
Make audio the source of truth for lip-sync
If you are using a video face or an avatar that is driven by audio, the cleanest architecture is:
LLM produces text.
TTS produces audio.
Lip-sync uses audio timestamps, phoneme timing, or waveform-derived features.
That ordering is important. Text is useful for semantics, but audio is what the player actually hears, and audio timing is what keeps the mouth plausible. In other words, the face should follow the rendered speech, not the other way around.
If your stack supports it, forward the audio stream immediately and let the avatar engine render visemes incrementally. If the avatar service instead consumes text, make sure it has access to the exact text that was spoken and a stable mapping from text segments to audio segments. Otherwise, pauses, breaths, and truncations will drift.
Handle interruption, barge-in, and turn resets
Game NPCs are not linear playback devices. Players interrupt them, walk away, trigger another event, or ask a second question before the first answer is done. Your pipeline needs an explicit reset path.
At a minimum, define these events:
barge-in: player starts talking while the NPC is speaking,
cancel: current response is no longer relevant,
truncate: finish the current phrase and stop,
resume: continue after a brief interruption, if appropriate.
The safest implementation is usually to stop the audio stream, clear any buffered tokens, and reset the lip-sync state machine at the same moment. If you only stop one of those three, you get the classic failure mode where the mouth keeps moving after the voice has stopped, or vice versa.
Also pay attention to the NPC’s idle state. After an interruption, the face should quickly transition to a neutral or listening pose rather than freezing on the last phoneme frame. That transition is small, but it is what makes the interaction feel intentional instead of broken.
Keep the game loop and the voice loop decoupled
Do not tie audio playback directly to your simulation tick. Games often run at a variable frame rate, while speech pipelines need stable buffering and network I/O. If you couple them too tightly, a rendering spike can cause audible jitter, or a temporary network stall can block gameplay.
A better pattern is to run the voice stack as an asynchronous subsystem:
the game loop decides when the NPC should speak,
the voice subsystem handles LLM, TTS, and avatar streaming,
the renderer consumes the resulting state changes and animation cues.
That lets you preserve responsiveness even when generation is delayed. For example, the NPC can already look attentive while the first audio chunk is still on the wire.
A minimal streaming pipeline in practice
Here is a simplified Python sketch of the control flow. The exact API shape will depend on your stack, but the sequencing is what matters:
Two implementation details are easy to miss:
is_boundary()should be conservative. Early flushes improve latency, but overly aggressive flushes produce chopped prosody.The TTS subsystem should expose backpressure. If it cannot keep up, you need to either buffer or intentionally delay the start of the next utterance.
Example: attach a talking face to a LiveKit voice agent
If your game backend already uses LiveKit for real-time voice, the most direct way to add a synchronized face is through the LiveKit Agents plugin. The plugin wraps the avatar stream around the voice agent so the NPC speaks with a matching talking video face instead of separate, loosely coupled audio and animation systems. The repository has examples worth reading: https://github.com/protoface-ai/protoface-quickstart-openai-realtime.
Conceptually, the flow looks like this:
What you want from this integration is not just “a video overlay.” You want a single real-time pipeline where audio timing and face motion are derived from the same live speech turn, which is what keeps the mouth from drifting behind the words.
REST and SDK workflows when you need explicit session control
Sometimes you do not want the avatar embedded directly in the voice stack. You may want to create sessions ahead of time, manage them from a backend service, or run orchestration from gameplay servers. In that case, use the API or Python SDK to create and manage avatars and realtime sessions. The exact request fields are in the docs, but the pattern is straightforward: authenticate with an API key, create a session, then hand the session details to the runtime that will speak.
If you prefer Python, the SDK gives you the same shape of control from server code, which is useful when a matchmaker or NPC orchestration service decides when an avatar should come online. See the SDK repo for examples: https://github.com/protoface-ai/protoface-sdk-python.
Common gotchas in game-NPC setups
Double buffering: one buffer in the LLM layer and another in TTS can add a full second of latency if both wait “just a bit longer.”
Prosody loss: chopping text into tiny pieces makes speech sound robotic even when latency is great.
State drift: if the gameplay state changes while the NPC is mid-sentence, you need a deterministic cancel path.
Frame-dependent animation: if lip movement is updated only on render frames, low FPS will visibly degrade sync.
Network jitter: real-time avatar streams need buffering and reconnect logic; do not assume a perfect connection.
The right architecture is usually a compromise between responsiveness and linguistic quality. Short segments reduce perceived latency. Slightly longer segments preserve natural speech. Your job is to pick the boundary that fits the game’s interaction model.
Where Protoface fits
For teams using a voice-agent stack, Protoface is the part you can drop in when you need the avatar layer to stay synchronized with the speech layer without building your own lip-sync runtime. The most relevant surface for this use case is the LiveKit Agents plugin, which turns an existing agent into a talking video face while keeping the real-time voice pipeline intact. If you are wiring this into a backend service or want to create sessions explicitly, the REST API and Python SDK are the better fit. For setup details and exact request shapes, use the docs.
Conclusion
Synchronizing STT, TTS, and lip-sync is mostly about treating speech as a live stream with explicit state transitions, not as a one-shot text response. Segment text conservatively, let audio drive the face, keep interruption handling explicit, and decouple the voice subsystem from the game loop. If you get those pieces right, the NPC feels responsive instead of merely “animated.”
Next, build one narrow vertical slice: one NPC, one dialogue turn, one interruption path. Then test under jitter, low frame rate, and mid-utterance cancellation. If you want implementation details for a LiveKit integration, session management, or SDK-based orchestration, start with the docs and the relevant quickstart linked from the repository.