Connecting STT, TTS, and a Realtime Avatar for Dynamic Game NPC Interactions
Build realtime game NPCs with streaming STT, TTS, turn detection, barge-in handling, and synced avatar lip sync.
Introduction
If you are building a game NPC that can actually talk back, the interesting part is not “make it speak.” It is the full loop: capture player speech, transcribe it, decide on a response, synthesize audio, and keep an avatar video face in sync with that audio so the character feels coherent in real time. The hard constraints are latency, turn-taking, interruption handling, and not letting your rendering pipeline drift out of sync with the voice pipeline.
This post walks through that pipeline from a developer’s point of view. By the end, you should have a practical mental model for stitching together STT, TTS, and a realtime avatar for NPC interactions, plus enough implementation detail to know where the common failure modes are. I will also show where Protoface fits when you want the avatar layer to stay synchronized with a voice agent.
Architecture: what actually has to happen on each turn
A dynamic NPC interaction is usually a streaming system, not a request/response system. The player speaks, the client streams audio, STT produces partial and final transcripts, the agent decides whether to answer or continue listening, TTS emits audio chunks, and the avatar renderer needs the same text/audio timing information so lip sync and facial motion stay believable.
The minimal dataflow looks like this:
There are two important implications:
You need streaming at every layer. If STT waits for a full utterance, the NPC feels delayed. If TTS waits for the full response, the avatar can’t start moving until too late.
The avatar is not a separate “effect”. It is part of the same turn state. The video face must start, pause, and stop in sync with the synthesized speech, including barge-in and interruption.
For game NPCs, this is especially important because players do not tolerate unnatural pauses the way they sometimes do in support chat. If the character takes a full second to begin speaking, the illusion is already weakening.
Streaming STT and turn detection
Speech-to-text in a realtime NPC loop is usually about partial hypotheses and endpointing. Partial transcripts let the agent reason before the player has fully finished, while endpointing determines when the player has actually yielded the floor. If you treat every pause as a turn boundary, you will interrupt players mid-thought. If you wait too long, the game feels sluggish.
In practice, you want three behaviors from your STT/turn layer:
Partial transcripts for low-latency intent detection.
Final transcripts for stable downstream reasoning and logging.
Voice activity detection or endpointing to decide when the NPC should respond.
For NPCs, partials are especially useful when the player says something like “Where is the blacksmith?” You do not need the last word before the agent can already start preparing a route answer. But you should avoid speaking too early if the player is still adding context such as “Where is the blacksmith, and can he repair my sword?”
The practical trade-off is that aggressive endpointing improves responsiveness but increases false turn endings. For conversational game characters, a slightly conservative threshold is usually better than premature interruption.
Keeping TTS and avatar motion synchronized
Once the agent has a response, the next question is how to get the audio and video face to start together and remain aligned. This is where many implementations fall apart. If you generate TTS first and then separately start a video animation, you often get a visible lead or lag. If the avatar is driven by a local animation timer instead of the actual audio stream, drift accumulates over longer utterances.
The clean approach is to treat the synthesized audio as the source of truth and drive avatar timing from the same streaming session. That means:
start the avatar only when the response turn starts, not when the first audio file is fully ready;
keep audio chunk boundaries and mouth motion aligned;
stop both when the turn ends or when the player barges in;
prefer a backend-controlled session over client-side ad hoc animation if you need predictable synchronization.
For games, barge-in matters a lot. If the player interrupts the NPC, you should stop the TTS stream and immediately transition the avatar out of speaking mode. Otherwise you get the uncanny effect of a character continuing to mouth words after the player has already moved on.
A subtle but important detail: if your TTS provider emits audio incrementally, you can start playback before the full response is complete. That improves perceived latency, but it also means your avatar layer must be able to consume streaming speech, not just a final clip.
Implementation sketch with a voice agent and realtime avatar
If you are already using a voice-agent stack, the simplest mental model is to add the avatar at the point where the agent emits speech. The exact APIs vary by framework, but the pattern is consistent: the agent handles STT and reasoning, TTS produces speech, and a video-face plugin consumes the same turn stream so the avatar stays synchronized.
With LiveKit Agents, that usually means dropping in the Protoface plugin and letting it attach a synchronized talking face to the agent session. The example below is intentionally short; treat the concrete fields as illustrative and check the docs for the current API.
The operational benefit here is that the avatar becomes part of the same realtime session as the voice agent, rather than a separate UI widget that you have to keep in sync manually. That is usually the difference between “looks okay in demos” and “survives production latency, interruption, and reconnects.” The plugin and examples are documented in the repository here: github.com/protoface-ai/protoface-plugin-pipecat. If you are using Pipecat specifically, the integration guide is also worth a look: docs.pipecat.ai/api-reference/server/services/video/protoface.
For teams that prefer to orchestrate sessions directly, the REST API is useful for avatar and session lifecycle management. A typical pattern is to create a session server-side, hand the client a short-lived session reference, and keep the API key only on the backend.
That kind of server-controlled session is a good fit when the NPC needs game-state context, because you can inject world state, quest flags, or player inventory into the agent prompt without exposing credentials in the client.
Game-specific trade-offs: latency, state, and interruption
Game NPCs have a few constraints that generic assistants do not.
Latency budget. If you want the interaction to feel like part of the game loop, keep total turn latency tight. You are not just optimizing for first token; you are optimizing for when the avatar begins visibly reacting. A half-second delay can be acceptable in a web demo and still feel awkward in a combat or exploration context.
World-state grounding. The agent should know what the player has already done, what region they are in, and whether the NPC is allowed to reveal certain information. This is usually a prompt/state injection problem, not an avatar problem, but the avatar makes bad grounding more noticeable because it feels like a character with intent.
Interruptibility. Game speech should be cancellable. If the player walks away, attacks, or starts speaking over the NPC, stop generation and stop playback immediately. The avatar should reflect that transition rather than finishing a sentence the player no longer cares about.
Determinism vs. improvisation. You probably do not want a fully unconstrained model for every NPC line. A useful pattern is to combine authored dialog templates with AI-generated elaboration, then keep the avatar layer agnostic to the text source. The avatar only needs timing and speech cues; the game logic decides how creative the response can be.
Where Protoface fits
Protoface is the part of the stack that gives the voice agent a synchronized realtime face without forcing you to build the streaming video layer yourself. If you are already running a voice agent and need an avatar that tracks TTS output, the LiveKit plugin is the most direct path. If you want to manage avatars and sessions yourself, the REST API and Python SDK are the control plane. If you are embedding a character on a site, the iframe route avoids exposing backend credentials entirely.
For developers implementing game NPCs, the important point is that the avatar is session-bound. You create or manage the avatar and session, then let the realtime transport handle synchronization. That is much easier than trying to animate a face client-side from text alone.
If you want to dig into the mechanics, the public docs are the right starting point: docs.protoface.com. For Python-based orchestration, the SDK is available here: github.com/protoface-ai/protoface-sdk-python.
Conclusion
Building a convincing dynamic NPC is mostly an exercise in realtime systems engineering. STT has to produce useful partials, the agent has to decide quickly without talking over the player, TTS has to stream rather than batch, and the avatar has to stay locked to the same session timing as the speech. Once those pieces are aligned, the interaction starts to feel like a character instead of a chatbot with a skin.
If you are wiring this up now, start by deciding where turn control lives, then make the avatar follow that same control path instead of inventing a separate animation clock. From there, use the docs, quickstarts, and plugin examples to fit the avatar layer into your existing voice pipeline. The public docs at docs.protoface.com are the best next step.