From STT to Avatar: A Beginner’s Guide to Realtime AI NPCs for Games
Build realtime AI NPCs for games with streaming STT, agent logic, TTS, avatar sync, and low-latency session control.
Introduction
If you want an NPC that can actually talk, interrupt, react, and keep up with a player in real time, the hard part is not speech-to-text alone. It is the full loop: capture audio, transcribe it with low latency, decide what the agent should say, generate speech, and keep a visual face synchronized with the voice so the interaction feels like one system rather than a chain of disconnected services.
This post walks through that pipeline from a game-dev perspective. By the end, you should understand the moving parts of a realtime AI NPC, where latency enters the system, how to structure the agent loop, and how to attach a live talking avatar without turning your stack into a science project.
What “realtime AI NPC” actually means
A realtime NPC is not just a chatbot with a skin. It is usually a bidirectional streaming system with four distinct components:
Audio input: the player’s microphone audio arrives continuously, usually over WebRTC or a comparable low-latency transport.
STT: speech-to-text converts partial utterances into text incrementally. Good systems expose interim transcripts, not just final ones.
Agent reasoning: the dialogue layer decides whether to respond, ask for clarification, trigger game actions, or wait for more speech.
TTS + avatar rendering: the response is synthesized as audio and a synchronized face/video stream is rendered so mouth movement tracks the generated speech.
For games, the important constraint is that these stages overlap. You do not want to wait for a full sentence if the player has already said enough to identify intent. Likewise, you do not want the avatar’s mouth to lag behind the speech by 400 ms, because that immediately reads as broken.
Designing the interaction loop
The common beginner mistake is to treat the NPC like a request/response API: user speaks, you transcribe, the model answers, end of story. That works for demos, but it feels slow and unnatural in a live conversation. A better pattern is a streaming turn-taking loop:
Capture microphone audio continuously.
Send audio frames to STT and consume interim transcripts.
Detect end-of-turn using a combination of silence, prosody, and application logic.
Pass the final transcript, plus conversation state and game context, to the agent.
Stream the agent’s response into TTS.
Render the avatar video in sync with the audio output.
For an NPC, you usually also want a control layer that can interrupt or branch. Examples: if the player asks about a quest item, respond immediately; if they start yelling over the NPC, cancel the current TTS; if they say something out of scope, fall back to a short, character-consistent clarification.
Latency budgets and where they actually go
Most of the user experience is decided by total round-trip latency, not by model quality in isolation. You can think about the budget in rough buckets:
Audio capture and network transport: WebRTC is commonly used because it keeps jitter low and handles realtime media better than generic HTTP polling.
STT partials: incremental transcripts should arrive fast enough to support interruption and intent detection.
Agent inference: the model or rules layer needs to stay predictable. Long prompts and unnecessary context increase tail latency.
TTS start time: the time until the first audio chunk matters more than total synthesis duration.
Avatar synchronization: the face should begin moving with the audio stream, not after it.
Two practical rules help here:
First, keep the conversational state compact. Feed the agent only the last few turns plus game-specific state that is actually relevant. Second, separate “thinking” from “speaking.” The agent can decide while the audio pipeline is already warming up, but the response should not be held back waiting for a perfect paragraph.
Implementation pattern in Python
If you are building the server side yourself, a Python SDK is the most direct way to create and manage sessions, then wire the session into your game backend or voice service. Exact request/response fields are in the docs, but the shape is usually straightforward: create an avatar, start a realtime session, then hand the client a session identifier or streaming URL.
That pattern is useful when your game server is the source of truth for state. You can attach quest flags, zone info, faction reputation, or moderation rules before starting the session. It is also the right place to enforce rate limits, choose quality tiers, and decide when a session should be torn down.
Using STT and agent streaming without overcomplicating the game server
For games, it is usually a mistake to put the whole media pipeline inside the main gameplay process. Keep the game authoritative for state, but let a separate realtime service own the audio/video session. That reduces coupling and makes it easier to restart the conversation layer without rebooting the game.
A typical architecture looks like this:
The game client sends the player’s microphone audio to the realtime service.
The service performs STT, agent reasoning, and TTS.
The same service streams the avatar video back to the client.
The game server receives only compact events, such as “quest accepted” or “player insulted the NPC,” rather than raw media.
This separation matters because you want the NPC to fail gracefully. If the avatar pipeline is unavailable, the game should still be playable. If the game logic lags, the NPC should not block the voice loop. Keeping these concerns apart also makes testing saner: you can unit test dialogue behavior without recording audio, and you can test media sync without booting the full game world.
REST control plane for sessions and avatars
When you need direct control from your backend, the REST API is the cleanest integration point. Authenticate with an API key server-side, create avatars, create sessions, and then hand the realtime session details to the component that handles playback. This is the right tool if you are integrating an NPC into a game backend, a matchmaking service, or a proprietary conversation pipeline.
Use the REST surface when you want explicit lifecycle control. For example, your game can create a session when the player enters an interactive zone, update instructions when the character switches mood, and terminate the session when the encounter ends. That is much easier to reason about than embedding hidden state in a client-side widget.
Where Protoface fits
This is the point where Protoface is useful: it gives you the avatar and session layer without forcing you to build the lip-sync/video side yourself. If you are already using a voice agent stack, the LiveKit plugin can drop a synchronized talking face into the agent with minimal glue. If you want more direct control, the REST API and Python SDK let your backend create avatars and sessions programmatically. The docs at docs.protoface.com are the place to check exact fields and current request shapes.
For developers already on LiveKit, the plugin approach is often the fastest way to get from “audio agent” to “NPC with a face.” It keeps the transport and conversation logic in the existing agent stack while adding the visual layer as a composable piece, which is exactly what you want if you are iterating on behavior rather than rebuilding media infrastructure.
Common gotchas in game integrations
Do not block on full sentences: use partial STT to detect intent early.
Do not treat the avatar as decorative: if lip sync drifts, players notice immediately.
Do not expose secrets in the browser: keep API keys server-side; if you need a browser embed, use a customer-managed iframe flow rather than shipping credentials to the client.
Do not let the agent overtalk the game: the NPC should be interruptible and bounded by gameplay rules.
Do not ignore cost controls: quality tiers, session duration, and rate limits matter once you have real traffic.
Also, test with messy audio. Players speak over effects, music, and other characters. If your agent only works in a quiet room, it is not ready for a game.
Conclusion
Building a realtime AI NPC is mostly an exercise in systems design: low-latency audio transport, incremental STT, disciplined agent state, streaming TTS, and synchronized avatar rendering. Once you separate those concerns cleanly, the problem becomes tractable and debuggable.
If you are putting this into a game or any other interactive product, start with a narrow use case: one character, one conversation mode, one set of instructions, and clear turn-taking rules. Then expand from there. For implementation details, integration examples, and current API shapes, check docs.protoface.com and the relevant quickstarts in the GitHub repo linked there.