Introduction

If you want a game NPC to feel present, it is not enough to send text back to the client and play a random TTS clip locally. A conversational NPC needs a low-latency audio path, a synchronized avatar face, and a transport that can keep up with partial responses, barge-in, and session state. In practice, that means treating the NPC as a realtime system: text or intents go in, tokens or speech come back out, and the avatar stays aligned with the voice stream.

In this post, I’ll show the pieces you need to stream a conversational game NPC avatar over WebSocket and TTS, how to think about the buffering and synchronization problem, and where a realtime avatar API fits in. By the end, you should be able to wire a game client to a backend NPC service that streams speech, keeps lip sync stable, and handles interruptions without feeling brittle.

Why “streaming NPC voice” is different from a normal chatbot

Game dialogue has a few constraints that web chat systems often gloss over:

• Latency matters more than completeness. A 300 ms delay in a menu bot is fine; in a character conversation it feels dead.

• The output is multimodal. The player hears speech, sees mouth movement, and often expects animation state changes like idle, talking, and listening.

• Interruptions are normal. Players click away, walk off, or speak over the NPC. The system must cancel or reframe responses quickly.

• State has to be owned somewhere. The game client should not be building prompts, calling TTS, and doing face sync directly unless you want a lot of complexity in the engine layer.

A clean way to build this is to keep the game client thin and use a backend session that orchestrates LLM output, TTS, and avatar streaming. The client talks to that backend over WebSocket, because that gives you bidirectional control and a natural way to push partial events, cancellation, and playback state.

Transport design: WebSocket for control, audio frames for playback

For a conversational NPC, the server usually owns three jobs:

1. Interpret player input. That can be text, recognized speech, or a gameplay event like “player entered the room.”

2. Generate assistant output incrementally. You want token streaming or sentence-level chunks so the NPC can begin speaking before the full answer is complete.

3. Convert speech into playable audio and avatar timing. The client should receive a stream it can render immediately, not a giant finished file.

WebSocket is a reasonable fit because it can carry both control messages and streaming events on one connection. A typical event flow looks like this:

client -> server: {"type":"player_text","text":"Who runs this place?"}

client -> server: {"type":"player_text","text":"Who runs this place?"}

client -> server: {"type":"player_text","text":"Who runs this place?"}

The exact schema is up to you, but the important idea is that the client should not wait for a complete utterance before starting playback. If you do, the NPC will always feel a beat behind. Sentence streaming or phrase streaming is usually the sweet spot: it gives the TTS engine enough context for natural prosody without forcing the user to wait for the full paragraph.

Handling TTS without making the game client do too much

Text-to-speech is where many prototypes get messy. The obvious implementation is “send text to a TTS provider, get back audio, play it in the client.” That works for short clips, but it breaks down once you need:

• token streaming or partial sentence synthesis,

• cancellation when the player interrupts,

• consistent voice selection across sessions,

• and avatar lip sync tied to the actual speech timing.

A better pattern is to let the server own the TTS session and stream audio to the client in small chunks. The client’s job is to buffer just enough for smooth playback, typically a few hundred milliseconds, while keeping the UI responsive. If you buffer too much, you lose interactivity. If you buffer too little, network jitter becomes audible.

Two practical gotchas:

• Do not block on full synthesis. Start playback as soon as the first stable chunk is ready.

• Keep an explicit cancel path. When a player interrupts, stop the current audio stream and mark the assistant turn as aborted so you do not continue animating the old response.

If your game already has a realtime voice layer, you can keep that intact and route only the avatar/video side through a separate service. The main thing is that the avatar must follow the same speech timeline the player hears, or the illusion falls apart immediately.

Synchronizing the avatar face with speech

Lip sync is not just “open mouth while audio plays.” In a realtime avatar, the face animation should be driven by the speech stream itself or by timing metadata derived from it. If the TTS engine emits phoneme or viseme timing, use that. If it does not, you need an avatar system that can infer mouth motion from the audio frames without adding much latency.

The synchronization contract should be simple:

• talking when audio is being produced or played,

• listening when the player is speaking or when the NPC is idle and waiting,

• idle when there is no active turn.

That sounds trivial, but these state transitions are what keep the character readable. Game AI often fails here by switching animations based on LLM state alone. The right signal is the actual audio pipeline state.

Also pay attention to turn boundaries. If the NPC begins speaking while the previous response is still winding down, you will see mouth jitter or state flicker unless you explicitly fade between turns. A small amount of overlap can be fine; uncontrolled overlap is not.

Backend orchestration in Python: a minimal pattern

For a practical implementation, I usually keep a backend service that accepts player events over WebSocket and coordinates the assistant turn. Here is a minimal sketch of the shape, not a production-ready contract:

import asyncio

asyncio.run(main())

import asyncio

asyncio.run(main())

import asyncio

asyncio.run(main())

The details will vary depending on your LLM and TTS providers, but the shape is the same: one realtime session, streamed output, and explicit state messages. If you add speech input later, keep the same session model and treat speech-to-text as just another ingress path.

Where Protoface fits in this architecture

This is exactly the kind of problem Protoface is meant to solve: you keep your game logic and conversational backend, and let the avatar layer handle the synchronized talking face. For a LiveKit-based voice stack, the OpenAI Realtime quickstart and the LiveKit plugin path are the most direct ways to see the pattern end-to-end, and the docs at docs.protoface.com cover the API and integration details.

In practice, the main benefit is that you do not need to reinvent the avatar transport. Your backend can keep streaming conversation state and audio, while the avatar surface stays synchronized to that realtime session. If you are already using LiveKit agents, the plugin approach is especially low-friction because it drops the face into the voice agent rather than forcing you to build a separate media pipeline.

Implementation trade-offs and operational gotchas

A few things are worth deciding up front:

• Session ownership. Keep the authoritative NPC turn state on the server. The client should render and relay player events, not derive conversation state itself.

• Backpressure. If audio chunks arrive faster than the client can play them, you need a bounded buffer and a policy for dropping or coalescing stale chunks.

• Interrupt semantics. Define what happens on player interruption: cancel current speech, queue a new turn, or switch to a fallback line.

• Voice consistency. Pick a single voice profile per NPC unless your design explicitly wants variation. Nothing breaks character faster than accidental voice drift.

• Authentication and session scoping. For browser-based experiences, do not expose API keys in the client. Keep secrets server-side and issue only narrow session credentials or use an embedded surface designed for that constraint.

One final note: latency budgets add up quickly. Network round trips, prompt assembly, first-token latency, TTS startup, and media jitter can each cost a little. If you measure only the LLM time, you will miss the real problem. Measure end-to-end from player input to first audible phoneme and from first word to visible speaking state.

Conclusion

A conversational game NPC feels good when the whole pipeline is treated as realtime media, not just “chat plus sound.” WebSocket gives you a clean control plane, streaming TTS keeps the turn responsive, and the avatar layer has to follow the actual speech timeline rather than an abstract assistant state.

If you want to implement this without building the avatar stack yourself, start with the docs and a quickstart that matches your voice system. The docs at docs.protoface.com are the right place for exact API shapes, and the GitHub quickstarts are useful when you want to see a working integration before wiring it into your game server.

Build the NPC backend as a realtime session, keep interruption handling explicit, and make audio playback the source of truth for animation state. That combination is usually enough to turn a static character into something that actually holds a conversation.