Introduction

If you want an NPC to feel present in a React game, text alone is not enough. You need a low-latency loop that can take player input, generate a response, synthesize speech, and render a face that stays visually synchronized with the audio. The hard part is not “calling an LLM”; it is keeping the experience responsive enough that the player feels like they are interacting with a character rather than a delayed chatbot.

This post shows a practical architecture for a realtime conversational NPC: a React client for the game UI, a FastAPI backend that handles authentication and session orchestration, and a TypeScript frontend that streams events and audio/video state in realtime. By the end, you should have a clear mental model for where each piece belongs, what state needs to be shared, and how to avoid the usual pitfalls around latency, browser media, and session management.

What “realtime NPC” actually means

For a game character, “realtime” does not mean “instant.” It means the player can speak or type, get a response fast enough to maintain turn-taking, and see a face that tracks the conversation closely enough to feel alive. In practice, that involves four concurrent streams:

• Input: microphone audio or text from the player.

• Understanding: transcription and/or LLM reasoning.

• Output: synthesized speech and a visual avatar with lip sync.

• Session state: who the player is, which NPC they are talking to, and what the current conversation context contains.

The common mistake is to treat the avatar as a UI ornament. In reality, it is part of the agent pipeline. If the audio arrives before the avatar animation updates, or if the video and audio drift apart, the illusion breaks immediately.

For that reason, a good integration keeps the agent session authoritative on the backend and lets the client render only the live media and minimal control state. The browser should not be responsible for deciding what the agent “knows”; it should just establish a realtime connection, display the face, and relay events to and from the game.

Architecture: keep game logic, agent logic, and media separate

A clean split for a React game looks like this:

1. React client: renders the game, opens the realtime session, and plays the avatar stream.

2. FastAPI backend: creates NPC sessions, issues any short-lived tokens or metadata, and stores conversation context if needed.

3. Agent runtime: runs the voice or multimodal agent, produces speech, and attaches the avatar stream.

This separation matters because the browser is a hostile environment. Anything embedded directly into the client can be inspected, copied, or abused. Keep API keys server-side. Let the client request a session from FastAPI, then hand back only ephemeral session data needed to connect.

There is also a performance reason. If the player is in the middle of a combat scene or dialogue tree, you do not want the avatar connection logic tangled up with game state reducers and UI transitions. Treat the NPC as an external service with its own lifecycle.

FastAPI: create a session, not a secret

In the simplest setup, the browser asks FastAPI for an NPC session. The backend authenticates the request, decides which avatar and instructions apply, and then calls Protoface server-side using your API key. The response contains only the information the client needs to join the session.

The exact request and response fields depend on the API shape in the docs, but the pattern is straightforward:

from fastapi import FastAPI, Header, HTTPException

return resp.json()

from fastapi import FastAPI, Header, HTTPException

return resp.json()

from fastapi import FastAPI, Header, HTTPException

return resp.json()

Use this endpoint as the only place where your long-lived API key exists. In a real game, you will likely also associate the session with a player account, a match ID, or a room ID, and persist that mapping in your own database so the NPC can resume context later.

Two backend gotchas are worth calling out:

• Timeouts: session creation should fail fast. If avatar provisioning or a downstream service stalls, return an error and let the client retry cleanly.

• Idempotency: if the player double-clicks “Talk,” you do not want duplicate NPC sessions. Make the endpoint safe to retry or deduplicate on your side.

React and TypeScript: connect once, then treat the avatar like a media peer

On the client, the NPC should behave like a realtime peer connection with some extra state. Your React component should request a session from FastAPI, connect, and then keep UI updates minimal: connection status, speaking indicator, transcript text, and whatever game-facing event stream you need.

type NpcSession = {

}

type NpcSession = {

}

type NpcSession = {

}

If you are using a websocket or WebRTC-based client SDK, the important part is not the specific call shape but the lifecycle:

1. request session details from your backend,

2. establish the realtime connection,

3. attach remote media tracks to the avatar container,

4. subscribe to transcripts or agent events,

5. tear everything down when the scene ends.

In React, keep the connection object outside render state so you do not reconnect on every rerender. A ref plus an effect is usually enough. If the NPC is only relevant in one part of the game, unmounting the component should fully close the session and release camera/microphone/media resources if you enabled them.

Also pay attention to autoplay policies. Browsers will often block audio playback until the user has interacted with the page. In a game, that means “Start Conversation” should be a deliberate click that both opens the session and satisfies media playback requirements.

State synchronization: the avatar is not the game, but it must reflect the game

The best NPC integrations send the agent just enough game context to respond coherently. Do not dump the entire world state into the prompt on every turn. Instead, send compact, structured facts that matter for the current interaction:

• player name, faction, quest status

• current scene or location

• NPC role and relationship to the player

• any constraints, such as “do not reveal the hidden quest yet”

When the player makes a game choice, use the same backend that creates sessions to update the conversation context. That keeps the agent’s memory consistent across reconnects and prevents the frontend from becoming the source of truth for narrative state.

A practical pattern is to model the NPC as a finite interaction state machine on your side and let the language model fill in the natural-language surface. For example, the backend can decide whether the NPC is in “greeting,” “quest offer,” or “farewell” mode, while the agent generates the actual phrasing. This prevents the model from wandering across gameplay boundaries.

Where Protoface fits

This is the point where a realtime avatar layer becomes useful instead of merely decorative. Protoface gives you a developer-facing avatar API so the agent can present a synchronized talking face without you having to build the avatar streaming stack yourself. In this kind of React game integration, the most relevant surface is the REST API: your FastAPI backend creates and manages sessions server-side, and your client receives only the runtime data needed to join. If you are already using a Python agent stack, the Python SDK can handle the same orchestration programmatically; if you are embedding a voice agent, the LiveKit plugin route is the more direct path.

The main operational benefit is that you can keep API keys and session policy on the backend while the browser only handles rendering and user interaction. That matches the trust boundary you want in a game client.

Latency, lip sync, and debugging realities

Three failure modes show up immediately in production:

1. Audio starts but the face lags: usually a buffering or track-subscription issue. Check that the video stream is attached after the session is fully ready.

2. The NPC feels slow: often prompt size or downstream model latency. Trim context aggressively and keep responses concise when possible.

3. Reconnects break the scene: session state is not being persisted outside the frontend. Store the authoritative conversation state on the backend.

For debugging, log the session lifecycle explicitly: created, connected, speaking, interrupted, disconnected, resumed. That gives you a much clearer timeline than browser console noise alone. If you have the agent emit transcript or turn events, surface them in a developer-only overlay during integration; it makes timing bugs much easier to spot.

One more practical point: if your NPC can be interrupted by player input, make sure your audio pipeline supports barge-in behavior. The agent should stop talking when the player starts speaking, or the dialogue will feel unresponsive even if the underlying transport is fine.

Conclusion

A believable conversational NPC in a React game is mostly an integration problem: keep the browser thin, keep session creation server-side, preserve authoritative state in FastAPI, and treat the avatar as part of the realtime agent pipeline rather than a static widget. Once those boundaries are in place, the remaining work is mostly tuning latency, managing conversation context, and making sure media lifecycles are clean.

If you want to build this with less plumbing, start with the docs at docs.protoface.com and one of the quickstarts linked from the public examples repo. The important thing is to wire up a small end-to-end path first: one NPC, one session, one conversation loop, then expand from there.