Introduction

Game NPCs feel more convincing when their face, voice, and timing all line up. The hard part is not “making an avatar talk” in the abstract; it is wiring a low-latency media pipeline that keeps lip motion synchronized with generated speech, survives network jitter, and fits cleanly into a React app without turning your UI tree into a media stack.

This post shows the practical shape of that system. By the end, you should understand how streaming speech, visemes/lip sync, and avatar video fit together; how to render an interactive NPC in React; what can go wrong when you mix real-time media with browser rendering; and where a service like Protoface fits into the architecture if you want to avoid building the avatar layer yourself.

What “streaming lip-synced dialogue” actually means

For an NPC, you typically have three concurrent streams:

• Text or intent from your game logic or dialogue system.

• Audio from a TTS engine or voice model, produced incrementally.

• Video facial motion that is aligned to the audio as it is being generated and played.

“Lip-synced” is only useful if the timing is tight enough that the mouth shapes match the phonetic content of the audio the player hears. In practice, that means the avatar renderer must consume audio timing information or phoneme/viseme hints with low end-to-end latency. If the video lags behind the audio by a few hundred milliseconds, players notice immediately. If the NPC speaks in bursts rather than as a continuous stream, the face should still transition smoothly instead of snapping between static frames.

In a React app, this usually implies the avatar is not a normal DOM component with a few props. It is a media endpoint: something you mount, connect, and keep alive while the rest of the UI changes around it.

Choose the right integration shape for your app

There are three common ways to wire an avatar into an NPC experience:

1. Local UI only: you animate a 2D or 3D character in the browser from your own speech pipeline. Maximum control, maximum work.

2. WebRTC/avatar service: your app sends conversation state to a service that returns synchronized video/audio for the avatar. Lower integration cost, better latency characteristics.

3. Embedded session: the avatar lives in an isolated iframe or a separate client, and your app communicates with it through a narrow interface. Strong isolation, fewer security headaches, less UI control.

For game NPCs, the second option is usually the sweet spot: you still own the game logic, but you do not want to own the media plumbing. The browser is good at rendering video; it is less good at being a real-time audio/video synchronization engine unless that is the entire product.

React integration patterns that actually hold up

When you add an avatar to a React game UI, treat it like any other real-time resource. Mount once, connect once, and avoid recreating the stream on every render.

1) Keep the media object outside the render loop

Use a ref for the connection/session object and initialize it in an effect. If you create a new session every time props change, you will get reconnect storms and visible lip-sync resets.

import { useEffect, useRef } from "react";

import { useEffect, useRef } from "react";

import { useEffect, useRef } from "react";

The important part is not the exact API shape; it is the lifecycle. A media session should outlive unrelated React re-renders.

2) Decouple dialogue state from rendering state

Your NPC dialogue manager may change text quickly: partial transcript updates, tool calls, interrupts, reactions, and so on. Do not push every one of those through React state if they only affect the avatar session. Keep a small, explicit bridge between game state and media state.

A useful pattern is:

• game engine emits a speech intent,

• dialogue service generates audio/text chunks,

• avatar session receives those chunks,

• React only reflects high-level status like “listening”, “speaking”, or “idle”.

This prevents your UI thread from becoming the control plane for real-time media.

3) Plan for interruption and turn-taking

NPCs in games rarely speak in uninterrupted monologues. Players interrupt, walk away, or trigger another event. Your avatar pipeline needs a clear turn model:

• When the player starts talking, stop or duck the NPC audio.

• When the NPC resumes, restart from a coherent state rather than forcing the mouth animation to “continue” a stale sentence.

• If you support barge-in, make sure the avatar can transition out of speaking without a visual pop.

For this reason, I prefer systems that expose explicit session control over systems that only hand you a passive video element. You need to know when the avatar is speaking, when it is listening, and when the media pipeline is back-pressured.

Latency and sync: the constraints that matter

For lip-synced dialogue, latency is not one number. You care about:

• First-token / first-audio latency: how long before the NPC starts speaking after the trigger.

• A/V sync error: how far the mouth motion drifts from the audio.

• Jitter: whether the avatar stays smooth under network variation.

• Recovery time: how quickly the session resumes after a transient drop.

If you are streaming speech from a model, partial outputs are useful only if your avatar layer can consume them incrementally. Otherwise you are buffering the entire utterance and turning a “realtime” system into a batch system.

Also remember browser constraints: autoplay policies, muted-video bootstrap patterns, and CORS/iframe isolation can all affect when media actually starts. In React, make sure your avatar element is present before connecting the media stream, and keep a stable container so layout shifts do not cause unnecessary re-renders.

Example: controlling a realtime avatar session from Python

If your NPC logic runs server-side, the cleanest model is often: game backend decides what should be said, backend opens or updates the avatar session, and the browser just renders the resulting video. The exact SDK method names are in the docs, but the shape is straightforward:

from protoface import Client<p></p>

from protoface import Client<p></p>

from protoface import Client<p></p>

That is the basic pattern: create or select an avatar, create a session, then stream dialogue into it as your game logic advances. Check the docs for the exact request and object fields.

Where Protoface fits: use the plugin when your agent already lives in LiveKit

If your NPC is powered by a LiveKit voice agent, you do not need to build a separate media bridge. The LiveKit integration is the simplest path: drop the Protoface plugin into the agent and the voice pipeline gets a synchronized talking face.

That matters because the avatar should track the same conversational events as the audio agent. The plugin approach keeps the speech stack and the face stack aligned inside one agent lifecycle instead of trying to coordinate them across separate browser components.

# Pseudocode: exact setup depends on your agent stack.<p><

# Pseudocode: exact setup depends on your agent stack.<p><

# Pseudocode: exact setup depends on your agent stack.<p><

If you are already using LiveKit for voice, this is the least error-prone way to get a lip-synced NPC in front of a player. If you are not using LiveKit, use the REST API or Python SDK directly and keep the browser focused on rendering.

Security and operational gotchas

Game-facing avatar systems usually end up on public web surfaces, so basic API hygiene matters:

• Do not expose long-lived API keys in the browser.

• Keep session creation on the server when you need privileged access.

• Set clear limits on session duration and rate to avoid abuse.

• Assume the avatar session can fail independently of the rest of the UI and handle fallback states gracefully.

If you embed the experience in an iframe, you can isolate the media surface from your app and avoid shipping credentials to the client. That is useful when you want a self-contained conversational NPC on a website, but it is less flexible than a deeply integrated in-app component.

Conclusion

For game NPCs, the core problem is not generating speech; it is preserving believable timing between dialogue, audio, and facial motion while keeping the integration maintainable in React. The practical recipe is to treat the avatar as a long-lived realtime media session, separate dialogue state from UI state, and choose the integration surface that matches your stack.

If you are already on LiveKit, the plugin path is the fastest way to add a synchronized face. If you want server-controlled sessions, use the REST API or Python SDK. For implementation details and exact request/response shapes, start with docs.protoface.com and the relevant quickstarts in the GitHub org.