Introduction

If you want a game NPC to feel alive, the bar is not “it can answer questions.” The bar is: it speaks with low enough latency that players can interrupt it, its mouth movement matches the audio, and the interaction pipeline doesn’t fall apart when the scene gets busy.

That usually means combining three systems: a conversational model, real-time audio transport, and a video/animation layer for the avatar. In practice, the cleanest architecture is to keep the conversational logic in your app or agent stack, then attach a synchronized face that renders from the same live session. By the end of this post, you should be able to wire a talking NPC into a React app, understand where WebRTC fits, and avoid the common mistakes that make realtime avatars feel “off.”

What you actually need from the browser

For a talking NPC, the browser is usually just the presentation layer. It should:

• Join a realtime session over WebRTC.

• Receive audio and/or video tracks with minimal buffering.

• Render the avatar face in sync with the agent’s speech.

• Handle reconnects, visibility changes, and network jitter gracefully.

React is a reasonable choice here because the UI is stateful, but React itself is not the transport. The transport is typically WebRTC: peer connections, negotiated media tracks, and adaptive jitter handling. If your NPC is driven by a voice agent, you want the agent and avatar to share the same timing source so the face movement tracks the actual utterance, not a delayed transcription.

That distinction matters. Lip sync is not “animate mouth on text arrival.” It is a rendering problem tied to media timing. If audio is late, the face should be late by the same amount. If the user interrupts, the session should be able to stop or overlap gracefully without leaving the avatar in a dead pose.

Building the React side around a realtime session

A good mental model is:

1. Your React app starts or joins a session.

2. The app receives a session token or WebRTC credentials from your backend.

3. The browser joins the live room and subscribes to the avatar’s media track.

4. Your UI shows connection state, speaking state, and optional subtitles.

Keep the browser dumb about secrets. API keys should stay on the server. In a game or web app, the browser should only receive short-lived session credentials or an embed URL that is already scoped to the user’s interaction.

// Sketch only: create a session on your server, then hand a short-lived token to React.

await connectToLiveSession({ url: wsUrl, token });

// Sketch only: create a session on your server, then hand a short-lived token to React.

await connectToLiveSession({ url: wsUrl, token });

// Sketch only: create a session on your server, then hand a short-lived token to React.

await connectToLiveSession({ url: wsUrl, token });

On the React side, treat the avatar like any other realtime media component. Mount it once, keep the connection stable, and avoid unnecessary re-renders. If you recreate the connection on every prop change, you will get renegotiation churn and visible glitches. Use refs for connection objects and state only for UI signals like “connecting,” “speaking,” or “reconnecting.”

Why WebRTC is the right transport for a talking NPC

WebRTC exists because live media has different requirements from normal HTTP requests. You need:

• Low latency: the user should hear and see the NPC in near real time.

• Jitter tolerance: networks are not stable, so packets arrive unevenly.

• Bidirectional media: the NPC may need to hear the player as well.

• Track-level synchronization: audio and video should stay aligned.

For an NPC, the most common failure mode is a mismatch between audio and the face. The audio starts immediately, but the video is delayed by another pipeline. Or the face keeps animating while the speech has already stopped. The fix is to keep both tracks under the same session semantics and let the media stack handle timing.

Another common issue is over-processing in the browser. If you add a canvas-based animation on top of an already-rendered video face, you may introduce extra frame delay for no real gain. For a game NPC, the avatar face itself should usually be the primary visual layer. Add overlays, captions, or game-state badges around it, not inside a custom animation loop unless you truly need it.

Server-side orchestration: keep secrets and agent logic out of the client

The browser can join a session, but it should not decide what the avatar says or reveal your API key. A sane split is:

• Backend: creates sessions, configures the NPC persona, and talks to the avatar service.

• Frontend: joins the live session and renders the result.

• Agent layer: handles conversation state, tool calls, and interruption logic.

If you are already running a voice agent, this is usually the point where the avatar layer plugs in. You keep the voice agent’s response stream intact and attach the face to the same realtime session. That way, the avatar reacts as the speech is generated, not after the response is fully complete.

import os

print(session)

import os

print(session)

import os

print(session)

The shape of the request will depend on your workflow, but the pattern is stable: authenticate on the server, create or configure the realtime session, then pass the session details to the browser. If you need repeatable automation, the Python SDK is a better fit than raw REST calls because it keeps the session management code in one place and makes it easier to script avatar lifecycle operations. The Python SDK repository is a good place to start if you want to generate sessions from game backend code or admin tooling: https://github.com/protoface-ai/protoface-sdk-python.

Protoface in this setup

This is the part where Protoface fits naturally: you use it as the avatar layer that attaches to your voice agent and turns the agent’s audio into a synchronized talking face. For LiveKit-based stacks, the relevant integration is the LiveKit Agents plugin published on PyPI, so the avatar can follow the same agent session instead of being an after-the-fact video overlay. If you are using LiveKit Agents already, the plugin path is the least invasive way to add a face without redesigning your transport.

The integration itself is straightforward in concept: the agent speaks, the avatar receives the live session context, and the client renders the synchronized media. The details—avatar IDs, session parameters, voice selection, and lifecycle hooks—belong in the docs, because those are the pieces that change as the product evolves. Start here: https://docs.protoface.com.

# Illustrative only: plugin usage depends on your LiveKit agent setup

# Illustrative only: plugin usage depends on your LiveKit agent setup

# Illustrative only: plugin usage depends on your LiveKit agent setup

If you prefer to inspect the plugin surface directly, the examples in the repository are usually the fastest way to understand how it is wired into a real agent: https://github.com/protoface-ai.

Practical gotchas when embedding an NPC in React

There are a few things that routinely cause trouble:

• Recreating connections on render. Memoize connection state and keep the peer connection stable.

• Ignoring visibility changes. Browsers may throttle hidden tabs; pause nonessential work and expect reconnects.

• Leaking keys into the client. Never ship an API key in the React bundle.

• Trying to “fake” lip sync in UI code. Let the media pipeline drive timing.

• Not handling barge-in. In conversational NPCs, interruption is a feature, not an edge case.

It also helps to design the NPC UX around what realtime media can do well. Keep responses short enough to feel interactive. Use streaming speech rather than waiting for a full paragraph. Surface listening/speaking indicators clearly so the player knows when the NPC is processing input versus just idling.

Finally, watch your quality tier and session duration. For game usage, you often want many short interactions rather than a handful of long calls. That makes session cleanup and rate limiting part of your product logic, not an afterthought.

Conclusion

A talking NPC in React is mostly a systems integration problem: keep the browser focused on rendering, keep secrets on the server, and use a realtime media stack that preserves audio/video timing. WebRTC gives you the transport, your agent handles conversation logic, and the avatar layer provides the synced face.

If you want to implement this without building the avatar/session infrastructure yourself, start with the docs at https://docs.protoface.com, then wire it into your LiveKit or backend flow. For a game UI, the end result should feel like a single coherent character, not a chat widget with a video tile attached.