Building a Horizontal Scaling Architecture for Streaming AI Avatars in Next.js and Node.js
Next.js and Node.js architecture for streaming AI avatars: control plane/data plane split, stateless workers, Redis, and scaling tips.
Introduction
Streaming AI avatars are a systems problem before they are a graphics problem. The hard parts are not “make a face move”; they are keeping audio, lip motion, session state, and browser rendering aligned while traffic grows, latency stays low, and failures don’t cascade. If you are building this in Next.js and Node.js, the first version is usually simple: one server process, one avatar session, one websocket, one room. The first real version needs to survive bursty traffic, multiple concurrent sessions, and horizontal scale without introducing visible lag or broken state.
This post walks through a practical architecture for that setup. By the end, you should be able to design a Next.js front end plus a Node.js backend that can create avatar sessions, route realtime streams through stateless workers, and scale out cleanly behind a load balancer. I’ll also show where a service like Protoface fits when you do not want to build the avatar side yourself.
What “horizontal scaling” means for avatar streaming
For a regular web app, horizontal scaling mostly means stateless request handling. For a streaming avatar system, you also have long-lived sessions and media transport. That changes the shape of the problem.
A session usually has four pieces of state:
Identity: who the user is and which avatar/session they are attached to.
Realtime transport: the websocket/WebRTC connection carrying audio, video, and events.
Conversation state: the transcript, agent turn state, voice settings, and instructions.
Media generation state: lip-sync timing, current utterance, frame pacing, and any queued response.
The scaling rule is simple: keep the durable state in a shared store, and make the realtime workers disposable. In practice that means your Next.js app should not “own” the media pipeline. It should authenticate users, request session creation, and hand the browser a short-lived token or session descriptor. A Node.js service should coordinate session state and delegate actual media generation/transport to workers that can be recreated at any time.
Split the system into control plane and data plane
The cleanest architecture is to separate control plane from data plane.
Control plane responsibilities:
Authenticate the user
Create or look up an avatar session
Issue short-lived credentials for the browser or agent runtime
Persist session metadata, billing counters, and audit records
Apply rate limits and admission control
Data plane responsibilities:
Accept realtime audio from the agent or browser
Generate video frames or drive a remote avatar service
Synchronize lip motion with the current utterance
Stream media back to the client with bounded latency
Once you make this split, horizontal scaling becomes straightforward: the control plane can be stateless Node.js API instances, and the data plane can be one or more worker pools behind a queue or session router.
State management: keep the browser thin, keep the server authoritative
In a Next.js app, the frontend should not hold privileged API keys or complex session logic. The browser should request a session from your backend, then connect to a realtime endpoint using ephemeral credentials. The backend remains authoritative for session creation and permissioning.
A typical flow looks like this:
User clicks “Start avatar”.
Next.js calls your Node.js API route.
The API route creates a session record and returns a short-lived token or session URL.
The browser connects directly to the realtime endpoint.
The backend receives session events asynchronously and updates durable state.
This pattern avoids the classic scaling trap where your app server becomes a bottleneck because every frame, transcript update, and websocket message has to pass through it. Let the browser talk to the realtime layer directly whenever possible. The server should coordinate, not proxy everything.
For session state, use something shared and low-latency: Redis is the usual default for ephemeral state, while Postgres or another durable store handles billing, audit logs, and user-owned metadata. If you need ordering guarantees for agent events, write them through a single session-owned stream or queue partition keyed by session ID. That keeps all events for one conversation in order without requiring a monolith.
Scale the Node.js backend by sharding responsibility, not by sharing memory
Node.js is a good fit for the control plane and for session orchestration, but only if you avoid in-memory coupling.
Do not store active sessions only in process memory. If a pod restarts, the session disappears. If two requests land on different pods, they disagree about current state. Instead:
Use a shared session store keyed by session ID
Make API routes idempotent where possible
Use a queue or pub/sub channel for async events
Keep websocket handlers thin and stateless
For example, if a request asks to create a session, the backend can atomically create a record with a status like provisioning, then update it to ready when the realtime worker or external service is attached. If the same request is retried, return the existing session rather than creating a duplicate. This matters a lot under load, especially when browsers retry on flaky networks.
How to handle the media path without introducing latency spikes
Realtime avatars are sensitive to jitter. Lip sync looks wrong when audio frames arrive late or out of order, and a video face that is 400–600 ms behind the voice already feels broken to users. The architecture should therefore optimize for predictable latency rather than maximum throughput alone.
A few practical rules:
Prefer streaming over batch: generate and forward incremental utterances rather than waiting for full responses.
Bound queue depth: if the user is speaking, do not let a worker backlog several seconds of stale output.
Use session affinity only where needed: keep the control plane stateless, but route one session’s media stream to the same worker while it is active.
Backpressure aggressively: when a worker is overloaded, reject or delay new sessions instead of letting all sessions degrade.
If your avatar is driven by a voice agent, the best experience usually comes from a tight turn loop: speech recognition emits partial text, the LLM produces a streaming response, text is chunked into utterance-sized segments, and the avatar layer renders while audio is still flowing. That requires the media worker to understand turn state, not just raw bytes.
Next.js integration pattern: server actions or API routes, never direct secrets
With Next.js, keep client components dumb. Use an API route or server action to create sessions and return only what the client needs to connect. The common mistake is to call third-party APIs from the browser because “it works in dev.” It works until you ship API keys and create a security problem.
A minimal control-plane endpoint might look like this:
On the client, you then attach to the realtime stream using the returned token. The browser never sees your long-lived credentials, and your backend can revoke or expire sessions centrally.
If you need to run your own voice pipeline in Node.js, use worker processes or separate services for CPU-heavy tasks. Node’s event loop is good at coordinating connections; it is not where you want to do expensive media processing.
Where Protoface fits: offload the avatar layer, keep your app architecture
If your product is about the conversation and not about building a video synthesis stack from scratch, the avatar layer is a strong candidate to outsource. The practical integration point for a voice-agent stack is the LiveKit Agents plugin, which drops a synchronized talking face into an existing agent. That keeps your application focused on session orchestration while the avatar service handles the media-specific details.
For example, in a Python-based agent service, you would install the plugin and wire it into your agent pipeline; the exact fields and configuration live in the docs, but the shape is the usual one: initialize the avatar service, attach it to the agent, and stream the agent’s speech to the avatar layer.
If you are scripting session creation instead, the REST API is the clean integration point. A typical flow is to POST a session from your backend, then hand the browser or agent runtime an ephemeral handle. For example:
Exact request and response fields are documented in the docs. The important architectural point is that the API gives you a clean boundary: your Next.js and Node.js code stays focused on auth, routing, and product logic, while the avatar runtime stays isolated.
Operational gotchas that matter in production
There are a few failure modes worth designing for up front:
Reconnect storms: when a provider blips, many clients reconnect at once. Add jittered retries and a session lease with expiration.
Orphaned sessions: if a browser closes unexpectedly, mark the session inactive after a timeout and reclaim resources.
Double-submit creation: make session creation idempotent so refreshes do not create duplicate avatars or billing events.
Uneven load: shard sessions across workers by session ID, not by user ID or random choice, so turn ordering stays intact.
Observability gaps: log session IDs, worker IDs, latency, and disconnect reasons in every layer.
Also distinguish between session lifetime and connection lifetime. A connection can drop and reconnect without invalidating the conversation. Treat the session as the durable unit, and the connection as an attach/detach detail. That design is what lets you horizontally scale without making every transient network event catastrophic.
Conclusion
The scalable architecture is not complicated once you separate concerns. Make Next.js responsible for presentation and session initiation. Make Node.js the control plane with shared state and idempotent APIs. Keep the realtime media path stateless where possible, and route per-session streaming through bounded, disposable workers. That gives you a system that can add capacity by adding instances, not by rewriting assumptions about process memory.
If you want to avoid building the avatar/media layer yourself, use the LiveKit plugin or REST API pattern above and keep your own backend thin. Start with the docs at docs.protoface.com, then wire the smallest possible end-to-end flow before you optimize for scale. The architecture will be much easier to reason about if you get the control-plane/data-plane split right on day one.