How to Stream TTS, STT, and Avatar Video for a Next.js Support Bot
Next.js support bot architecture for streaming STT, TTS, and synchronized avatar video with server-side session tokens.
Introduction
If you’re building a Next.js support bot, the hard part is not “getting a chatbot on the page.” The hard part is streaming three things at once: speech-to-text from the user, text-to-speech from the agent, and a video surface that stays visually synchronized with the audio. If any one of those paths stalls, you get the usual failure modes: dead air, clipped responses, lip sync drift, or a UI that feels like three loosely coupled demos instead of one system.
This post shows the architecture that actually works in production: a browser client that captures mic audio, a realtime backend that handles STT and agent turns, and a video avatar stream that is driven by the same conversation state as the voice. By the end, you should be able to reason about the data flow, choose sane transport boundaries in Next.js, and integrate a realtime avatar without leaking secrets into the browser.
Start with the transport model, not the UI
The main mistake I see is treating “voice bot with video” as a frontend animation problem. It is a streaming systems problem. The browser is producing audio frames; your agent is consuming them and emitting transcripts, responses, and audio; the avatar layer is consuming the agent’s speech stream and producing synchronized video frames. Those streams have different latency and reliability characteristics, so they should not be collapsed into one monolithic request/response API.
A practical split looks like this:
Browser: captures mic audio, renders assistant audio/video, shows transcript, handles reconnects.
Agent runtime: performs STT, dialogue logic, tool calls, and TTS synthesis.
Avatar/video service: turns the agent’s speech into a talking face, typically as a realtime media stream.
For Next.js, this usually means: keep the page interactive in the client, use server routes only for issuing short-lived tokens or creating sessions, and keep any provider secret off the client entirely.
Build the browser side as a streaming client
In a support bot, you generally want a “push-to-talk” or “open mic” interaction that streams audio incrementally rather than waiting for the user to stop speaking. That gives you lower end-to-end latency and lets the backend start STT before the utterance is complete. The same principle applies in the reverse direction: once the agent starts speaking, you want TTS audio and avatar motion to begin streaming immediately, not after the full sentence is synthesized.
In Next.js, a typical pattern is:
Render the UI with a client component.
Call a server route to create or authorize a realtime session.
Open your media connection from the client using the returned ephemeral credentials.
Stream user audio up, and render agent audio/video down.
At the UI layer, keep the transcript and status indicators separate from the media connection. Don’t block the whole interface on STT completion. A good bot should visibly transition through states like “connecting,” “listening,” “thinking,” and “speaking,” because those states are the user’s only feedback when network conditions are imperfect.
Keep STT, TTS, and avatar sync loosely coupled but ordered
The cleanest mental model is a single conversation timeline with three consumers:
STT consumes user audio frames and emits partial and final transcripts.
Agent logic consumes transcripts and emits assistant text plus optional tool actions.
TTS/avatar consumes the assistant text and emits audio/video.
Partial transcripts are useful for responsiveness, but only final transcripts should usually trigger durable state changes or tool calls. If you use partials too aggressively, you end up with repeated tool execution or agents responding to half-finished sentences.
On the output side, the avatar should be driven by the exact same assistant utterance that TTS is speaking. If the avatar receives a different text stream, or if you synthesize audio from one revision while animating from another, lip sync will drift in subtle but visible ways. In practice, this means you want one “assistant turn” object that fans out to both TTS and avatar rendering.
Two implementation details matter a lot:
Backpressure: if the client cannot keep up with incoming frames, prefer bounded buffering and dropping non-essential visual frames over letting latency grow unbounded.
Turn cancellation: if the user interrupts the bot, you need to stop TTS and avatar playback promptly and mark the current turn as abandoned.
Next.js implementation: keep secrets server-side
Your Next.js app should never expose long-lived API keys to the browser. The browser can hold ephemeral session material, but the credential that creates those sessions belongs on the server.
A minimal server route to create an avatar session might look like this. The exact field names depend on your setup, but the shape is representative:
In the client, you fetch that route, then connect your media layer with the returned session data. The important point is not the precise JSON schema; it’s that session creation is server-side and connection setup is client-side.
If you need to inspect or rotate keys, manage usage, or replay a session in a controlled environment, the dashboard at docs.protoface.com points you in the right direction for the platform primitives and operational details.
Where Protoface fits without complicating the stack
If your agent is already running in a LiveKit-based voice stack, the most direct integration is the LiveKit plugin. It lets you drop a synchronized talking face into an existing agent so you don’t have to invent a separate avatar pipeline. The plugin is published as livekit-plugins-protoface, and the public repository and examples are the best place to start if you want to wire this into a voice agent quickly: GitHub quickstart and the plugin docs on Pipecat if that is your orchestration layer.
In a LiveKit agent, the integration point is usually the point where the assistant’s synthesized speech is already flowing. At that stage, the avatar service can use the same turn audio to render the face, which avoids the common “audio says one thing, face says another” problem. Conceptually, you are not adding a second independent bot; you are attaching a video output to an existing realtime voice agent.
For teams that want direct control outside LiveKit, the REST API and Python SDK are the other useful surfaces. The SDK is appropriate when your backend creates avatars or sessions programmatically, while the REST API is convenient for automation, admin tools, and backend services that already talk HTTP. A simple curl flow looks like this:
Again, exact endpoints and fields live in the docs; the key idea is to create assets and sessions server-side, then stream them into your app with short-lived runtime credentials.
Practical gotchas in support-bot integrations
1. Don’t wait for full sentences. Waiting for complete STT transcripts before responding makes the bot feel sluggish. Use partial transcripts for UX, but only finalize actions on stable text.
2. Handle barge-in explicitly. In support flows, users interrupt constantly. When that happens, stop TTS, stop avatar playback, and let STT take over immediately.
3. Keep token issuance server-side. If your Next.js app needs to expose a session or embed, do it through a backend route or a customer-managed iframe flow. The browser should never contain your main API key.
4. Measure end-to-end latency, not just model latency. A fast STT model with a slow video path still feels slow. Track capture-to-first-transcript, transcript-to-first-audio, and audio-to-first-frame separately.
5. Test reconnects and tab suspension. Browsers will suspend tabs, mobile devices will sleep, and networks will change. Your client should reconnect cleanly without duplicating turns or replaying stale audio.
When an iframe is the better answer
If your immediate goal is to embed an interactive avatar on a support page and you do not want to expose any backend logic in the browser, a customer-managed iframe is often the simplest route. It gives you an isolated embed with parent-origin allowlisting, per-embed voice and instructions, and runtime limits such as duration and per-IP controls. That is especially useful when the avatar is the interface itself rather than a component inside a larger app shell.
For a Next.js support bot, I would still treat the iframe path as a separate deployment model, not the default architecture for a complex agent. It is excellent when you want a clean boundary and minimal frontend work. It is less suitable when you need deep integration with your app’s auth, navigation, and state.
Conclusion
Streaming TTS, STT, and avatar video in a Next.js support bot is mostly about respecting the shape of the system: audio in, text and events through the agent, synchronized audio/video out, with secrets kept off the client. If you keep those channels loosely coupled, handle interruption and reconnects explicitly, and issue runtime credentials from the server, the result feels much more like a realtime assistant and much less like a stitched-together demo.
If you want to go deeper, start with the public docs at docs.protoface.com, then pick the integration surface that matches your stack: the LiveKit plugin for existing voice agents, the REST API for backend-driven session management, or the iframe embed if you need the fastest path to a browser-facing avatar.