Introduction

If you are building a voice agent in Next.js, the hard part is usually not transcription or LLM orchestration. It is the realtime plumbing: keeping a low-latency audio loop stable, streaming partial ASR results quickly enough to feel conversational, and attaching a video face without turning your frontend into a media-engine project.

Protoface is designed for that last mile: realtime avatars that can be dropped into an agent stack without you having to build facial rendering, lip sync, or session management from scratch. In this post, I’ll show how to wire a Next.js app to a streaming ASR pipeline, how to pass results into your agent loop, and where Protoface fits when you want the agent to speak through a synchronized video face.

By the end, you should be able to:

• capture microphone audio in a Next.js client safely,

• stream audio to an ASR backend with incremental transcripts,

• feed transcript events into your voice-agent logic, and

• attach a realtime avatar session so the agent is not just audible, but visibly present.

How streaming ASR fits into a Next.js voice agent

For a voice agent, ASR is not a one-shot RPC. You want chunked audio upload, partial hypotheses, and endpointing so your downstream LLM or dialogue manager can react before the user finishes every sentence. The basic flow looks like this:

1. The browser captures microphone audio using getUserMedia.

2. You encode or forward those frames to a backend streaming endpoint.

3. The ASR service returns partial and final transcript events.

4. Your agent loop consumes transcript events, decides when to respond, and generates speech.

5. If you have an avatar session, the TTS output and lip-sync/video session are driven from the same response stream.

The important architectural detail is that Next.js should not become the media server. Use it for the UI and lightweight session orchestration, but keep the realtime audio path on the client-to-backend channel that is simplest for your stack. In practice, that means a browser component, a streaming transport, and a minimal API route or separate worker for session setup.

Capturing microphone audio in a Next.js client

The browser side is straightforward, but there are a few constraints worth respecting. Microphone access must happen in a client component, and you should treat the audio stream as ephemeral data: don’t buffer more than you need, don’t store raw audio in React state, and don’t block the render thread while processing frames.

A minimal client-side start looks like this:

useEffect(() => {

}, []);

useEffect(() => {

}, []);

useEffect(() => {

}, []);

From there, you typically connect the stream to an AudioWorklet, MediaRecorder, or a WebRTC/WebSocket transport depending on your ASR backend. For low-latency voice agents, the most important thing is the cadence of chunks and the timing of partial transcripts. A 20–100 ms audio frame size is common; larger chunks reduce overhead but increase latency and make the agent feel slower.

Two implementation details matter more than most people expect:

• Endpointing: decide when silence means “the user is done” versus “the user is pausing.” Bad endpointing makes barge-in and turn-taking feel broken.

• Backpressure: if your ASR backend falls behind, drop or coalesce frames deliberately rather than letting the browser queue grow without bound.

Streaming ASR events into your agent loop

Once you have partial and final transcripts, the agent loop should be event-driven. Don’t wait for a complete transcript if you want fast reactions. A partial hypothesis can be enough to update intent detection, turn state, or a live captioning UI. Final transcript events should be the trigger for a full LLM turn or a confirmed semantic action.

One simple pattern is to keep a small transcript state machine on the backend:

type TranscriptEvent =

}

type TranscriptEvent =

}

type TranscriptEvent =

}

The practical reason to separate partial from final is control. If you kick off generation on every partial update, you will waste tokens, create race conditions, and make interruption handling ugly. If you wait too long, the system feels sluggish. The middle ground is usually: react lightly to partials, commit on final.

For a Next.js app, this often means a server route that authenticates the session, hands out temporary credentials or a websocket target, and then lets the browser stream audio directly. Keep the state you need for turn-taking small and explicit: listening, thinking, speaking, and interrupted are usually enough to start.

Session setup and API orchestration

When you add a realtime avatar to the agent, session lifecycle becomes part of your application logic. You need a way to create the avatar session, associate it with the current conversation, and tear it down when the call ends or the user navigates away.

Protoface exposes a REST API for managing avatars and sessions, authenticated with API keys. For backend usage, that means your server, not the browser, should talk to the API. A typical request shape looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

The exact resource fields depend on the API surface you are using, so treat the payload above as illustrative and confirm the schema in the docs. The design principle is the same either way: create the session on the server, return only the minimum data the client needs, and never expose your API key to the browser.

If you prefer Python for orchestration or background jobs, the Python SDK can handle the same lifecycle from server-side code. A minimal pattern looks like this:

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

Use this layer for things like per-user session creation, usage tracking, or provisioning avatars from your own app database. The benefit is that your Next.js frontend stays thin: it only manages UI state and transports, while the server owns credentials and durable session state.

Where Protoface fits in the stack

The most natural integration point for a Next.js voice agent is the avatar session itself. If you already have ASR, TTS, and dialogue logic working, Protoface sits beside that pipeline and gives the agent a synchronized video face without requiring you to build a lip-sync/rendering subsystem. In other words, your agent still owns speech and turn-taking; the avatar session consumes the resulting speech stream and reflects it visually.

For teams using LiveKit Agents, the integration is even tighter. The Protoface integration for Pipecat and the LiveKit plugin both let you insert an avatar into an existing agent graph rather than redesigning your whole stack. If you are already using LiveKit Agents, the plugin is the most direct route: install it, configure the avatar/session details in your agent process, and let the agent’s audio output drive the face.

from livekit.agents import JobContext

pass

from livekit.agents import JobContext

pass

from livekit.agents import JobContext

pass

The point of using the plugin or SDK is not to replace your agent stack. It is to keep the avatar aligned with the same timing model your voice agent already uses: partials, final responses, interruptions, and clean shutdown. That matters more than fancy avatar configuration.

Practical gotchas

A few issues come up repeatedly in production:

• Latency compounding: ASR latency, LLM latency, and avatar/video latency add up. Optimize the whole path, not just one stage.

• Turn confusion: if your ASR endpointing is aggressive, the avatar may start responding while the user is still speaking.

• Credential handling: keep API keys server-side. If a browser ever needs session access, give it a narrow, short-lived capability, not your master key.

• Cancellation: support interruption. If the user starts talking over the agent, stop playback and reset the speaking state immediately.

• Mobile constraints: autoplay, permission prompts, and background throttling behave differently on mobile Safari and Chrome. Test there early.

If you are debugging the experience, instrument timestamps at each stage: mic capture, frame send, first transcript, final transcript, first token, first audio, avatar session start. The deltas will tell you where the perceived delay is actually coming from.

Conclusion

A Next.js voice agent is mostly a realtime systems problem: get audio off the device, turn it into reliable transcript events, and keep the dialogue loop responsive under interruption and latency. Once that works, adding a synchronized avatar is an integration problem, not a research project.

Use the browser for capture and UI, keep API keys on the server, treat partial transcripts as control signals rather than final truth, and choose an avatar integration that matches your stack. If you want the exact REST payloads, SDK methods, or plugin setup details, start with the docs and the relevant examples in the project repositories.

From there, the path is straightforward: wire ASR, connect your agent, attach the avatar session, and measure the end-to-end latency before you ship.