Introduction

Realtime AI avatars look simple from the outside: a user speaks, the model responds, and a talking face animates in sync. Under the hood, the audio path is where most integrations get messy. If you pipe microphone input through the wrong sample rate, resample twice, block the event loop, or let jitter accumulate between ASR, LLM, TTS, and video synthesis, the avatar will drift, clip, or feel laggy even if the model itself is fine.

This post is about setting up a clean audio pipeline for a Next.js application that fronts a realtime avatar experience. By the end, you should be able to reason about the audio path end to end, choose where to do conversion and buffering, and avoid the common failure modes that make realtime agents feel brittle.

Start with the pipeline, not the UI

For a realtime avatar, the browser is just one participant in a larger media graph. The typical flow is:

Mic input in the browser → capture and encode → transport to your agent service → speech recognition or audio-forwarding → model turn generation → text-to-speech → synthesized audio streamed back → avatar lip sync and video rendering.

The important detail is that audio is not “just another websocket payload.” It has timing constraints. You care about:

• Sample rate: keep it consistent across capture, processing, and synthesis whenever possible.

• Frame size: small enough for low latency, large enough to avoid excessive overhead.

• Jitter: buffering too little causes choppiness; buffering too much increases conversational lag.

• Clock domain: browser audio clocks, server clocks, and media server clocks are not the same thing.

In Next.js, a common mistake is to push raw MediaRecorder chunks directly into an API route and call it a day. That works for demos, but it usually introduces variable chunk sizes, container overhead, and unnecessary latency. For realtime agent turns, you want a streaming path that preserves cadence and avoids unnecessary transcoding.

Capture clean audio in the browser

On the client, use the Web Audio APIs or a media stack that gives you predictable PCM or Opus output. The key is to choose one format and stick to it. If your backend or media layer expects 16 kHz mono PCM, don’t capture 48 kHz stereo in the browser and resample several times in different layers.

In a Next.js app, the capture component should do the minimum possible work:

1. Request microphone access with explicit constraints.

2. Normalize to mono unless you have a specific reason not to.

3. Package audio into small, steady chunks.

4. Forward the stream to your realtime backend over a low-latency transport.

A minimal capture example might look like this:

const stream = await navigator.mediaDevices.getUserMedia({

const stream = await navigator.mediaDevices.getUserMedia({

const stream = await navigator.mediaDevices.getUserMedia({

For production, prefer an AudioWorklet or a purpose-built streaming library over MediaRecorder if you need stable latency. MediaRecorder is convenient, but it emits containerized chunks at intervals you do not fully control, which is often the wrong shape for agent turn-taking.

Keep conversion in one place

Clean pipelines are usually boring pipelines. One of the best things you can do is decide where audio normalization happens and prevent it from happening elsewhere. That means answering these questions up front:

• What sample rate will the agent service consume?

• Will the agent generate PCM, Opus, or another transport format?

• Does the media layer need raw audio, or can it accept encoded frames?

• Where do you resample, if at all?

For example, if your browser captures 48 kHz mono, but your speech engine and avatar pipeline want 24 kHz or 16 kHz, resample once at the edge or in a dedicated media worker. Do not resample in the browser, again in your Next.js route, and again inside the agent. Every extra conversion adds CPU cost and may introduce artifacts.

Likewise, be explicit about buffering. If you accumulate too many milliseconds before forwarding audio, the user hears turn-taking lag. If you forward tiny fragments without smoothing, you can create network chatter and unstable downstream decode behavior. A practical target is to stream consistent short frames and let the media layer handle packetization.

Next.js specifics: server boundaries matter

Next.js is fine as the frontend shell for a realtime avatar app, but it should not become your media engine. API routes and server actions are useful for authentication, session creation, and issuing temporary tokens, not for long-lived audio processing loops.

In practice, the architecture is usually:

• Browser handles capture, playback, and avatar UI.

• Next.js server code issues session setup and policy decisions.

• A dedicated realtime service handles streaming audio and model orchestration.

This separation matters because serverless or edge-style execution environments are poor fits for persistent audio streams. They introduce cold starts, time limits, and concurrency constraints that are awkward for bidirectional media. Use Next.js to control access and render the app, then move the actual realtime work into a service that is designed for it.

Jitter, echo, and the practical gotchas

Most “audio quality” bugs in realtime avatars are actually pipeline bugs. A few worth checking early:

• Echo feedback: if the agent’s own output is re-captured by the microphone, turn on echo cancellation and keep speaker output isolated from input when possible.

• Sample-rate mismatch: if you see subtle pitch shifts or timing drift, verify every hop in the chain.

• Chunk boundary errors: if frame sizes vary wildly, downstream models may stutter or respond late.

• Backpressure: if your client keeps capturing while the network is stalled, you need to decide whether to drop, buffer, or resync frames.

• Double encoding: raw PCM should not be encoded and decoded repeatedly just to move between components.

A useful debugging habit is to inspect the actual frames at each stage. If possible, log metadata such as sample rate, channel count, frame duration, and queue depth. That makes it much easier to see whether the issue is capture, transport, synthesis, or rendering.

How Protoface fits into this

This is where a dedicated avatar layer helps. With Protoface, you do not have to build the lip-sync/video face handling yourself. If you are already running a voice agent, the LiveKit plugin can drop a synchronized avatar into the agent so the audio pipeline and the talking face stay aligned. For a Next.js app, that means your frontend can stay focused on clean capture and playback while the avatar logic lives in the media layer where it belongs.

If you need to provision sessions or manage avatars programmatically, use the REST API or the Python SDK from your backend rather than exposing secrets in the browser. A simple server-side setup might look like this:

import os

print(resp.json())

import os

print(resp.json())

import os

print(resp.json())

And if you are using the LiveKit voice-agent path, the integration point is the plugin package documented in the GitHub repo and on PyPI. The important part is not the specific import path, but the division of labor: your agent owns conversational state and audio turn-taking, while the avatar layer consumes the agent’s spoken output and renders it in sync.

For implementation details, the docs are the right place to confirm exact request fields, session lifecycle behavior, and current integration patterns: docs.protoface.com.

A clean setup pattern for a Next.js app

If I were wiring this up today, I would keep the system intentionally boring:

1. Next.js frontend: microphone capture, playback, and UI state.

2. Backend route or service: authenticate the user and create a realtime session.

3. Realtime media layer: carry audio frames, agent responses, and avatar sync.

4. Avatar provider: consume the synthesized speech and produce the talking face.

That division makes it easier to test each boundary. You can swap capture implementations without touching avatar rendering, and you can tune audio framing without rewriting your app shell. It also makes production issues easier to isolate: if the model sounds fine but the face is late, you know to inspect the media sync path rather than the frontend.

If you want to look at concrete starting points, the repository of quickstarts is useful for seeing how different agent stacks are wired together, including voice-agent integrations and browser-based demos: GitHub quickstart.

Conclusion

Realtime avatar apps succeed or fail on audio discipline. Keep capture stable, convert formats once, avoid pushing long-lived audio logic into Next.js server routes, and make buffering and latency trade-offs explicit. If your audio path is clean, everything downstream becomes easier: the agent feels faster, lip sync is steadier, and debugging stops being guesswork.

When you are ready to wire this into a real stack, start from the docs, pick the integration surface that matches your architecture, and keep the browser focused on capture and playback rather than media orchestration. The shortest path to a good avatar experience is usually the one with the fewest audio transformations.