Implementing Speech-to-Text and Text-to-Speech for a Next.js Virtual Receptionist Avatar
Next.js virtual receptionist STT/TTS integration: streaming audio, turn-taking, barge-in, and secure avatar session setup.
Introduction
Building a virtual receptionist is mostly a systems problem: you need low-latency speech input, reliable turn-taking, responsive text generation, and speech output that feels natural when it is attached to a face. The hard part is not any single component; it is making the audio pipeline, the conversational state, and the avatar rendering stay synchronized under real network and model latency.
By the end of this post, you should be able to wire a Next.js front end to a speech-to-text and text-to-speech pipeline, understand where the latency and failure modes live, and know how to attach a realtime avatar without leaking credentials into the browser. I’ll keep the examples practical and focus on the integration details that tend to trip people up.
What the browser should do, and what it should not do
For a receptionist-style experience, the browser only needs to capture microphone audio, play back synthesized speech, and render the avatar video stream. It should not own sensitive API credentials, and it should not make ad hoc calls to model vendors directly unless you are intentionally building a fully client-side architecture.
The usual flow looks like this:
User speaks into the browser microphone.
Audio is streamed to a speech-to-text component.
The transcript is passed to your dialog logic or agent.
The agent returns a response string or tool result.
Text-to-speech converts that response into audio.
The avatar layer lip-syncs the video face to that audio.
The important detail is that STT and TTS are not just “input” and “output.” They determine the perceived responsiveness of the whole system. If STT is slow to finalize utterances, your agent feels sluggish. If TTS starts too late or chunking is inconsistent, the avatar’s mouth movement and audio cadence feel disconnected.
Speech-to-text: stream early, finalize carefully
For a receptionist, you usually want streaming transcription rather than waiting for full utterances. Partial transcripts let you start intent detection early, trigger backchannel behaviors, and reduce perceived latency. But you still need a final transcript boundary before you hand the turn to the agent, otherwise you will process incomplete input and generate incorrect responses.
In practice, there are three things to get right:
Audio format: keep it simple and consistent. Most realtime systems want mono PCM or Opus in a specific sample rate.
VAD/endpointing: decide when the user is done speaking. Too aggressive and you cut people off; too loose and the assistant waits too long.
Partial vs final text: partial results are for UX and early speculation; final results are for agent state changes.
If you are implementing this in the browser, don’t overcomplicate the capture side. Use the Web Audio API or MediaRecorder only to the extent needed by your STT backend. The main engineering goal is predictable framing and stable latency, not perfect local signal processing.
A minimal server-side STT endpoint might look like this shape:
That example is deliberately abstract. Exact payloads depend on the STT provider or agent framework you choose. The useful part is the contract: stream audio in, emit partial text quickly, and finalize only after endpointing.
Text-to-speech: plan for chunking and interruptibility
TTS is where many otherwise functional assistants start to feel bad. If your app waits for the full response before synthesizing anything, the avatar will sit idle too long. If it synthesizes very small fragments with no regard for prosody, the voice becomes robotic and the lip sync can drift from the video face.
For a good receptionist experience, the TTS layer should support:
Low startup latency: start playback as soon as the first audio chunk is ready.
Interruptibility: stop speaking immediately when the user barges in.
Stable chunk boundaries: don’t split text in ways that break pronunciation or prosody.
Backpressure handling: if the model produces text faster than you can speak it, buffer responsibly.
Operationally, the agent should own “speech state.” When the user speaks, cancel any in-flight TTS generation and stop audio playback. When the agent speaks, keep a single active audio track so the avatar can synchronize against one source of truth.
That is enough to illustrate the pattern, but in a production receptionist you usually want streaming audio, not blob-based playback. Blob playback introduces avoidable delay because you wait for the full file before starting. If your TTS provider supports incremental audio chunks, use them.
Turn-taking and barge-in are part of the architecture
A conversational receptionist is not a request/response chatbot with a microphone bolted on. It needs a turn manager. The turn manager decides when the user is speaking, when the assistant is speaking, and when the assistant should yield. Without that layer, the system will talk over the user, repeat itself, or continue speaking after the caller has already interrupted.
At minimum, keep these states explicit:
Listening: accept mic input and stream STT.
Thinking: transcript finalized, agent generating a reply.
Speaking: TTS audio is playing or being streamed.
Interrupted: user barged in; cancel TTS and return to listening.
That state machine is also what keeps avatar motion honest. If the assistant is marked as speaking, the avatar should animate. If speech is interrupted, the avatar should stop cleanly rather than “finish the sentence” visually after audio has already been cut off.
Next.js implementation details that matter
In Next.js, the biggest mistake is trying to keep everything in one component without separating browser-only code from server-side orchestration. Keep mic capture and audio playback in client components. Keep API key usage, token minting, and session creation on the server.
A common shape is:
Client: capture microphone, render avatar, play audio.
Server route: create session, exchange messages with STT/TTS/agent services, enforce policy.
Persistent state: store conversation metadata, call context, and usage records.
For the browser side, remember that audio autoplay restrictions still apply. You generally need a user gesture before starting playback. That matters for an interactive receptionist because your first response might otherwise be blocked until the user clicks somewhere on the page.
Also avoid sending raw user audio to the client-side runtime if you do not need to. Keep streaming and synthesis on the server when possible so your trust boundary stays manageable. The browser should receive the minimum data needed to render the experience.
Where Protoface fits: avatar rendering without exposing secrets
This is where Protoface is useful: once you have the speech pipeline, you still need a synchronized talking face. The cleanest path for a Next.js app is to keep your speech logic where it belongs and attach the avatar through a customer-managed iframe embed or a server-created session. That lets you keep API keys out of the browser while still giving the user an interactive, lip-synced video face.
If you are wiring the backend directly, the REST API is straightforward: create avatars and sessions from your server with a bearer key, then hand the browser a session-specific embed or tokenized URL rather than the secret itself. The docs cover the exact request shape, but the security model is the main thing to preserve.
If you are already using a voice-agent stack, the LiveKit plugin is the shortest route to a synchronized video face. The plugin drops the avatar into the agent pipeline so the avatar stays aligned with the assistant’s speech instead of being treated as a separate, loosely coupled animation layer. See the relevant quickstart and plugin examples in the GitHub organization and the docs for the exact integration points.
Common failure modes
The main bugs in this kind of system are usually not “AI bugs.” They are media and state bugs:
Transcript lag: STT finalization takes too long, so the assistant feels inattentive.
Double-speaking: a new user utterance arrives while the assistant is still producing audio.
Drift between audio and avatar: the video face keeps animating after audio stops, or vice versa.
Credential leakage: API keys end up in client code or environment variables shipped to the browser.
Poor endpointing: the system cuts off users or waits too long for silence.
Fixing these usually means tightening your state machine and shortening the path between final transcript, response generation, and first audio byte. Measure each stage independently so you can see whether latency comes from capture, transcription, generation, synthesis, or rendering.
Conclusion
A usable receptionist avatar is mostly about disciplined streaming: stream mic audio into STT, finalize turns deliberately, synthesize speech with low startup latency, and keep the avatar synchronized with the active audio source. In Next.js, keep capture and playback in the client, keep secrets on the server, and make turn-taking explicit.
If you want to add a realtime avatar without building the video face pipeline yourself, start with the docs and the relevant integration examples for your stack. From there, you can plug the avatar into your voice agent, test latency end to end, and iterate on the conversational experience before you ship.