Adding Dynamic TTS Voices to a WebRTC Avatar Stream in TypeScript
TypeScript patterns for dynamic TTS voice switching in WebRTC avatar streams, with state, buffering, and lip-sync handling.
Introduction
Adding a dynamic text-to-speech voice to a WebRTC avatar stream is mostly an exercise in state management and latency control. You need to take text from your application, synthesize audio with the right voice for the current moment, and keep that audio aligned with the avatar’s video so the result still looks like one coherent real-time agent instead of two unrelated streams.
In practice, that means handling three things well:
choosing or switching voices at runtime without tearing down the session,
routing TTS audio into the same realtime pipeline as the avatar, and
making sure timing, buffering, and voice selection remain deterministic enough for conversational UX.
This post walks through the implementation shape in TypeScript: how to model voice changes, how the WebRTC path fits together, where the common failure modes are, and how Protoface fits into the stack when you want a synchronized talking face rather than a plain audio bot.
What “dynamic TTS voice” actually means in a WebRTC avatar pipeline
In a browser or server-driven WebRTC session, the avatar is typically consuming audio as a live stream. If you swap voices dynamically, you are not just changing a config flag in the UI. You are changing the source characteristics of the audio being rendered and, depending on the system, possibly the speech rate, prosody, and phoneme timing that drive lip sync.
There are two common models:
Per-utterance voice selection — choose a voice before synthesizing each response. This is the simplest and safest approach.
Mid-session voice switching — change the voice while the session stays active. This can be useful for role changes, multilingual flows, or “persona” transitions, but you must avoid cutting off buffered audio or desynchronizing the avatar.
The main rule is: keep the session stable, vary the synthesis parameters. In other words, do not recreate the WebRTC connection just because the user selected a different voice. Create a new audio generation task, let it produce audio for that voice, and feed that audio into the existing avatar session.
Model the voice as part of conversation state
In TypeScript, the cleanest pattern is to represent voice as a small piece of conversation/session state that can be updated independently from the rest of the agent. That keeps your agent logic explicit and makes it easier to preserve voice preference across turns.
When a user asks to “sound calmer” or when your agent switches personas, update state.voice, then route the next utterance through the new voice. If your TTS provider supports streaming synthesis, you can apply the change on the next chunk boundary; if it only supports full-utterance synthesis, switch at utterance boundaries only.
The practical reason for this separation is reliability. Voice state is application state; WebRTC is transport state. Mixing them creates brittle reconnection logic and increases the chance of audio glitches.
Route synthesized audio into the avatar stream
The avatar pipeline usually looks like this:
your app produces text,
a TTS engine synthesizes audio for the currently selected voice,
the audio is streamed or uploaded to the avatar session,
the avatar lip-syncs against that audio and publishes a video track over WebRTC.
For a browser client, the important distinction is whether your TTS happens client-side or server-side:
Client-side TTS can reduce backend complexity, but it exposes vendor credentials if the provider requires them and makes high-quality audio timing harder to control.
Server-side TTS is usually the better choice for production agents. It centralizes voice selection, keeps secrets off the client, and gives you one place to normalize timing and barge-in behavior.
If your avatar expects audio frames or a synthesized speech clip, the exact handoff depends on the avatar surface you use. The important part is that the audio you send should be aligned to the response that produced it. If the user interrupts, cancel the in-flight synthesis and discard the remaining buffered audio. Otherwise you will get a visible “keeps talking after interruption” bug even if the audio path itself works.
TypeScript example: dynamic voice selection per turn
The snippet below shows the basic orchestration: take a selected voice, synthesize the response, then forward the audio into the live avatar session. The exact SDK methods and payload fields vary by provider, so treat this as structure rather than copy-paste.
There are two useful extensions to this pattern:
Voice overrides — accept a transient voice for one response, then revert to the default session voice.
Persona-bound voices — bind a specific voice to a role, such as support agent, narrator, or NPC.
For the latter, keep the mapping in application config, not scattered through business logic. It makes A/B tests and updates much easier.
Handle buffering, interruption, and speech boundaries
Most real bugs in realtime avatar audio are not about synthesis quality; they are about boundary handling. A good implementation needs to know when speech starts and ends, and what to do when it is interrupted.
Three rules help:
Start only when the response is stable. If your LLM is still generating, do not emit partial speech unless you deliberately support streaming TTS.
Cancel aggressively on barge-in. If the user starts speaking, stop synthesis and clear any unplayed audio buffered for the avatar.
Preserve utterance ordering. If you allow multiple turns to synthesize concurrently, tag each response with an ID and discard stale audio when a newer turn wins.
In a WebRTC pipeline, even small delays become visible. If the audio arrives late relative to the video frame timing, the avatar can appear to “prepare” a mouth shape and then lag behind the spoken content. Avoid long prebuffers unless your transport is unstable; real-time conversational UX usually favors low latency over perfect audio smoothing.
Operational concerns: voice catalogs, retries, and observability
Dynamic voice switching becomes manageable when you treat voice selection as a first-class config object. A few implementation details matter in production:
Validate voice IDs early so you fail before generating audio.
Cache provider metadata if voice catalogs are large or slow to fetch.
Retry synthesis carefully; retrying a partially heard utterance can sound duplicated or uncanny.
Log voice choice per utterance so you can debug quality regressions and user reports.
If you are exposing user-selectable voices, be explicit about whether the setting applies immediately or on the next turn. “Immediate” often implies interrupting current speech, while “next turn” is safer and easier to reason about.
For teams shipping multiple voice agents, it is worth keeping a small internal matrix of approved voices per use case. An energetic voice that works for a game NPC may be a poor fit for support, and vice versa. Technical success here is less about having many voice options and more about making the choice deterministic and testable.
Where Protoface fits
This is exactly the kind of orchestration Protoface is built to sit behind: you keep your own conversation logic and dynamic voice selection, and then hand the resulting speech/audio into a synchronized avatar session over the API or a LiveKit-based integration. If you are already running a LiveKit voice agent, the quickstart repo and the LiveKit plugin path show the shape of dropping in a talking face without rewriting your agent transport. If you need to create or manage sessions directly, use the REST API or SDK and keep the browser free of secrets; the exact request fields are documented at docs.protoface.com.
The exact endpoint and payload shape depend on the resource you are creating, but the pattern is consistent: authenticate from the server, create or update the session, and keep the browser limited to the WebRTC/media path.
Conclusion
Dynamic TTS voice selection is straightforward once you separate session transport from utterance synthesis. Keep the WebRTC/avatar connection stable, treat voice as mutable conversation state, switch on utterance boundaries, and be strict about cancellation and buffering. That gives you a system that can change persona, locale, or tone without visible artifacts.
If you want to implement this with a managed avatar backend rather than wiring the media path yourself, start with the docs, pick the integration surface that matches your stack, and test the voice-switching behavior under interruption and back-to-back turns. The fastest path to a robust result is usually a small, explicit state machine and one well-defined speech handoff point.