Introduction

Building a realtime avatar is not just “stream audio into a face.” In a production voice agent, text generation, speech synthesis, speech recognition, and video rendering all operate on different clocks, and the system has to survive interruptions: barge-in, language changes, partial transcripts, model retries, and network jitter. If you get the integration wrong, the avatar will speak over the user, lip-sync will drift, or the agent will continue answering in the wrong language after a user switches mid-conversation.

This post focuses on the implementation details that matter: how to structure TTS, STT, and language switching so your avatar behaves like a single coherent conversational system. By the end, you should be able to design a loop that preserves latency, keeps video and audio aligned, and handles multilingual interactions without rebuilding your agent pipeline every time the user changes language.

Start with a single source of truth for conversation state

The most common mistake is to let STT, LLM, TTS, and avatar rendering each infer the conversation state independently. That usually works for a demo and fails under real interaction. Instead, keep one canonical conversation controller that owns:

• the current language or language set

• the active turn and whether the user is speaking

• the current assistant response generation

• the TTS job associated with that response

• the video avatar session that should be rendering the audio

Every subsystem should publish events into that controller, not directly mutate each other. For example, STT emits partial and final transcripts; the controller decides whether a partial is strong enough to trigger a response, whether to cancel TTS on barge-in, and whether the current utterance language differs from the last one.

This matters because language switching is not just a text problem. If the user starts in English and then asks a follow-up in Spanish, you need to re-evaluate:

• which STT locale to use for the next chunk

• which prompt or system instruction to feed the LLM

• which TTS voice or voice profile to synthesize with

• how the avatar session should continue lip-syncing without a visible reset

Keep the controller explicit. A small state machine is better than “best effort” heuristics sprinkled throughout the codebase.

TTS and STT should be chunked, not batch-oriented

Realtime avatar systems work best when both speech recognition and synthesis are streaming. If you wait for full-turn transcripts before responding, latency accumulates. If you wait for a full TTS sentence before starting playback, the avatar feels disconnected. The sweet spot is incremental processing with clear cancellation semantics.

On the STT side, you want partial hypotheses for responsiveness, but only final transcripts should generally trigger irreversible actions. Partial text is useful for:

• language detection

• barge-in detection

• turn-taking heuristics

• speculative response preparation

On the TTS side, prefer an interface that emits audio chunks as soon as the model can produce them. That allows the avatar to begin lip-syncing early. However, you also need the ability to cancel an in-flight synthesis job cleanly when the user interrupts. If the user starts talking, the current assistant speech should stop immediately, and the avatar should transition back to listening state without a visible tail of stale audio.

A useful pattern is:

1. STT emits partial transcript events.

2. The controller determines a tentative language and turn boundary.

3. The LLM starts producing response text.

4. TTS streams audio chunks as response text arrives.

5. The avatar renders audio and lip-sync in lockstep.

6. If the user interrupts, cancel both LLM generation and TTS synthesis.

This pipeline keeps your turn latency low while preserving control. The key is that each stage is abortable. Realtime means “interruptible,” not just “fast.”

Language switching is a routing problem, not a UI toggle

It is tempting to think of language switching as “change the voice and prompt when the user says a different language.” That is too coarse. In practice, you need to route each turn through the right combination of STT, prompt context, and TTS voice based on the user’s current utterance, not the session’s original language.

Detect language early, but commit carefully

Language detection usually comes from the STT stream, not from a separate detector. Partial transcripts can be enough to estimate language quickly, but they are noisy. A practical strategy is:

• use the first few hundred milliseconds to get a provisional language guess

• prefer the language with the highest confidence over a short window

• only switch the session’s active language after a stable threshold or a final transcript

This avoids thrashing when a user mixes terms from multiple languages or when ASR confidence is low. It also avoids a bad user experience where the assistant flips between voices mid-response.

Keep STT locale and TTS voice decoupled

People often couple STT locale and TTS voice too tightly. They are related, but not the same setting. STT locale should match the user’s current input language as closely as possible, because it affects recognition quality. TTS voice, by contrast, is an output choice and can be switched for clarity, branding, or user preference.

That means a multilingual agent can legitimately recognize Spanish, answer in English, and still keep the same avatar session. The correct behavior depends on product requirements:

• Customer support: often mirror the user’s language.

• Sales or tutoring: often answer in the user’s language, but with a consistent accent or voice profile.

• Game NPCs or assistants: sometimes intentionally fix the output language while accepting multiple input languages.

Architecturally, treat language selection as metadata carried with each turn. Don’t bake it into the session globally unless your product truly requires a single-language conversation.

Barge-in and cancellation are first-class features

In an avatar UI, users expect to interrupt. If the assistant keeps talking after the user starts speaking, the experience feels broken even if the underlying language model is correct. So implement barge-in as a high-priority event that cancels downstream work:

• stop accepting new TTS chunks

• cancel synthesis for the current response

• stop or fade avatar playback immediately

• switch the conversation controller back to listening mode

There is an important edge case here: if your avatar is driven by a WebRTC media pipeline, the audio buffer and the video render path may not stop at the exact same instant. Make the cancellation boundary explicit and consistent. A tiny amount of tail audio is usually worse than a clean, immediate stop, because the user hears the assistant talking over them.

In practice, you want cancellation to be idempotent. Multiple interrupts should not corrupt state or create stuck sessions. A robust controller should tolerate duplicate “stop current response” events and ignore late-arriving TTS chunks from jobs that were already canceled.

Practical implementation with Protoface and a voice agent

If you are using Protoface inside a voice agent, the integration point is usually the video face attached to your existing audio pipeline. For LiveKit-based agents, the Pipecat/Protoface plugin is the cleanest way to keep avatar rendering synchronized with your agent’s audio output. Your agent continues handling STT and LLM logic, while the avatar session consumes the synthesized speech stream and renders a talking face with matching timing.

For lower-level session management or programmatic control, the REST API and Python SDK are the right tools. The exact request fields are documented, but the shape is straightforward: create an avatar or session, pass the configuration you need, and attach it to your runtime. A minimal API call looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

For Python, the SDK is useful when you want to create or inspect sessions from your own backend before wiring them into a conversational stack:

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

The important implementation detail is not the exact method name; it is that the avatar lifecycle should be owned by your backend or agent runtime, not by scattered frontend code. That keeps cancellation, locale selection, and session cleanup consistent. If you are building with LiveKit Agents, the plugin examples in the plugin repo are a good reference for how to wire the avatar into the media pipeline without inventing your own lip-sync layer from scratch.

Operational details that save you later

A few production concerns show up repeatedly in multilingual avatar systems:

• Rate limits and session duration: long-lived sessions need guardrails so runaway loops do not consume resources indefinitely.

• Per-language voice choice: predefine acceptable TTS voices per language instead of selecting one dynamically from arbitrary text.

• Fallback behavior: if language confidence is low, ask a clarifying question rather than forcing a bad translation path.

• Logging: record language decisions, STT confidence, cancellation events, and TTS timing so you can debug mismatched turns.

• Prompt hygiene: if the LLM is expected to answer in the user’s language, include that rule explicitly in the system or per-turn instructions.

Also remember that visual continuity matters. Even if your conversation backend changes language, the avatar should feel like the same entity. Avoid resetting the session unless you have a concrete reason. Keep the face, timing, and speaking style stable; only switch the language-specific parts of the pipeline.

Conclusion

Multilingual realtime avatars are mostly an orchestration problem. The hard part is not generating speech or recognizing speech in isolation; it is coordinating STT, LLM, TTS, and avatar playback so that the system stays interruptible, low-latency, and language-aware. The best implementations treat language as per-turn state, make every downstream stage cancelable, and keep avatar rendering attached to the same conversation controller that owns voice and text flow.

If you are integrating this into an existing voice agent, start by making your turn state explicit and instrumented. Then add streaming STT, streaming TTS, and barge-in cancellation before you tackle fancier behavior like automatic language switching. For integration details and examples, see the docs and the relevant quickstarts in the GitHub org.