Introduction

If you are building a realtime avatar into a voice agent, the hard part is not rendering a face. It is keeping three streaming systems aligned: automatic language detection, text-to-speech selection, and the avatar’s lip-sync / timing loop. In practice, the agent may start in English, switch to Spanish mid-session, and need a different TTS voice without breaking turn-taking or producing visibly stale mouth movement.

This post shows a practical way to structure that pipeline with Pipecat and a realtime avatar backend. By the end, you should understand how to:

• detect the user’s language from live audio or transcript fragments,

• route generation to the correct TTS voice model,

• switch voices cleanly during a session, and

• keep avatar video synchronized so the experience still feels like one continuous conversation.

What makes multilingual avatar systems hard

A multilingual avatar is not just “speech in, speech out.” It is a state machine running over a streaming transport. Your app is typically receiving partial ASR hypotheses, deciding when the user has finished a turn, generating an answer, synthesizing audio, and feeding that audio into a video avatar pipeline that needs reasonably stable timing to lip-sync well.

The common failure modes are predictable:

• Language drift: the user code-switches, but the system keeps speaking in the old language.

• Voice mismatch: the TTS engine changes language but not speaker identity, so the voice sounds unnatural for the locale.

• Timing gaps: you swap voices or models mid-stream and the avatar shows dead air or an obvious reset.

• Over-eager detection: a few foreign words trigger a language change too early.

The fix is to treat language detection as a confidence-driven control signal, not a single magic label. Use it to select defaults, then allow deliberate switching only when the evidence is strong enough.

Detect language from streaming input, not just final text

In realtime systems, waiting for a fully finalized transcript is often too late. You want a cheap, low-latency language signal early in the turn so downstream components can prepare the right model or voice.

A practical strategy is:

1. Run ASR with partial results.

2. Aggregate short text windows, for example the last 1–3 seconds of stable partial transcript.

3. Detect language on those windows with a confidence threshold.

4. Only switch if the new language remains stable across multiple windows or crosses a high-confidence threshold.

That gives you a debounce layer over the detector. Without it, code-switching users will cause unnecessary churn.

In code, the shape usually looks like this:

def maybe_update_language(state, transcript_chunk):

return state

def maybe_update_language(state, transcript_chunk):

return state

def maybe_update_language(state, transcript_chunk):

return state

The exact detector is up to you. The important part is the policy around it. For multilingual conversational systems, low-latency and false-positive resistance matter more than raw benchmark accuracy.

Route TTS by language, but keep the session stateful

Once the system has a language decision, the next step is selecting a TTS voice that matches both the language and the conversational persona. A good multilingual setup usually maintains a mapping like:

• language code → preferred voice ID

• language code → prosody / speaking-rate defaults

• optional fallback voice for unsupported languages

The important design choice is where that mapping lives. If you rebuild the voice choice on every turn from scratch, you will get inconsistent behavior. Instead, keep session state that tracks the current active language and voice, and only update it when the detector says the conversation has actually changed.

Voice switching itself should be treated as a boundary event. In a realtime pipeline, this usually means:

• finish or flush the current audio chunk cleanly,

• reconfigure the TTS service for the next response,

• preserve the conversation context so the assistant does not “forget” tone or persona, and

• avoid changing voices in the middle of a single synthesized sentence unless you have a very specific reason.

If you are using a provider that supports multiple language-specific voices, the best experience usually comes from switching voices between turns, not inside a turn. That keeps the avatar’s speech natural and avoids audible seams.

Here is a minimal configuration pattern in Python:

VOICE_BY_LANG = {

state.current_voice = select_voice(state.current_language)

VOICE_BY_LANG = {

state.current_voice = select_voice(state.current_language)

VOICE_BY_LANG = {

state.current_voice = select_voice(state.current_language)

If your TTS vendor exposes speaker embeddings or language-specific synthesis controls, the same pattern still applies: choose a stable voice profile per language and switch only when the language state changes.

Keep avatar lip-sync aligned with the audio stream

The avatar layer does not need to know your language detector exists. It only needs a clean audio stream with predictable timing. That separation is useful: language selection is an application concern, lip-sync is a media concern.

A few practical rules help a lot:

• Keep buffering bounded. Excess buffering improves smoothness but increases latency and makes turn-taking feel sluggish.

• Emit audio in consistent chunks. Very small chunks can increase overhead; very large chunks can make the mouth motion lag behind speech onset.

• Do not tear down the media session to switch voices. Reconfigure the generation side, but keep the realtime transport alive.

• Allow a brief silence gap only at turn boundaries. If a voice switch requires a short pause, hide it between turns rather than during speech.

In practice, your architecture is usually:

mic -> ASR -> language detection -> LLM -> TTS router -> avatar audio input -> synced video output

mic -> ASR -> language detection -> LLM -> TTS router -> avatar audio input -> synced video output

mic -> ASR -> language detection -> LLM -> TTS router -> avatar audio input -> synced video output

The avatar service should be the last step in that chain. Everything before it is about deciding what to say and in which voice; everything after it is about rendering the result consistently.

How to wire this into Pipecat

Pipecat is a good fit when you want a programmable pipeline with explicit control over language routing and TTS selection. The integration point is the avatar/video service stage, which can be inserted into the agent flow so the assistant produces a talking face along with speech.

The relevant integration is documented in the Pipecat guide for the Protoface service: https://docs.pipecat.ai/api-reference/server/services/video/protoface. If you prefer to start from a known implementation, the plugin repository is here: https://github.com/protoface-ai/protoface-plugin-pipecat.

At a high level, your Pipecat graph can look like this:

# Pseudocode: exact class names and fields depend on the current docs

# Pseudocode: exact class names and fields depend on the current docs

# Pseudocode: exact class names and fields depend on the current docs

The useful pattern is to let the language detector update shared session state before the TTS step runs. Then the TTS router can pick the appropriate voice for the next response, while the avatar service receives only the resulting audio stream.

If you need a quick way to verify the integration from a frontend or an existing LiveKit voice agent, Protoface also has a LiveKit plugin package on PyPI, but for a multilingual pipeline the main architectural point is the same: keep the language decision upstream of speech synthesis, and keep the media path continuous.

Operational details: session control, fallback, and observability

Multilingual systems are usually messy in the first few days because language detection is only one source of truth. You also need operational guardrails:

• Fallback language: choose a default when confidence is low or the user speaks an unsupported language.

• Explicit overrides: if the user selects a language in the UI, prefer that over automatic detection until they change it.

• Per-session logging: record detected language, confidence, selected voice, and switch events for debugging.

• Latency budgets: watch the time from end-of-speech to first audio byte; voice switching should not meaningfully regress it.

A useful debugging trick is to log every state transition in the language router:

{ "event": "language_candidate", "lang": "es", "confidence": 0.91 }
{ "event": "language_switch_committed", "from": "en", "to": "es" }

{ "event": "language_candidate", "lang": "es", "confidence": 0.91 }
{ "event": "language_switch_committed", "from": "en", "to": "es" }

{ "event": "language_candidate", "lang": "es", "confidence": 0.91 }
{ "event": "language_switch_committed", "from": "en", "to": "es" }

If your logs show frequent flips between languages on the same turn, raise the confidence threshold or require more consecutive hits before committing a switch. If the assistant stays in the wrong language for too long, lower the threshold slightly or feed the detector a larger stable text window.

Where Protoface fits

When you want the avatar side handled as a reusable service rather than a custom media project, Protoface gives you the session and avatar layer while you keep control of the language and TTS logic in your agent. For Pipecat users, the integration point is the Protoface video service plugin, so the avatar stays synchronized with whatever audio your pipeline emits. The docs at https://docs.protoface.com are the right place to check the exact session and avatar configuration fields.

If you want to stand up a session programmatically, the Python SDK and REST API are the cleanest surfaces. Example shape:

import requests

session = resp.json()

import requests

session = resp.json()

import requests

session = resp.json()

Exact request fields depend on the current API schema, so treat this as a pattern rather than a copy-paste contract.

Conclusion

A multilingual realtime avatar is mostly a pipeline design problem. Detect language early, debounce aggressively enough to avoid false switches, route TTS through a stateful voice map, and keep the avatar transport uninterrupted so lip-sync stays stable.

If you are building this with Pipecat, start by inserting language detection before TTS selection and keep the avatar service at the end of the media path. Then validate the behavior with a few realistic code-switching conversations, not just monolingual tests.

For implementation details and current integration specifics, check the documentation at docs.protoface.com and the Pipecat integration guide linked above. Once the routing logic is stable, the rest is mostly tuning: thresholds, voice mappings, and latency budgets.