Introduction

If you are adding a realtime avatar to a Vapi-based voice agent, multi-language support is usually the first “works in English, breaks in production” problem you hit. The audio pipeline may be fine, but once you introduce multiple languages, you have to keep speech synthesis, transcription, avatar lip sync, and conversation state aligned. That means choosing the user’s language early, preserving it across turn-taking, and making sure every component in the chain agrees on what language is being spoken.

This post walks through a practical way to do that in Python. By the end, you should be able to:

• route conversations by language at session start,

• pass language preferences through your agent logic cleanly,

• avoid common lip-sync and latency pitfalls in realtime avatars, and

• wire the avatar layer into a voice agent without exposing secrets in the browser.

How multilingual realtime voice agents actually fail

The hard part is not translating text. The hard part is keeping a realtime system consistent across four separate concerns:

• Language detection: from user input, locale, profile data, or explicit selection.

• Agent prompting: the model must answer in the right language and style.

• Speech pipeline: STT and TTS need the correct locale or model variant.

• Avatar timing: the face should animate from the same audio that the user hears.

When people say “multi-language support,” they often mean only the LLM response text. In a realtime avatar app, that is not enough. If your TTS engine emits one language while the transcript says another, the avatar mouth shapes will be wrong, and users notice immediately.

A good architecture treats language as session state, not as a string passed into one prompt.

Pick a language strategy before you write code

There are three common ways to choose language in a voice agent:

1. Explicit selection — the user picks a language in UI before the call starts. This is the most reliable approach.

2. Profile-based default — use a stored preference from the account or tenant.

3. Detected language — infer language from the first utterance, then lock it for the session.

In practice, you usually combine them: explicit selection wins, profile default is the fallback, and detection is a recovery path if the user starts speaking a different language than expected.

For realtime systems, I recommend locking the language once the session begins. Mid-session language switching is possible, but it increases the chance of STT errors, prompt drift, and awkward partial responses. If you do support switching, make it a deliberate state transition rather than an implicit guess.

Propagate language as session metadata

The cleanest implementation is to store language in your own session object and pass it through every downstream component. Your orchestration layer should be the source of truth. For example:

from dataclasses import dataclass

voice_id: str

from dataclasses import dataclass

voice_id: str

from dataclasses import dataclass

voice_id: str

Then use that language value when building the system prompt, configuring TTS, and selecting any language-specific STT options. If you are using a Vapi workflow, this usually lives in the application layer that receives the call/webhook event and creates the realtime agent session.

A practical rule: keep language in one place, and avoid scattering hard-coded language checks across tool handlers, prompt templates, and UI callbacks. That makes it much easier to support new locales later.

Python implementation pattern

A simple Python orchestration layer can set the language once, then hand it to the agent and avatar session setup. The exact SDK fields depend on your stack, so treat this as a structural pattern rather than copy-paste code.

def build_system_prompt(language: str) -> str:

}

def build_system_prompt(language: str) -> str:

}

def build_system_prompt(language: str) -> str:

}

Two things matter here:

• The prompt is language-specific. That keeps the model from answering in the wrong language even when the user’s last utterance was ambiguous.

• The audio stack must match the prompt. If you choose a Spanish prompt, the TTS voice and STT locale should also be Spanish-aware.

If your provider supports per-language voices, map language to a voice explicitly instead of letting the system auto-pick. Automatic voice selection is convenient, but it can produce awkward results when you need a specific accent or formality level.

Handle language detection carefully

Automatic language detection is useful, but it is not free. The first few hundred milliseconds of speech often contain hesitation, names, code-switching, or background noise. That can lead to false positives. For that reason:

• prefer explicit user choice when you have it,

• use detection only to fill in missing data, and

• store the detected language once the confidence is high enough.

A good compromise is to detect language from the first user turn, then confirm it against a small allowlist. If detection returns something you do not support yet, fall back to English rather than trying to improvise.

Also remember that “multi-language” and “code-switching” are different problems. If a user mixes English and Spanish in the same sentence, your system should still have a primary session language. You can support occasional foreign words, but do not assume the agent can fluidly operate in two locales at once without additional training or prompting.

Keep avatar lip sync tied to the generated audio

For a talking avatar, the most important invariant is simple: the avatar should animate from the same audio stream that the user hears. Do not generate text, then separately generate speech, then try to animate the face from text timing. That path is fragile, especially across languages where syllable timing differs.

Instead, the avatar should sit downstream of your TTS output or the realtime media pipeline. If the agent speaks Spanish, the avatar receives the Spanish audio. If the agent switches to French in the next turn, the avatar receives the French audio. The lip sync layer should not care which language it is; it should care that it has synchronized audio frames.

That matters in multilingual apps because speech rate, phoneme distribution, and pause structure vary by language. If you build around audio timing rather than text timing, your avatar stays stable when you add new locales.

Where Protoface fits

This is exactly the kind of problem Protoface is meant to sit behind: your app decides the language, and the avatar layer renders the synchronized talking face for whatever audio your agent produces. In a LiveKit voice-agent setup, the Protoface plugin for Pipecat drops the avatar into the media pipeline so the agent gets a realtime video face without you building a separate streaming stack.

If you are managing sessions directly from Python, the SDK is the other useful surface: create the avatar/session on the backend, pass in your per-session metadata, and keep the API key out of the browser. The public docs at docs.protoface.com cover the exact request and SDK fields. A minimal REST-style creation flow looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","language":"es","metadata":{"tenant":"acme"}}'

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","language":"es","metadata":{"tenant":"acme"}}'

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","language":"es","metadata":{"tenant":"acme"}}'

The exact payload will depend on your session model, but the operational pattern stays the same: create the session server-side, attach the chosen language, and hand the resulting session details to the voice agent and avatar pipeline.

Practical guardrails for production

Once the basics work, the bugs tend to be in the edges. A few guardrails save a lot of time:

• Validate supported languages up front. Reject unknown locales early instead of allowing silent fallback.

• Make voice selection deterministic. One language should map to one approved voice profile per product tier or tenant.

• Log the chosen language with each session. You will want this when debugging mismatched transcripts or customer complaints.

• Test turn boundaries. The switch from user speech to assistant speech is where most realtime bugs show up.

• Measure end-to-end latency per language. Some STT/TTS combinations are noticeably slower outside English.

If you support browser-based experiences, keep secrets server-side and use an embedded flow or backend-issued session token rather than putting provider keys into frontend code.

Conclusion

Multi-language support in a realtime avatar app is mostly about discipline: choose the language once, propagate it through the session, keep STT/TTS aligned with the prompt, and drive the avatar from synchronized audio rather than text. Do that, and the experience scales from one locale to many without turning your pipeline into a pile of special cases.

If you want implementation details, SDK references, and current request shapes, start with the docs at docs.protoface.com. If you are building on Pipecat or LiveKit, the relevant examples in the GitHub repos are the fastest way to adapt this pattern to your stack.