Introduction

Language detection and voice switching in realtime agents usually fail for the same reason: the system is trying to make an irreversible decision too early, or it is making a “language” decision in one layer while the user experience is controlled in another. In a Vapi-style voice agent, that can mean the ASR pipeline hears one language, the LLM responds in another, and the TTS layer keeps speaking with the wrong voice or pronunciation profile. When you add a realtime avatar on top, you also have to keep lip-sync and audio timing stable.

This post is about the practical debugging path: how to identify where detection is breaking, how to switch voices safely when a language changes, and how to avoid common realtime mistakes like hard-coding the wrong locale, switching too often, or letting partial transcripts trigger bad state changes. By the end, you should be able to reason about the whole path from microphone input to transcript, language classification, response generation, TTS voice selection, and synced avatar playback.

Start by separating the three decisions

Most “language detection” bugs are actually three independent problems:

1. Input language detection: what language does the user appear to be speaking?

2. Response language policy: what language should the agent answer in?

3. Voice selection: which TTS voice should speak that response?

If you collapse those into one variable like language = "es", you lose important distinctions. For example, a user can speak English but ask the agent to answer in Spanish. Or a user can code-switch mid-sentence, which should not immediately flip the agent’s voice. A good implementation keeps these decisions separate and only promotes a language change when confidence is high and stable.

Common failure modes in realtime agents

The most common bug is reacting to partial ASR hypotheses. Partial transcripts are useful for latency, but they are not a stable signal for language switching. If you switch voice on the first “bonjour” you hear, you may bounce between voices when the transcript later resolves as “brand new job” or “bonjour, thanks.”

Other failure modes I see a lot:

• Switching on a single token: one foreign word is not a language change.

• No hysteresis: the agent flips back and forth when transcripts alternate between languages.

• Locale mismatch: ASR is configured for one language, but the TTS voice is chosen from another locale.

• Prompt leakage: the LLM is told “always answer in the user’s language” but the downstream voice selection is still pinned to a default.

• State shared across sessions: one user’s language preference leaks into another session because it is stored globally instead of per-call/per-connection.

For realtime systems, the right fix is almost always state management, not model magic.

Use confidence, stability, and a cooldown

The simplest robust approach is to require a language signal to satisfy three conditions before switching voices:

1. Confidence: the detector or ASR metadata must be above a threshold.

2. Stability: the signal must persist for multiple turns or a short time window.

3. Cooldown: after a switch, ignore new candidates for a few seconds unless the evidence is overwhelming.

This is standard hysteresis. It prevents oscillation when the user code-switches or when the recognizer briefly misclassifies accented speech. In practice, a short “observe mode” is often enough: record the likely language from the first complete utterance, confirm it on the second, and only then switch the voice.

A useful implementation pattern is to keep a per-session state object:

state = {
}

state = {
}

state = {
}

Then update it only on stable signals, not on every token. If you use the LLM to decide response language, treat that as another input, but still gate the final switch through the same state machine.

Don’t switch voices faster than the audio pipeline can breathe

Voice switching is not free. Even if your TTS provider can swap voices instantly, the overall audio path still has buffering, chunking, and network jitter. If you are also rendering a realtime avatar, the mouth movements need to stay aligned with the audio stream. Abrupt mid-utterance voice changes can look and sound broken.

The safest rule is: switch at utterance boundaries. That means you let the current response finish, update the voice for the next response, and only then start generating the next audio stream. If you truly need mid-turn language adaptation, you need a design that can segment the response and reinitialize downstream audio cleanly. That is possible, but it is much easier to get wrong.

Practical tips:

• Keep the current voice fixed for the duration of one agent response.

• Update voice selection only before the next synthesis request.

• Cache the mapping from language to voice so you are not doing lookup logic on the hot path.

• Log every switch with session id, detected language, confidence, and the reason for the change.

Debug from the bottom up: transcript, detector, policy, then voice

When this breaks, trace the pipeline in order. Do not start with the avatar. Start with the transcript and work outward.

1. Inspect ASR output: Are you getting the right language at the transcript level? If not, fix the recognizer configuration first.

2. Inspect language detection: Are you using partial or final transcripts? Are you thresholding confidence properly?

3. Inspect policy: Is the agent supposed to answer in the detected language, the user’s preferred language, or the conversation’s initial language?

4. Inspect voice mapping: Does the chosen voice actually support the language? Does it match the locale and pronunciation rules you need?

5. Inspect audio handoff: Are audio chunks being interrupted, queued twice, or sent before the new voice is active?

6. Inspect avatar sync: If the lip-sync looks off, you may have timing drift or a race between audio generation and video frame updates.

This order matters because developers often fix the wrong layer. For example, they may change the avatar rendering code when the real issue is that the system is synthesizing Spanish with an English voice preset.

Implement a simple voice policy

In most production agents, I prefer a deterministic voice policy over “let the model decide everything.” A small lookup table is easier to reason about and easier to test.

VOICE_BY_LANGUAGE = {

return VOICE_BY_LANGUAGE.get(language, fallback)

VOICE_BY_LANGUAGE = {

return VOICE_BY_LANGUAGE.get(language, fallback)

VOICE_BY_LANGUAGE = {

return VOICE_BY_LANGUAGE.get(language, fallback)

The actual voice identifiers depend on your TTS provider, so treat the mapping as illustrative. The important part is that voice selection is a pure function of the chosen policy state, not a side effect spread across prompts and event handlers.

If you need per-user preferences, keep them separate from detection. For example, a user might prefer English output even while speaking Spanish. In that case, the detector informs understanding, but the policy still returns the user’s preferred response language and voice.

How Protoface fits into the debugging picture

If you are adding a realtime face to an existing voice agent, the most important thing is to preserve the same audio boundary discipline for the avatar. The LiveKit integration on PyPI, pipecat-protoface, is the cleanest place to do this if your agent is already built around a realtime pipeline. The underlying idea is straightforward: keep the voice and transcript decisions in your agent, and let the avatar render whatever audio the agent has already committed to speak.

For example, in a LiveKit-based flow you would typically keep a stable voice for the current turn, then feed the synthesized speech into the avatar stage so the mouth motion stays aligned. The exact API shape depends on your stack, but a typical integration looks like this in spirit:

# Illustrative only; check the docs and repo examples for exact imports and parameters

# Illustrative only; check the docs and repo examples for exact imports and parameters

# Illustrative only; check the docs and repo examples for exact imports and parameters

If you are working from a Pipecat-based architecture instead, the integration guide in the docs is the right place to verify the current wiring and service names: docs guide. The operational lesson is the same either way: language switching should happen upstream of synthesis, not inside the avatar layer.

Testing strategy that actually catches regressions

Language switching bugs are hard to catch manually because they depend on timing, accents, partial transcripts, and user behavior. I recommend a small test matrix that exercises the policy layer directly.

• Monolingual input: English-only, Spanish-only, etc., with expected stable voice choice.

• Code-switching: one utterance with mixed language to verify hysteresis.

• Accent-heavy speech: ensure confidence thresholds are not too aggressive.

• Preference override: user speaks one language but requests another response language.

• Rapid turn-taking: confirm the voice does not switch mid-response.

It is also worth recording session logs with timestamps for: ASR finalization, detected language, chosen response language, chosen voice, and synthesis start. Those five points usually tell you where the bug is.

A minimal API-driven workflow

If you manage sessions from your backend, the REST API is a good fit for creating or updating the session state that your agent reads. The exact fields will depend on the object you are updating, but the shape is the same: authenticate with an API key, send the session metadata, and keep the state server-side.

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

That kind of server-side control is useful when your voice agent needs deterministic behavior across reconnects. The browser or client should not be deciding voice policy on its own.

Conclusion

The main takeaway is that language detection and voice switching are control-flow problems, not just model-selection problems. Keep input language, response policy, and voice choice separate. Switch only on stable signals. Prefer utterance-boundary changes over mid-stream changes. And log the full path so you can see where a mismatch was introduced.

If you are adding a face to an existing agent, keep the avatar layer downstream of the speech decision so lip-sync stays aligned with the audio you actually intend to speak. For implementation details, integration examples, and current API shapes, start with docs.protoface.com and the relevant quickstart or plugin repository. If you already have a Vapi-style voice agent and want to make its language behavior predictable, the same debugging discipline applies regardless of the rendering layer.