Introduction

Code-switching is one of the first things that breaks an otherwise solid realtime voice experience. A user starts in English, slips a product name in Spanish, answers a follow-up in Hindi, or mixes two languages in the same sentence. If your agent pipeline assumes one fixed language per session, you usually get one of three failures: the speech recognizer mis-transcribes the input, the LLM responds in the wrong language, or the avatar speaks the correct words with the wrong prosody and timing.

This matters more for avatars than for text-only agents because the system has to keep four streams aligned at once: audio input, transcription, response generation, and lip-synced video output. By the end of this post, you should be able to design a realtime voice/avator pipeline that handles code-switching without falling apart, choose where language detection should happen, and understand where the avatar layer needs to be informed so the face stays believable.

What code-switching actually changes in a realtime pipeline

In a typical voice agent, the basic loop is simple: capture mic audio, run ASR, feed text to the agent, synthesize response audio, and stream both audio and avatar video back to the client. Code-switching makes each boundary less deterministic.

The key point is that language is not a single session attribute. It is often a property of:

• the user’s current utterance,

• individual spans inside that utterance,

• the assistant’s response policy, and

• the TTS voice or avatar behavior you pick for the reply.

That means you should stop thinking in terms of “this session is English” and start thinking in terms of “language is a signal that can change at turn boundaries, or even within a turn.”

For practical purposes, there are three common code-switching patterns:

1. Turn-level switching: user speaks one full turn in one language, then the next in another.

2. Span-level switching: a single utterance contains short phrases or named entities in another language.

3. Style-switching: the user stays in one language but expects the agent to mirror terminology, accent, or register from a different locale.

Your implementation does not need perfect multilingual understanding to be useful. It needs to detect the likely language quickly, preserve ambiguous spans, and keep the response aligned with the user’s last stable language signal.

Detect language early, but do not over-commit

The first mistake is to wait until after the full transcript is finalized. In realtime agents, you usually want language hints from partial audio or partial transcripts because they influence every downstream step. But you also do not want to lock the session into a language based on one noisy phrase.

A workable approach is:

1. Use streaming ASR with per-chunk or per-utterance language probabilities if available.

2. Treat the first strong signal as a candidate, not a hard decision.

3. Update the language state only when evidence is stable over a short window.

4. Keep named entities, URLs, and product terms in their original form even if the surrounding language changes.

In code, that usually means keeping metadata alongside the transcript rather than baking language into the text itself. For example:

{
}

{
}

{
}

That distinction matters. The agent can answer in Spanish while preserving the English phrase if the phrase is a brand name or borrowed term. If you flatten everything to “Spanish,” you risk mistranscribing or translating away meaning.

Make the agent prompt code-switching explicitly

Most LLMs will handle mixed-language input better if you say what to do. The prompt should define the policy, not just the persona. In practice, the policy is something like: answer in the user’s dominant language for the current turn, preserve proper nouns and technical terms, and mirror the user if they switch languages mid-conversation.

A concise system instruction can be enough:

You are a realtime voice agent.
- Keep responses short enough for speech

You are a realtime voice agent.
- Keep responses short enough for speech

You are a realtime voice agent.
- Keep responses short enough for speech

That policy helps the model avoid making a bad global guess. It also reduces ugly behavior like responding in English to a Spanish question just because one token was unfamiliar.

For mixed-language turns, it is usually better to preserve the user’s phrasing than to translate it. Translating on the fly can distort intent, especially in support or sales contexts where the exact wording of a plan, error message, or address matters.

Keep transcription and synthesis in the same language frame

Realtime voice systems fail when ASR and TTS use different assumptions about the language of the turn. If ASR sees “quiero cancelar my subscription” and normalizes it into pure English or pure Spanish, the LLM response may be technically correct but socially wrong. The same happens when TTS renders a mixed-language reply with the wrong accent or pronunciation for key terms.

A few practical rules help:

• Preserve the raw transcript in addition to any normalized version.

• Pass language hints downstream so TTS can select a voice or pronunciation model appropriate to the current turn.

• Avoid gratuitous translation of entities, code, and UI terms.

• Re-evaluate language at each turn rather than assuming the entire session remains in one language.

If you’re using a voice agent framework, make sure it exposes turn metadata. The language detector should emit an event that the rest of the pipeline can consume, not just a side log.

Also remember that code-switching affects latency. If you try to run a heavyweight classifier on every audio chunk, you can add enough delay to make barge-in and turn-taking feel sluggish. The better pattern is a lightweight streaming detector plus a more accurate confirmation step once the utterance stabilizes.

Avatar synchronization is mostly a timing problem

For an AI avatar, the visual layer does not need to “understand” the languages; it needs to stay synchronized with the audio and present consistent speaking behavior. The problem is that code-switching can cause timing drift if your pipeline changes TTS settings, inserts pauses for translation, or regenerates text after the avatar stream has already started.

To keep the face believable:

• start avatar playback from the same response event that starts audio playback,

• avoid re-writing the response after synthesis has begun,

• do not block video output while you decide which language the next sentence should use,

• keep latency variance low across different language paths.

If your avatar vendor is driven by the audio signal, the safest implementation is to synthesize one final response audio track and stream the avatar from that same track. If the language changes, switch only at clean turn boundaries unless your TTS and orchestrator are designed for intra-utterance language changes.

There is also a practical UX consideration: a face that keeps speaking with the same mouth cadence while the language changes abruptly can look broken even when the words are correct. Short responses, stable timing, and avoiding unnecessary restarts help more than fancy visual tricks.

Where Protoface fits

Protoface is useful here because it lets you add a synchronized avatar layer without rewriting your agent architecture. In a LiveKit-based voice agent, the LiveKit integration can drop a talking video face into the same realtime session that already handles ASR, LLM orchestration, and TTS. That means your language logic stays in the agent, while the avatar simply follows the audio timing.

For developers who want to manage sessions directly, the REST API and Python SDK are the right place to persist per-session metadata such as the current language policy, voice choice, or user locale. The exact request fields are documented in the docs, but the shape is familiar: create an avatar or session, attach configuration, and stream the response into your app or agent.

A minimal curl flow looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

And a Python SDK call typically looks like this:

from protoface import Client

from protoface import Client

from protoface import Client

Keep the metadata on the server side; your agent can update it as the conversation shifts, but the avatar should only consume the final, stable response stream.

Implementation patterns that work in production

If you are building this for real users, the safest pattern is to make language detection a first-class event in your agent state machine. Do not bury it inside a prompt or a one-off ASR callback.

A production-friendly sequence looks like this:

1. Receive audio and begin streaming ASR.

2. Emit a provisional language estimate from the first stable transcript segment.

3. Use that estimate to select prompt behavior and TTS voice defaults.

4. Preserve the raw transcript and any mixed-language spans.

5. Generate a response in the dominant language of the user’s current turn.

6. Start TTS and avatar playback from the same committed response payload.

For observability, log at least three things per turn: detected language, confidence/stability, and whether the assistant mirrored or switched. That data is how you’ll catch subtle failure modes like “always replies in English after a Spanish greeting” or “mispronounces company names only on mixed-language turns.”

If you need to support customer support or sales workflows, also define an escalation path for ambiguity. When confidence is low, it is better to ask a short clarification question than to confidently answer in the wrong language.

Conclusion

Code-switching is not an edge case in realtime voice agents; it is a normal property of multilingual conversation. The practical fix is to treat language as dynamic session state, detect it early but cautiously, preserve mixed-language spans, and keep the avatar synchronized with the final audio rather than with intermediate guesses.

If you are implementing this now, start by wiring language detection into your turn state machine, then make sure your ASR, prompt policy, and TTS all consume the same language signal. If you want a video face on top of that stack, review the documentation and the relevant quickstarts in the GitHub org to see how your agent can stream a synchronized avatar without exposing browser-side credentials or adding unnecessary latency.