Introduction

If you are building a travel concierge that can answer in multiple languages, the hard part is not “getting an LLM to talk.” It is moving audio and text through a low-latency pipeline that behaves well under real network conditions, preserves turn-taking, and keeps the avatar’s mouth synchronized with the final speech audio.

This post walks through a practical architecture for a multilingual concierge avatar in Kotlin: microphone audio goes to speech-to-text, the agent reasons over the user’s request, text-to-speech produces the reply audio, and a websocket connection carries realtime updates between your backend and the client. By the end, you should be able to wire together the streaming pieces, understand where latency accumulates, and know where an avatar layer fits without turning your app into a pile of race conditions.

The pipeline: STT, agent logic, TTS, and a streaming transport

For a voice-first travel assistant, the minimal realtime loop looks like this:

1. The client streams microphone audio.

2. STT emits partial and final transcripts.

3. Your agent decides whether the user is asking for flights, hotels, itinerary changes, translation, or escalation.

4. TTS generates speech in the user’s preferred language.

5. The client plays audio while the avatar renders a lip-synced face from the same turn.

The main technical constraint is that each stage is asynchronous. You want partial STT results early enough to detect turn boundaries and intent, but you do not want to speak too early and then interrupt yourself when the transcript changes. In practice, treat partials as speculative and finals as commit points. That lets you support “barge-in” and multi-language code-switching without making the assistant sound unstable.

WebSocket is a reasonable transport when you own both ends of the realtime session and want a simple bidirectional channel for events, state, and audio metadata. For browser clients, you will usually still use WebRTC for the media path if you need live audio/video transport with jitter buffering and congestion control. The websocket then carries control-plane messages: session start, language selection, transcript events, agent state, and TTS metadata. Keeping those layers separate makes debugging much easier.

Designing the multilingual turn loop

Multilingual assistants fail in predictable ways: language detection happens too late, transcription is normalized incorrectly, or the assistant answers in the wrong language because the prompt context was not updated. The fix is to make language an explicit part of session state, not an inference afterthought.

Language selection and fallback behavior

Start with a per-session language policy:

• Known locale: the user chooses the language up front, or you infer it from account settings.

• Auto-detect: the first few seconds of speech determine the active language.

• Mixed mode: support code-switching, but keep the response language stable unless the user explicitly changes it.

For travel use cases, a strong default is to respond in the user’s current language and preserve named entities verbatim. City names, hotel brands, airline codes, and reservation numbers should not be translated. “Cancel my booking at NH Collection Berlin” should stay semantically intact even if the surrounding response is in Spanish or Japanese.

That means your prompt or agent state should include something like:

// Pseudocode: keep language as part of session state
)

// Pseudocode: keep language as part of session state
)

// Pseudocode: keep language as part of session state
)

Then, when a new transcript arrives, route it through an intent layer that is aware of the current locale. If the user starts in French and then switches to English for a named destination, do not forcibly normalize everything into one language before the LLM sees it. You want the model to preserve the user’s intent and the original surface forms where necessary.

Stream transcripts as events, not just strings

Do not wait for a final transcript if you can avoid it. Emit events for partial hypotheses, endpointed final text, and confidence changes. A small websocket message schema is enough:

{
}

{
}

{
}

When a final transcript arrives, you can trigger intent classification and generate a response. For travel, the highest-value intents are usually fixed and domain-specific: flight search, hotel search, itinerary lookup, currency conversion, translation, and support handoff. Keep the first pass deterministic where possible. For example, a lightweight router can decide whether to query a booking API or ask a follow-up question before handing the final turn to the LLM.

That approach reduces wasted tokens and, more importantly, reduces latency. If the user asks “Find me a late-night train from Paris to Lyon tomorrow,” you do not need a long generative response before you know that a timetable lookup is required.

TTS and avatar sync: one turn, one audio timeline

The avatar should be driven by the same turn output that produces speech audio. If you decouple them too aggressively, the face will start talking before the audio buffer, or the lips will keep moving after the user hears silence. The simplest rule is: one agent turn creates one authoritative audio stream, and the avatar subscribes to that stream’s timing.

For a multilingual system, TTS adds two more constraints:

• Voice-language compatibility: use a voice that sounds natural in the response language.

• Prosody stability: preserve punctuation and sentence boundaries so pauses land correctly.

In practice, feed the final response text into TTS only after your agent has chosen the response language and confirmed the content. If you support streaming TTS, you can start playback before the full response is complete, but you should still chunk by sentence or clause so the avatar does not stutter through unstable text.

For voice agents that need to interrupt themselves, implement cancelation at the turn level. If the user barges in while TTS is speaking, stop the current audio stream, cancel the pending response generation, and reset the avatar state. Partial overlap is better than dead air, but only if your media graph can actually stop cleanly.

Kotlin implementation shape

In Kotlin, the cleanest implementation is usually a coroutine-based session manager with a websocket client and a small event reducer. The websocket carries state updates, while audio I/O can be handled separately depending on your client stack.

data class TranscriptEvent(

}

data class TranscriptEvent(

}

data class TranscriptEvent(

}

The important part is not the exact API shape; it is the separation of responsibilities. Keep audio capture, STT, agent logic, and TTS as distinct stages. If one fails, you should be able to restart that stage without collapsing the entire session.

Also pay attention to timing on the client:

• Buffer microphone audio in small frames.

• Debounce UI updates from partial transcripts.

• Do not render an “idle” avatar state until both STT and TTS are actually finished.

If you are debugging lag, instrument every hop. Measure from end of speech to final transcript, from final transcript to first token of the response, and from first TTS audio byte to playback start. That will tell you whether the bottleneck is recognition, model latency, synthesis, or network jitter.

Where Protoface fits

This is where Protoface is useful: it gives you the avatar layer without making you build the video-face synchronization yourself. In a LiveKit-based voice agent, the LiveKit plugin can drop a realtime avatar into the agent so the video face stays aligned with the speech turn. If your architecture is Python-heavy, the documentation and SDK are the right place to look for session and avatar lifecycle details.

The practical benefit is that you can keep the language, STT, and TTS logic in your own service while delegating avatar rendering and lip sync to a dedicated realtime surface. That reduces glue code and avoids inventing a custom media synchronization protocol just to show a speaking face.

Operational gotchas

A few things usually bite teams building this kind of system:

• Locale drift: the assistant starts in one language and slowly answers in another because prompts accumulate mixed-language history. Reset response-language state every turn.

• Entity mutation: translation pipelines sometimes “helpfully” translate airport codes or hotel names. Preserve named entities explicitly.

• Transcript churn: if you react to every partial transcript, the agent will thrash. Use debouncing and finalization thresholds.

• Audio/video skew: if TTS starts before the avatar has a committed turn, the face will look off. Bind both to the same turn ID.

• Backpressure: slow clients need bounded queues. Drop stale partials, not final turns.

One useful rule: never let more than one “current response” exist per session. Everything else should either be pending, canceled, or already committed. That makes barge-in, retries, and failure recovery tractable.

Conclusion

A multilingual travel concierge avatar is mostly a systems problem: stream audio reliably, treat transcript updates as events, make language explicit in session state, and keep one authoritative turn timeline for both speech and avatar motion. If you get those boundaries right, the rest becomes incremental tuning rather than architecture churn.

For implementation details, API shape, and examples of the supported integration surfaces, start with the docs and the relevant quickstarts. The main thing is to build around turn boundaries and state transitions first; the avatar layer should follow the conversation, not drive it.