Introduction

If you are adding a realtime hotel concierge avatar to a Kotlin-based voice agent, latency is the thing users notice first. Not “frames per second” in the abstract, but the gap between the end of a guest’s sentence and the avatar’s first visible reaction: a nod, a glance, lip motion that matches the returned speech, or even just a tiny acknowledgment that keeps the interaction feeling alive.

This post is about reducing that gap in a practical way. By the end, you should be able to identify where latency is coming from in a realtime avatar pipeline, choose the right integration points in Kotlin, and make a few concrete architectural changes that usually matter more than micro-optimizing a single SDK call.

What actually creates latency in a concierge avatar

A realtime hotel concierge is usually a chain of systems, not a single model:

mic input → speech recognition → agent reasoning → response generation → TTS → avatar/video synthesis → transport → browser playback

Each step adds delay. In practice, the biggest wins come from reducing the number of serial dependencies and from overlapping work when possible.

There are four latency buckets worth measuring separately:

• Input latency: how quickly the system detects that the guest finished speaking.

• Inference latency: how long the language model or dialog logic takes to produce the next action or text.

• Audio/video pipeline latency: how long TTS and lip-sync rendering take before any output can start.

• Transport and render latency: network jitter, buffering, and browser-side decode/render time.

For a concierge, the most important metric is usually time to first perceptible response. A partial response is often better than waiting for a perfect one. A brief “Sure — let me check availability” delivered quickly can hide a lot of backend work that continues in parallel.

Design for early audio and early motion

The first optimization is architectural: don’t wait for the entire answer before you begin output. If your agent stack supports streaming text or incremental intents, use it. The avatar should begin lip-syncing and speaking as soon as the first audio is ready, even if the rest of the sentence is still being generated.

That implies a few useful constraints:

• Prefer streaming TTS over buffering a full sentence before synthesis.

• Keep prompts and tool calls tight; long tool chains are latency multipliers.

• Emit short acknowledgments first when the user experience allows it.

• Avoid re-encoding or re-transcoding media more than once in the hot path.

In hotel use cases, this often means the avatar should quickly acknowledge the request, then continue with the details once the backend confirms room status, spa availability, late checkout policy, or restaurant times.

Also pay attention to turn detection. If your VAD waits too long to commit an utterance, you add dead air before the agent can even start. If it commits too aggressively, it interrupts guests. Tuning that threshold is usually worth more than shaving a few milliseconds off a database query.

Keep the hot path simple in Kotlin

In Kotlin services, latency tends to creep in through convenience abstractions: extra coroutine hops, blocking client calls hidden behind suspending wrappers, and unbounded fan-out to unrelated services. For a realtime avatar, your hot path should be explicit.

A useful pattern is to separate the session lifecycle from the response path:

1. Create or attach to the avatar session.

2. Establish the realtime media path early.

3. Only then start conversational work that depends on the session being live.

That means pre-warming anything you can: HTTP clients, auth tokens, model clients, and any backend lookups you know you will need. It also means that if the agent is idle, you should keep the session warm rather than tearing down and rebuilding on every message.

Example: if you are using a backend service to create a session before the user joins, keep the request small and idempotent. The exact payload shape depends on your avatar/session configuration, but the flow is straightforward:

val client = HttpClient()
}

val client = HttpClient()
}

val client = HttpClient()
}

That example is intentionally minimal. The key idea is not the field names; it is that session creation should happen before the user is waiting on a response path, and that the response path should use already-established connections whenever possible.

Reduce serialization, not just compute

A common mistake is focusing on “how fast is the model?” while ignoring all the time spent serializing requests, waiting on network round-trips, and bouncing between services. If the concierge asks your booking system a question synchronously, then asks a second service, and only then starts speaking, you have serialized the entire experience.

Instead, try to structure the conversation loop around parallelism:

• Start the avatar session while the user is still speaking.

• Kick off low-risk backend lookups as soon as the intent is clear.

• Generate a short spoken acknowledgment immediately.

• Continue gathering details while the acknowledgment is being rendered.

This is especially important for hotel workflows where the response often depends on a mix of static policy and dynamic inventory. Static policy can be summarized instantly; dynamic inventory may take a network trip. If you wait for both, the user gets silence. If you separate them, the avatar can speak while the slower query completes.

In Kotlin, coroutines make this easy to express, but easy to misuse. Avoid launching work into unrelated scopes that make cancellation hard to reason about. Realtime sessions should be cancellable when the guest disconnects or starts talking over the agent again. If a response is no longer relevant, stop spending cycles on it.

Watch the browser and media path too

Even if your backend is fast, the avatar can still feel sluggish if the browser is buffering, decoding, or waiting on a fragile media path. For iframe-based embeds, transport and rendering are usually handled for you, which simplifies the client side. For WebRTC-style voice and video flows, the big concerns are packet loss, jitter, and the time it takes for the first audio frames and video frames to reach the user.

Practical things that help:

• Minimize startup work on the page before the avatar connects.

• Avoid heavy main-thread JavaScript during the first seconds of the session.

• Keep the avatar visible early so users see motion as soon as media begins.

• Measure separately: agent response time, audio start time, and first frame time.

If you are embedding the experience in a web property, a controlled iframe can reduce client-side integration bugs and keep secrets out of the browser. That matters more than it sounds: many “latency problems” are really integration problems that only show up when auth, autoplay policy, and media startup all collide.

Where Protoface fits

This is the kind of pipeline Protoface is meant to simplify: you connect your voice agent to a synchronized talking face rather than building the avatar/video layer yourself. For Kotlin-backed systems, the useful part is that the avatar/session lifecycle is exposed through a developer API, so you can create or manage sessions from your service and keep the realtime path under your control. The details live in the docs, but the design goal is simple: make the avatar a thin, predictable layer in the middle of your existing agent.

If your stack is already using LiveKit, the plugin route is often the lowest-friction way to add the visual layer. If your system is orchestrated in Kotlin around an HTTP backend, the REST API is the natural fit for pre-creating sessions and wiring them into your concierge flow. In both cases, the important thing is that the avatar should not become the slowest or most opaque part of the experience.

Concrete tactics that usually move the needle

When I am tuning a realtime concierge, I usually work through the following checklist:

1. Measure end-to-end timing from user stop-speaking to first audio or visible motion.

2. Shorten the first response so the user hears acknowledgment quickly.

3. Stream output instead of waiting for a full answer.

4. Pre-warm sessions and clients before peak interaction begins.

5. Cut synchronous tool calls from the main conversational path.

6. Make cancellation cheap so obsolete responses do not keep running.

7. Keep the render path simple and avoid extra media transforms.

One useful trick is to instrument the avatar session with a small set of timestamps: user utterance end, agent first token, TTS first byte, avatar first frame, browser first render. That gives you a much clearer picture than a single “response latency” number. In many systems, the model is not the bottleneck; the turn boundary or media startup is.

Conclusion

Reducing latency for a realtime hotel concierge avatar is mostly about removing unnecessary serialization and getting something perceptible on screen and in the speakers as early as possible. In Kotlin, that means keeping the hot path explicit, pre-warming what you can, streaming responses, and treating session management as part of the user experience rather than a background detail.

If you want to build this with less infrastructure work, start with the documentation and then wire the avatar layer into your existing agent flow. For practical examples and quickstarts, the GitHub repositories linked from the docs are the fastest way to see the integration patterns end to end.