How to Build a Realtime AI Travel Concierge Avatar in Flutter with WebRTC
Build a realtime AI travel concierge avatar in Flutter with WebRTC, server-side sessions, and synced lip-motion.
Introduction
If you want a travel concierge that feels present instead of purely transactional, you need more than text streaming. You need low-latency speech, synchronized lip motion, and a UI that can render a talking face without blocking the rest of the app. In Flutter, that usually means combining a voice agent, WebRTC media plumbing, and a video surface that can keep up with realtime audio.
By the end of this post, you should have a clear implementation model for building a travel concierge avatar in Flutter: how to wire the call flow, how to think about latency and session state, what belongs in the app versus your backend, and where an avatar service fits in the architecture. I’ll use Protoface as the concrete avatar layer because it’s designed for exactly this kind of developer integration.
Start with the media architecture, not the UI
The biggest mistake is to start by building an avatar widget first. For a realtime concierge, the critical path is:
User speaks into the app.
Audio is sent to a voice agent over a realtime transport, typically WebRTC.
The agent generates a response and a synchronized video face.
Flutter renders the remote media stream as an avatar surface.
WebRTC matters here because it is built for low-latency, bi-directional media with NAT traversal, jitter buffering, and adaptive congestion control. For conversational UX, a few hundred milliseconds of extra latency is noticeable. Once you add lip sync, the system becomes sensitive not just to total latency, but also to media alignment: the face should move in step with the speech that the user hears.
For a travel concierge, this translates into a few practical requirements:
Fast turn-taking: users ask about flights, hotels, transfer times, and itinerary changes; they expect short pauses.
Streaming responses: the agent should begin speaking before the entire answer is computed.
Stable session state: trip context, preferences, and intent should survive multiple turns.
Avatar synchronization: the visual layer must follow the audio stream, not a separate clock.
In Flutter, treat the avatar as a remote media participant, not as a local animation. That distinction influences your widget tree, lifecycle handling, and error recovery.
Define the Flutter app boundary clearly
Keep the Flutter client thin. The app should handle:
mic permissions and audio capture,
joining/leaving a realtime session,
rendering the remote avatar/video track,
showing text state such as itinerary summaries or suggested actions.
Your backend should own anything that requires secrets or policy enforcement:
API key usage,
session creation,
avatar selection and instructions,
per-user access control,
trip data access, bookings, and integrations.
This split matters because the browser or mobile client should never hold privileged credentials. Even if your app is Flutter and runs on mobile or web, assume the client can be inspected. Generate short-lived session credentials server-side and hand only the minimum necessary token to the app.
Model the concierge as a conversation with state
A travel concierge is not just a generic Q&A bot. It needs explicit state around the trip domain. Before you wire up media, define the conversational contract. For example:
what itinerary fields are authoritative,
how to handle ambiguous airport codes or city names,
when to ask clarifying questions instead of guessing,
what tools the agent can invoke, such as flight lookup or hotel search,
what the avatar should say when data is stale or unavailable.
In practice, this means the agent prompt and tool layer should already know how to answer questions like “Can I make my connection in Frankfurt?” without hallucinating schedule details. The avatar is the presentation layer for that agent, not a substitute for good agent design.
A clean pattern is:
That “optionally surface a UI card” point is important. The avatar handles conversation; Flutter can still render structured itinerary data alongside it. Good concierge UX usually combines both.
Build the WebRTC/session flow in Flutter
At a high level, the client flow looks like this:
User opens the concierge screen.
Your backend creates or resumes a realtime session.
Flutter receives a session token or connection parameters.
The app joins the session, subscribes to the remote avatar track, and starts microphone capture if needed.
The agent streams audio and video back into the app.
You do not want to create the session from the client with a long-lived secret. Instead, expose a small backend endpoint that returns only the session material needed for the current user.
The exact request shape depends on the API surface you use, but the security model is the same: server-side creation, client-side consumption. The response should give your app what it needs to connect, not the secret used to create it.
In Flutter, once the session is established, render the remote video track in a dedicated region. Keep the widget stable across rebuilds so the media element is not torn down every time your app state changes. If you are using a WebRTC package, also make sure you handle:
audio focus on mobile,
speakerphone toggling if you need it,
background/foreground transitions,
reconnect logic when the network blips,
mute/unmute state that matches the visible UI.
For a concierge, I also recommend that you display a transcript or message feed alongside the avatar. If the user misses a spoken detail, they should not have to replay the entire answer.
What latency and lip sync actually depend on
Developers often focus on “video latency” as a single number, but in avatar systems there are multiple buffers and clocks:
capture latency from microphone and camera (if any),
transport latency across the media session,
agent processing latency for ASR, reasoning, and TTS,
render latency in the Flutter client.
Lip sync is only as good as the weakest alignment point. If the avatar is generating video from audio, the face animation needs to track the same audio stream the user hears. If your app adds extra buffering on the playback path, the visuals can drift.
There are a few practical guardrails:
Keep response generation incremental, not batch-oriented.
Avoid unnecessary transcoding between the agent and the client.
Do not rebuild the video widget unless the session actually changes.
Prefer a single authoritative stream for the avatar’s speech.
Test on real mobile networks, not just local Wi‑Fi.
For a travel concierge specifically, optimize for conversational responsiveness over cinematic quality. A user asking “What gate is my flight at?” cares much more about fast, accurate output than about a perfect 4K face render.
Where Protoface fits cleanly
This is where Protoface is useful: it gives you the avatar/session layer without forcing you to build the lip-synced video stack yourself. If you are already running a voice agent, the LiveKit integration is the most direct path. The livekit-plugins-protoface plugin lets a LiveKit agent gain a synchronized talking face with minimal glue, which is exactly what you want if your Flutter app is consuming the agent as a realtime media participant.
For teams that prefer to manage sessions directly, the REST API and Python SDK are the other useful surfaces. Create sessions server-side, keep API keys off the client, and hand the app short-lived connection data. If you want to inspect the API and session model before wiring it into Flutter, start with the documentation and the Python SDK examples.
That snippet is illustrative rather than exhaustive; the exact fields depend on the SDK version and the session shape you choose. The key idea is that your backend owns this step, and Flutter only receives the result.
Practical Flutter UI details that are easy to miss
A few implementation details tend to cause avoidable bugs:
Session lifecycle: create a single source of truth for join/leave state so repeated rebuilds do not duplicate connections.
Error presentation: show a human-readable reconnect state when the media path drops.
Avatar switching: if you support multiple concierge personalities, treat avatar changes as a new session or a clean renegotiation, not a visual toggle.
Accessibility: expose the transcript, since speech-only UX is brittle in noisy environments.
Fallback mode: if video fails, keep the voice agent usable.
For a travel product, I would also separate the “assistant screen” from the “trip details screen.” The avatar should support the workflow, not monopolize it. Users often want both a conversational explanation and a structured answer they can scan quickly.
Conclusion
To build a realtime AI travel concierge avatar in Flutter, treat the problem as a media architecture first and a UI problem second. Use WebRTC for low-latency transport, keep session creation server-side, render the avatar as a remote media stream, and design the agent with explicit travel-domain state. That gives you a foundation that can answer questions quickly, stay synchronized, and recover cleanly when the network misbehaves.
If you want to implement this without building the avatar pipeline yourself, start with the docs, the OpenAI Realtime quickstart if you are exploring a voice-agent stack, and the Pipecat integration if that is already part of your architecture. Then wire the session into Flutter as a normal realtime media participant and keep the client thin.