Introduction

If you already have a voice agent in LiveKit and you want it to feel less like a phone tree and more like a real conversational experience, the missing piece is usually a synchronized face. A realtime avatar gives the agent a visual presence, keeps lip motion aligned with audio, and makes the interaction easier to follow in product demos, support flows, and sales experiences.

This post shows how to connect LiveKit to a Flutter widget that renders a realtime avatar, with the practical constraints that matter in production: media sync, session lifecycle, latency, and where the avatar state should actually live. By the end, you should understand how to wire the voice agent, the avatar session, and the Flutter UI together without exposing secrets or building a fragile ad hoc video pipeline.

What “realtime avatar” means in this architecture

In a LiveKit-based app, audio is already a real-time media stream. The avatar layer is another synchronized stream, usually driven by the same agent state that generates text and speech. The important point is that the avatar should not be treated as a separate, loosely coupled animation. It needs to track the agent’s speaking turns, timing, and interruptions closely enough that the face and voice feel like one system.

For Flutter, that usually means your widget is responsible for rendering a video surface or an embedded playback surface, while the backend owns the conversational session. The Flutter app should receive only the minimum connection details it needs to join or display the avatar session. The backend should mint those details, enforce permissions, and keep the session state authoritative.

In practice, the flow looks like this:

1. A user opens the Flutter app and starts or joins a conversation.

2. Your backend creates or looks up the relevant voice/avatar session.

3. The backend returns short-lived connection data to the Flutter client.

4. Flutter connects to LiveKit for the voice stream and renders the avatar widget against the session.

5. The agent speaks, the avatar lip-syncs, and the UI updates based on session events.

Keep the session boundary on the server

The first design decision is where to create and manage the realtime session. Don’t push API keys or privileged session creation into the Flutter client. Even if the app is internal, the moment you ship a desktop or mobile client, you should assume the binary can be inspected.

The usual pattern is:

• Flutter authenticates your user with your own backend.

• Your backend creates the avatar/session resources.

• Your backend returns a join token, room name, or short-lived session payload.

• Flutter uses those values to connect to LiveKit and display the avatar widget.

This is also where you decide whether the avatar is session-scoped or reusable. For support and sales use cases, a fresh session per conversation is usually simpler. For product walkthroughs or in-app assistants, you may keep the same avatar identity but create a new realtime session per interaction.

Wire LiveKit audio to the avatar lifecycle

The avatar should follow the same conversation lifecycle as the voice agent. That means the UI should not guess when the avatar is “talking”; it should react to actual agent/audio events. In a well-structured implementation, the backend owns the agent and session orchestration, and the Flutter side only subscribes to state changes like connected, speaking, listening, disconnected, or errored.

If you are building the voice agent in Python, the agent process can create or attach an avatar session before the conversation starts. Here is a minimal example using a REST-style setup to create a session from your backend. The exact field names depend on the endpoint in the docs, but the shape should look familiar:

import os

import os

import os

The important part is not the exact endpoint; it is that session creation happens server-side and the response is then used by the client to join the live conversation.

For voice agents built on LiveKit, the avatar needs to stay synchronized with audio playback and turn-taking. If the agent is interrupted, the avatar should stop or blend out quickly. If the agent is silent but still listening, the face should reflect that state rather than continuing a talking loop. This is mostly a media timing problem, not a UI problem, so handle it close to the agent where you already know when text-to-speech starts and ends.

How to structure the Flutter widget

In Flutter, the avatar widget should be treated as a render target plus a connection state machine. Do not make the widget responsible for session creation, retries against privileged APIs, or business logic about which avatar to use. Keep that in a controller or service layer above the widget.

A useful separation is:

• Backend: creates sessions, returns short-lived connection data.

• Flutter service: joins LiveKit, subscribes to events, manages reconnects.

• Avatar widget: renders the video or embedded avatar surface.

If the avatar arrives as a video track, your widget should attach to the stream the same way you would attach any other LiveKit video track. If it arrives as an embedded surface, the widget can wrap that surface and still use LiveKit for the audio session around it. Either way, the widget should remain dumb about conversation logic.

A simple Flutter-side controller can expose connection state to the UI:

enum AvatarStatus { idle, connecting, connected, speaking, error }

enum AvatarStatus { idle, connecting, connected, speaking, error }

enum AvatarStatus { idle, connecting, connected, speaking, error }

That controller is intentionally boring. Boring is good here. It makes reconnection, retries, and UI state transitions much easier to reason about than a widget that talks directly to every backend.

Practical gotchas: latency, sync, and retry behavior

Realtime avatars are sensitive to latency in a way that static avatars are not. A 200 ms delay in the voice path is noticeable; a 500 ms delay between speech onset and lip motion feels broken. A few rules help:

• Keep media and avatar state close. If your agent runs in one region and the Flutter client joins from another, expect extra delay.

• Start the avatar only when the session is ready. Don’t render a talking face before audio is actually flowing.

• Handle barge-in explicitly. If the user interrupts, the agent should stop speaking and the avatar should transition immediately.

• Rejoin cleanly. Mobile clients will lose network sometimes; your session model should tolerate disconnects and resubscriptions.

Another common mistake is letting the client infer too much from network state. For example, “the track exists” does not necessarily mean “the agent is currently speaking,” and “the room is connected” does not necessarily mean “the avatar session is live.” Use explicit session events from the backend or agent where possible.

Also pay attention to how you ship the Flutter widget across platforms. If you are targeting web, the browser tab and autoplay policies may affect audio startup. If you are targeting iOS or Android, foreground/background transitions can interrupt media playback. Test those paths early, because they usually fail differently than the happy path you exercise on desktop.

Using the LiveKit plugin path when the agent already exists

If you already have a LiveKit voice agent and just want to give it a face, the cleanest integration is often the agent-side plugin. The plugin examples show the general pattern: the agent process acquires avatar session state and the avatar follows the same conversational turn-taking as the rest of the media stack. For teams using Pipecat, the Pipecat integration guide is the most direct reference.

That path is especially useful if your Flutter app is just the client shell. In that setup, Flutter only needs to join the LiveKit room and render the resulting media. The agent, avatar, and session coordination all stay in the backend where they belong.

For quick experimentation, the repository of quickstarts linked from GitHub is useful when you want to see a full path from agent to rendered avatar rather than stitching the pieces together from scratch.

Where Protoface fits in this stack

Protoface is the layer that makes the avatar/session side manageable without turning your app into a custom media platform. In this specific LiveKit + Flutter setup, the most relevant surface is the LiveKit agent plugin or the backend API that creates and manages avatar realtime sessions. That lets your voice agent acquire a synchronized video face while your Flutter app stays focused on connection state and rendering.

If you need to provision sessions programmatically, the REST API is the right tool from your backend. If you want to keep your Flutter client thin and your agent implementation idiomatic, the plugin path is usually the cleaner fit. The docs at docs.protoface.com cover the exact payloads, auth, and event model.

Conclusion

The main idea is straightforward: let LiveKit handle the realtime voice transport, let your backend own avatar/session creation, and let Flutter render and react rather than orchestrate. That division keeps secrets out of the client, makes reconnection predictable, and gives you a clean path from prototype to production.

If you are building this now, start with one of the agent examples, connect it to a single Flutter screen, and validate the lifecycle with a real interrupted conversation rather than a happy-path demo. Once that is stable, layer in the rest of the app concerns: auth, session history, usage tracking, and the UI polish that makes the experience feel coherent.

For implementation details and the current API shape, go to docs.protoface.com.