Introduction

If you already have speech-to-text (STT) and text-to-speech (TTS) working in a Flutter app, the next obvious step is usually a visual one: instead of playing an audio-only assistant, show a realtime talking face that stays synchronized with the agent’s speech. That sounds simple at the product level, but there are a few technical pieces to get right: audio capture, turn detection, streaming latency, lip-sync alignment, and the UI mechanics of rendering a video/avatar surface in Flutter without making the app feel laggy.

This post walks through the integration pattern I’d use in a production Flutter app. By the end, you should understand how to wire STT and TTS into a realtime conversation loop, where the avatar sits in that loop, and what trade-offs matter when you care about responsiveness more than demo polish.

What “realtime video avatar” actually means in an app

For developers, a realtime avatar is not “generate a video file from text.” It is usually a low-latency media session where:

• the user speaks into the app;

• STT turns live audio into text, often incrementally;

• your agent decides what to say next;

• TTS turns that response into streaming audio;

• the avatar renders a talking face that is synchronized to that audio, usually over WebRTC or a similar low-latency streaming path.

The important bit is synchronization. If the avatar’s mouth movement is tied to the same audio that the user hears, you avoid the uncanny mismatch that happens when you try to bolt together independent video and audio pipelines.

In Flutter, this tends to break down into two channels:

• Control plane: session creation, auth, instructions, voice selection, and lifecycle management.

• Media plane: the actual audio/video stream, usually handled by a realtime transport and rendered in a platform view or equivalent widget.

Keep those mentally separate. Most integration bugs come from mixing them up.

Build the conversation loop first, then add the face

Before you touch the avatar, make sure your voice loop is stable. A good minimal architecture is:

1. Capture microphone audio in Flutter.

2. Stream it to STT.

3. Detect a final user turn.

4. Send the recognized text to your assistant logic.

5. Stream the assistant’s response into TTS.

6. Play the audio back immediately.

That last step matters because the avatar should follow the spoken output, not the text response. If your agent generates a long answer, you want TTS streaming to start as soon as possible so the user gets feedback quickly and the lip sync stays aligned with what is actually audible.

Two practical points:

• Use incremental STT if possible. Final-only transcription adds avoidable latency. Interim results let you precompute intent, trigger barge-in handling, and shorten the turn.

• Keep turn detection explicit. Do not assume “silence means done” unless your use case is very simple. For conversational agents, you usually want a clear end-of-utterance rule and a separate interruption path.

Flutter integration patterns that hold up

There are three common ways to embed a realtime avatar in Flutter:

1) Native widget or platform view for the video surface. This is the cleanest when the avatar provider gives you a media stream that can be rendered directly. You keep the media path separate from the rest of your UI, and your Flutter code only owns layout and lifecycle.

2) WebView or iframe-like embed. This is useful when you want the simplest integration and you do not need deep native control. It is often a good fit for prototypes or for apps where the avatar is a contained panel rather than a core interaction surface.

3) Full programmatic session control. This is what you want if the avatar must react to app state, authenticated user context, or custom turn-taking logic. In this model you create a session in your backend, pass session parameters from Flutter, and attach the rendered media in the client.

For production, I prefer a design where Flutter never handles API keys directly. The app talks to your backend, and your backend talks to the avatar API. That keeps credentials out of the client and gives you a place to enforce session policy.

Why timing and turn-taking are harder than they look

When you add a face to a voice agent, the user notices latency much more than they would in an audio-only experience. A small delay that is tolerable in a chatbot becomes obvious when the avatar’s mouth is frozen.

Three failure modes show up repeatedly:

• Avatar starts speaking before TTS audio is ready. The mouth moves but the audio lags, so the animation feels detached.

• Audio starts before the avatar session is actually connected. The first syllables can be missed or desynced.

• Interrupted turns are handled late. The user talks over the assistant, but the UI keeps playing stale audio and the avatar keeps talking.

The fix is to treat the conversation as a state machine. You want states like connecting, listening, thinking, speaking, and interrupted. In Flutter, that usually means one source of truth in your app state and a stream of events from STT/TTS/session transport.

Also, be careful with audio sample rates and resampling. If your microphone input, STT service, and TTS output disagree on codec or sample rate, you may not notice in functional tests, but you will feel it in latency and sync quality.

// Pseudocode for the turn lifecycle in Flutter

}

// Pseudocode for the turn lifecycle in Flutter

}

// Pseudocode for the turn lifecycle in Flutter

}

Where Protoface fits in

This is the part where Protoface is useful: it gives you a developer-facing realtime avatar layer that you can attach to a voice agent instead of building the lip-sync/video piece yourself. For a Flutter app, the cleanest pattern is usually to keep your STT/TTS and agent logic where they already live, then connect the conversation to a managed avatar session.

The control plane is exposed through the REST API at api.protoface.com, authenticated with an API key, so session creation belongs on your server rather than in the Flutter client. If you want programmatic control from backend code, the Python SDK is the most direct route; if you are embedding a voice agent on the media side, the LiveKit plugin is the relevant integration surface for attaching the avatar to a realtime agent. The docs and quickstarts are the right place to check the exact request fields and lifecycle details: docs.protoface.com and the examples linked from the quickstart repository github.com/protoface-ai.

A minimal backend call might look like this conceptually:

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

The exact payload shape depends on the API version and session model, so treat the snippet as illustrative. The important design choice is that your Flutter app should receive only a short-lived session reference or connection metadata, not your long-lived API key.

Practical implementation notes for Flutter teams

Here are the details that usually matter in real apps:

• Do not block the UI thread. Session setup, transcript handling, and media events should all be asynchronous.

• Model reconnects explicitly. Mobile networks change, and realtime media sessions do drop. Your avatar widget should recover without forcing a full app restart.

• Decide where interruption lives. If the user speaks while the assistant is talking, stop TTS, stop playback, and signal the avatar session immediately.

• Measure end-to-end latency. Track “user stopped speaking” to “avatar started responding,” not just STT or TTS latency in isolation.

• Keep instructions server-side if they are session-specific. That gives you auditability and lets you change behavior without shipping a new app.

If you are already using a voice stack based on LiveKit, Pipecat, or another agent framework, the integration point is usually narrower than it first appears. You do not need to re-architect your agent; you need a media avatar that can keep up with your existing audio flow.

Conclusion

The core idea is straightforward: keep STT, agent logic, TTS, and avatar rendering as a single realtime conversation pipeline, and be strict about latency and turn state. In Flutter, that usually means your app owns the UI and mic interaction, while your backend owns session creation and sensitive credentials. Once the voice loop is stable, adding the avatar is mostly an integration problem, not a model problem.

If you want to implement this with less custom media plumbing, start with the docs at docs.protoface.com, then pick the integration surface that matches your stack: REST for backend-controlled sessions, the Python SDK for server-side orchestration, or the LiveKit plugin if your voice agent already runs there. The quickest path is usually to wire the conversation loop first, then drop the avatar into it once you can measure the latency budget end to end.