Introduction

A realtime concierge avatar API is the glue between three systems that usually live separately: an LLM-driven voice agent, low-latency media transport, and a rendered face that can speak naturally enough to feel like part of the interaction. The core problem is not “generate a video of a face.” The problem is to keep audio, lip motion, speech timing, and turn-taking synchronized under realtime constraints so the avatar feels present rather than laggy or fake.

In practice, that means your application needs a way to create avatar sessions, feed them agent output, and receive a streaming video surface that updates quickly enough to track speech. By the end of this post, you should understand the architecture, the latency trade-offs, how to wire an avatar into a voice agent, and where a platform like Protoface fits when you do not want to build the media stack yourself.

What “realtime concierge avatar” actually means

Let’s define terms precisely. “Concierge avatar” usually implies a customer-facing agent: a support rep, sales assistant, onboarding guide, or game NPC that speaks, listens, and shows a face. “Realtime” means the avatar is not pre-rendered. It is driven continuously by live audio and conversation state, with latency low enough that turn boundaries feel natural.

There are a few moving parts:

• ASR / speech input: user audio is transcribed or routed into the agent.

• LLM / dialog policy: the agent decides what to say next.

• TTS / speech output: text becomes audio, ideally streamed.

• Avatar rendering: the face must lip-sync to the outgoing audio and reflect the current speaking state.

• Transport: the app needs a low-latency way to move media between client and server, typically via WebRTC or a similar realtime streaming path.

The important implementation detail is that the avatar should not be treated as a static video asset. It is a live participant in a session. That changes how you think about lifecycle, state, and failure modes.

Architecture: what you actually need to build

If you are integrating a realtime avatar into a product, the cleanest architecture is usually session-based. A client joins a session, your backend or agent runtime establishes identity and permissions, and the avatar is attached to that session as a media source. The avatar then follows the speech state of the agent, not the browser tab or page lifecycle.

In a typical voice-agent flow:

1. The user opens your app and starts a call or chat.

2. Your backend creates or authorizes a session.

3. The agent receives user audio, responds with text or speech, and emits speaking state.

4. The avatar consumes that state and streams synchronized video back to the client.

Two things matter most here.

First, latency budget. If your pipeline spends too long on transcription, reasoning, TTS, or video generation, the avatar will feel disconnected. You generally want short end-to-end turn latency and continuous streaming rather than batching everything after a full response is computed.

Second, state coherence. The avatar should know when the agent is listening, speaking, or idle. Even if the mouth animation is the obvious visible signal, the best implementations also align gaze, nods, and speaking onset/offset to the conversation state.

Integration pattern: attach the avatar to the agent, not the UI

For voice-agent systems, the safest integration pattern is to make the avatar a media component of the agent runtime. That keeps turn-taking consistent and avoids duplicating logic in the frontend. If you already use LiveKit for your agent, this is particularly straightforward because the avatar can be inserted into the same realtime graph as the voice agent itself.

Here is the conceptual shape of the integration:

agent audio/text output -> avatar session -> synchronized video stream -> client UI

agent audio/text output -> avatar session -> synchronized video stream -> client UI

agent audio/text output -> avatar session -> synchronized video stream -> client UI

That is preferable to “send text to frontend and animate there” because the browser should not be responsible for reconstructing the agent’s conversational timing. The backend already knows when speech starts, pauses, and ends. Use that.

Python SDK and REST API: creating sessions programmatically

When you need explicit control over avatars and sessions, use the REST API or the Python SDK. The REST API is useful for provisioning, automation, and backend orchestration. The Python SDK is useful when your application is already Python-based and you want a thin layer over the HTTP calls.

A minimal REST call usually looks like this: authenticate with an API key, create a session, then hand the resulting session information to your agent or client flow. The exact request body depends on the fields you choose in the docs, but the pattern is stable.

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

In Python, the flow is similar. Keep the SDK usage small and explicit, and let the backend own secrets:

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

Use the SDK or REST API when you need server-side control over avatar lifecycle, usage tracking, or per-session configuration. For example, you may want to create sessions only after authenticating a user, or set instructions and voice parameters based on the customer account.

For implementation details and the current object model, refer to the docs at docs.protoface.com.

Flutter integration: what changes on the client

Flutter is a good fit for these experiences because it gives you a single UI stack across mobile, desktop, and web, while still allowing you to consume realtime media. The client-side job is not to “make the avatar intelligent.” The client job is to join the session, render the video, and keep the interaction responsive.

In Flutter, the practical concerns are:

• Session bootstrap: fetch session credentials or a join token from your backend.

• Media rendering: display the avatar’s video track with minimal buffering.

• Lifecycle handling: reconnect cleanly when the app goes background/foreground or network quality changes.

• UI state: show speaking/listening status without assuming the avatar is always active.

The exact widgets and transport layer depend on the media stack you use, but the general rule is the same: the Flutter app should be a thin consumer of a realtime session, not the source of truth for conversational state.

A common mistake is to over-animate the UI independently of the backend. If the agent has not actually started speaking, do not force the avatar to look like it is talking. Users notice these mismatches immediately, and they erode trust faster than a plain fallback UI.

Where Protoface fits

Protoface is the layer that provides the avatar session, the realtime media surface, and the developer-facing control plane so you can focus on the agent and product logic. In a LiveKit-based voice agent, the plugin approach is the most direct: add the avatar as part of the agent runtime so it inherits the same session timing and audio state. The published plugin on PyPI is pipecat-protoface, and the implementation examples are in the relevant GitHub repositories.

If you are using Pipecat, the integration guide is worth reading because it shows the adapter pattern cleanly: the agent remains the orchestrator, and the avatar becomes another realtime service in the pipeline. See the Pipecat docs for the service integration details at docs.pipecat.ai/api-reference/server/services/video/protoface.

For teams that want no backend exposure in the browser, the customer-managed iframe embed is the other practical surface. That model is useful when you want a website widget with per-embed instructions, parent-origin allowlisting, and rate limits without shipping API keys to the client. It is less flexible than a fully custom agent backend, but it is much easier to operationalize for a simple web experience.

Operational gotchas and trade-offs

There are a few things worth planning for up front.

Latency vs. realism. Higher-quality avatar rendering and more complex motion can increase cost and response time. If you are building a support flow where immediate responsiveness matters, pick a quality tier that keeps the interaction snappy rather than chasing the most detailed rendering.

Session ownership. Decide early whether sessions are created by your backend, by the agent runtime, or by a separate orchestration service. Do not let the frontend become the authority for session identity if you care about security and debuggability.

Security. API keys should stay server-side. If you need browser embedding, use a managed iframe model that never exposes the secret key in the client.

Fallback behavior. Always plan for degraded mode. If the avatar stream is unavailable, your agent should still work as a voice experience or a plain chat interface. The face is an enhancement, not the product’s only path to completion.

Conclusion

A realtime concierge avatar API is essentially a realtime media and session layer for conversational agents. The important ideas are session orchestration, low-latency transport, synchronized speech state, and clean separation between backend control and frontend rendering. If you get those right, the avatar feels like part of the agent instead of a bolted-on video effect.

For implementation details, start with the docs at docs.protoface.com, then choose the integration surface that matches your stack: REST API for orchestration, Python SDK for backend code, LiveKit or Pipecat plugins for voice agents, or an iframe embed for a browser-native experience. The quickest way to validate the whole flow is to build a minimal session end-to-end, then tighten latency, state handling, and error recovery once you can see the avatar speak in sync.