Introduction

If you are building a voice agent, “good enough” often stops at audio. The next obvious step is giving the agent a face that moves in sync with speech, so users can read intent, make eye contact, and stay engaged. In practice, that means wiring a realtime audio pipeline to a video surface that can keep up with low latency, lip sync, and session lifecycle events without turning your FastAPI app into a streaming server.

This post shows how to think about that integration cleanly: how to expose a FastAPI endpoint that starts a triage session, how the browser receives a realtime avatar stream, and what trade-offs matter when you connect voice, video, and application state. By the end, you should be able to add a working avatar-backed triage flow to a FastAPI app and know where the sharp edges are.

What “voice and video triage” actually means

In a triage flow, the avatar is not just decoration. It is part of the interaction loop:

• The user enters a support or intake flow.

• Your backend creates or selects a realtime avatar session.

• The avatar speaks, listens, and animates in sync with the conversation.

• Your app decides whether to answer directly, collect fields, or hand off to a human.

That implies a few technical requirements:

• Low latency: if the avatar trails the audio by more than a small amount, the illusion breaks.

• Session isolation: each user session needs its own avatar state, instructions, and permission boundaries.

• Browser-safe delivery: the client should not need your API key, and the video stream should not be proxied through your FastAPI app unless you have a very specific reason.

• Operational visibility: you want to know which sessions ran, how long they lasted, and what they cost.

That is why the integration pattern matters more than the individual SDK calls.

FastAPI as the control plane, not the media plane

The cleanest architecture is to let FastAPI handle application logic and session orchestration, while a dedicated avatar service handles realtime media. Your FastAPI app should:

1. Authenticate the user.

2. Decide which avatar or instruction set applies.

3. Create a session or embed token.

4. Return just enough data for the browser or agent runtime to connect.

This separation keeps the media path off your app servers. It also avoids common WebRTC mistakes: trying to relay RTC packets through your own backend, mixing ephemeral session state with long-lived process memory, or leaking API credentials to the browser.

For the browser side, the exact delivery mechanism depends on your product shape. If you are embedding an avatar directly into a web page, an iframe-based embed is the simplest route because it keeps credentials out of the client entirely. If you are integrating into an existing voice stack, you will usually create or attach an avatar session from the backend and then hand the client a URL or session identifier.

A minimal FastAPI flow for starting a triage session

The backend typically exposes a route that creates a session for the current user. The exact fields depend on the avatar API, but the shape is usually straightforward: send the intended voice, instructions, and maybe a reference to the avatar to use.

from fastapi import FastAPI, Depends

return resp.json()

from fastapi import FastAPI, Depends

return resp.json()

from fastapi import FastAPI, Depends

return resp.json()

That is intentionally illustrative. The response shape and request fields are documented in the docs, and you should treat those as the source of truth. The important idea is that FastAPI owns authorization and business rules, while the avatar service owns realtime delivery.

A few practical tips:

• Use short-lived session records in your own database if you need auditability or retries.

• Do not store API keys in browser-accessible config.

• Keep per-session instructions specific. Triage works better when the agent has a constrained role.

• Expect session creation to be a network hop and handle failures explicitly.

Where the avatar stream belongs in the client

Once your backend has created a session, the browser needs a way to attach to it. In WebRTC-based systems, this usually means the client joins a session and receives a media track that renders as video, while audio and state updates flow alongside it. The critical point is that your app should not try to “play” the avatar frame by frame itself. The realtime transport already handles jitter, playout timing, and sync.

If your triage UI is a standard web app, this usually looks like a small video element or iframe container plus a few UI controls: mute, restart, escalate, maybe a form for structured intake. Keep the avatar isolated from the rest of the app state. When the user navigates away, the session should end cleanly.

For triage specifically, the avatar should not be the only control surface. Pair it with deterministic UI when needed:

• Buttons for “billing,” “technical issue,” “sales,” or “human handoff.”

• Structured form fields for identifiers, order numbers, or contact info.

• Server-side branching when the session crosses a threshold or confidence drops.

This avoids forcing every interaction through speech when some steps are faster and safer as explicit UI events.

Using the LiveKit agent path when the avatar should ride along with speech

If your product already uses a LiveKit voice agent, the cleanest path is to add a synchronized talking face at the agent layer rather than bolting on a separate video subsystem. That keeps the avatar aligned with the agent’s speech lifecycle and simplifies timing.

The plugin lives in the LiveKit ecosystem, and the repository has examples you can adapt: GitHub org and the plugin package on PyPI are the two places to start. Conceptually, you drop the avatar service into the agent pipeline so when the agent speaks, the avatar renders the corresponding lip-synced video.

from livekit.agents import WorkerOptions

return agent

from livekit.agents import WorkerOptions

return agent

from livekit.agents import WorkerOptions

return agent

The exact import names and wiring depend on the package version, so treat the examples as a pattern rather than copy-paste code. The key architectural benefit is that the agent owns turn-taking and the avatar simply follows that state, which is usually what you want for conversational triage.

Operational concerns: latency, safety, and cost

Realtime avatars are sensitive to latency in a way ordinary REST APIs are not. A few guidelines help keep things predictable:

• Keep round trips short: session creation should be fast, and the browser should connect directly to the media service after that.

• Fail closed on auth: if a session cannot be created, do not expose fallback credentials in client error messages.

• Prefer explicit expiration: triage sessions should end when the user leaves or the task is complete.

• Track usage by quality tier: higher-fidelity video costs more, so make the tier a conscious product choice.

Another practical concern is escalation. An avatar can collect context, de-risk the first minute of contact, and route the user to the right queue. It should not pretend to solve every issue. Build in a handoff path to a human or a non-avatar workflow, and make that path first-class in your backend so you can trace the state transition later.

Finally, if you serve multiple brands or products from one FastAPI app, keep avatar configuration per tenant. Voice, instructions, and allowed session duration are all tenant-specific knobs that should live in your authorization layer, not in front-end code.

How Protoface fits into this pattern

Protoface is useful here because it gives you the avatar-specific pieces without forcing you to build a media backend. For a FastAPI app, the most relevant surface is the REST API: your server can create and manage avatars and sessions with an API key, then hand the browser or agent runtime the minimal data it needs. If you are already in the LiveKit ecosystem, the plugin path is a good fit because it keeps the avatar synchronized with the voice agent rather than treating video as an afterthought.

That separation is the main win. FastAPI stays focused on auth, business logic, and routing; the avatar service handles realtime sync and delivery; and your client code stays thin.

Conclusion

Adding a voice-and-video triage avatar to a FastAPI app is mostly an architecture problem. Keep FastAPI as the control plane, keep media off your backend, and make sessions explicit, short-lived, and tenant-aware. If you already have a voice agent, attach the avatar at the agent layer; if you are embedding into a web app, keep the browser integration credential-free and session-scoped.

For implementation details, check docs.protoface.com and the relevant quickstart or plugin repository for the stack you are using. Start with one flow, measure latency and handoff behavior, then expand once the session lifecycle is solid.