Introduction

If you already have a FastAPI phone system, the missing piece is usually not speech recognition or text-to-speech. It is the visual layer: a synchronized face that can speak with low latency, track turn-taking, and stay aligned with whatever your voice agent is saying. That sounds cosmetic until you ship it and realize the avatar becomes part of the interaction model: users infer responsiveness, confidence, and even intent from the face.

This post shows the practical path for adding a conversational AI avatar to a FastAPI-based phone workflow. By the end, you should understand the realtime architecture, the integration points that matter, and how to keep the avatar synchronized with your agent without turning your backend into a streaming science project.

What “phone system + avatar” actually means

In a typical FastAPI phone application, your server coordinates call control, media handoff, and agent logic. A telephony provider delivers audio to your backend or to a media gateway. Your agent processes the caller’s speech, decides what to say next, and streams audio back. The avatar layer sits beside that loop, not inside it. It needs the same conversational state as the voice agent, but it does not replace the agent.

The important distinction is that the avatar is usually driven by the agent’s output, not directly by audio from the phone line. In practice, the avatar service receives text, timing cues, and/or speech audio from your agent pipeline, then produces synchronized video that you can render in a web client, iframe, or embedded session. For a phone call, that often means the caller talks over the phone while an operator, support rep, or customer-facing dashboard shows the avatar in a browser.

Architecture: keep the phone path and the avatar path separate

The cleanest design is to split the system into two asynchronous paths that share conversation state:

1. Phone/media path: telephony audio enters your FastAPI app or media service, then flows through ASR, LLM, and TTS.

2. Avatar path: your agent emits the same conversational turns to a realtime avatar session so the face animates in sync with the response.

This separation matters because the avatar has different performance and reliability constraints than telephony. Phone audio can tolerate a few hundred milliseconds more latency than a visual mouth movement that looks obviously “off.” If the avatar falls behind, the interaction feels broken even if the call itself is fine.

For FastAPI, that usually means your backend does three jobs:

• receives call events and caller audio,

• runs or forwards the agent pipeline,

• creates and tracks the avatar session used by the frontend.

Do not block your request handlers on avatar creation or media startup. Treat avatar session creation as an async side effect, and persist session identifiers alongside your call record so you can reconnect or audit later.

FastAPI integration pattern

The backend part is simple if you keep it explicit. When a call is established, your API can create a conversation record, provision an avatar session, and hand the frontend a short-lived session reference or embed URL. Your phone agent continues handling audio separately.

Here is a stripped-down example of the sort of orchestration you might build with a Python SDK. Exact field names and methods depend on the SDK version, so use this as a shape, not a copy-paste contract.

from fastapi import FastAPI

}

from fastapi import FastAPI

}

from fastapi import FastAPI

}

The practical point is not the exact SDK shape. It is that your backend should treat the avatar as an external realtime dependency with its own lifecycle. Create it, store its identifier, and tear it down when the call ends.

Streaming and sync: what tends to go wrong

Most integration bugs come from mismatched timing rather than from the avatar service itself. A few common failure modes are worth planning for:

• Turn timing drift: your TTS starts speaking before the avatar has the right text or phoneme cues.

• State divergence: the voice agent says one thing while the avatar shows a response from a previous turn.

• Frontend reconnection gaps: the call stays alive, but the browser view drops the avatar session and never reattaches.

• Latency spikes: a backend task serializes avatar updates and call processing on the same worker.

To avoid this, treat the agent as the source of truth for conversation state. The avatar should subscribe to the agent’s turn events, not independently infer what should be displayed. If your voice stack already emits structured events like user_speaking, assistant_thinking, assistant_speaking, and turn_complete, forward those to the avatar integration layer. That gives you a stable hook for lip sync, gaze, and idle-state animation.

For most production systems, the avatar update path should be idempotent. If a turn event is retried, the avatar should either ignore the duplicate or safely reapply the same state. That is especially important if your FastAPI app uses retries, background tasks, or queue-based workers.

Using the LiveKit voice-agent path

If your phone system already uses LiveKit Agents, the cleanest way to add a face is usually to drop in the avatar plugin rather than building a custom media bridge. The plugin integrates the avatar into the agent loop so your voice agent gains a synchronized talking video face without having to manage low-level video plumbing yourself. The relevant code and examples are in the plugin repository on GitHub, and the published package is on PyPI.

Conceptually, your agent setup becomes: ASR and LLM produce the response, TTS generates audio, and the plugin mirrors that response to the avatar session. The browser then receives the video face alongside the voice experience.

from livekit.agents import Agent

agent.add_plugin(avatar)

from livekit.agents import Agent

agent.add_plugin(avatar)

from livekit.agents import Agent

agent.add_plugin(avatar)

That is intentionally schematic. The value of the plugin is that it handles the sync boundary for you. You still need to manage call state, but you do not need to hand-wire every speech segment into a custom video renderer.

If you want the concrete package and examples, start with one of the quickstarts or the plugin repository, then adapt the call/control layer to your own FastAPI app.

Direct API orchestration from FastAPI

If you are not using LiveKit Agents, you can manage avatar sessions directly from FastAPI using the REST API or the Python SDK. This is useful when you already have your own telephony stack and want the avatar to be just another service in your architecture.

A direct API flow usually looks like this:

1. your FastAPI endpoint receives a call-start or operator-start event,

2. you create an avatar session, optionally selecting an avatar and voice/instructions for that session,

3. you return a URL or session token to the frontend,

4. your frontend connects and renders the avatar stream.

Here is a minimal REST example to show the pattern. Replace placeholder fields with the exact ones from the docs.

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

The main implementation detail is security: keep API keys on the server, never in the browser, and issue only the minimum session-specific data needed by the client. For most FastAPI apps, that means a backend endpoint that mints a short-lived session reference after auth checks pass.

Why iframe embeds are often the right frontend choice

For many phone-system use cases, the frontend is not your main product surface. You just need an operator view, a customer-facing call companion, or a support dashboard that displays the avatar reliably. In those cases, a customer-managed iframe embed is the lowest-risk path because no backend is required in the browser and no API key is exposed client-side.

That embed model also gives you a clean boundary for web security. You can enforce parent-origin allowlists, set per-embed voice and custom instructions, and apply rate limits by IP and duration. From a systems perspective, it is easier to reason about than a hand-rolled WebRTC integration in every client app.

Use the iframe route when:

• you want a fast integration for an internal dashboard or customer web page,

• you do not want to ship avatar credentials to the browser,

• you want the avatar lifecycle managed outside your FastAPI deployment.

Use direct API/session control when you need tight backend orchestration or more explicit control over session creation and teardown.

Operational details that matter in production

A few things are worth doing from day one:

• Log conversation and avatar session ids together so you can trace a bad call end to end.

• Separate agent latency from avatar latency in metrics. They fail differently.

• Expire sessions aggressively after the call ends to avoid stale browser tabs keeping resources alive.

• Keep the avatar instructions session-scoped if your use case changes per call, per customer, or per language.

Also pay attention to rendering quality versus cost. Protoface is billed by quality tier, so you should decide early whether a given workflow needs the highest-fidelity avatar or whether a cheaper tier is sufficient for internal tooling, preview flows, or low-stakes interactions.

Conclusion

Adding a conversational avatar to a FastAPI phone system is mostly an exercise in clean realtime architecture: keep the voice path and avatar path separate, make the agent the source of truth, and treat the avatar as a sessioned streaming dependency with its own lifecycle.

If you are already on LiveKit, the plugin route is the fastest way to get a synchronized face into the agent loop. If you are running your own stack, the REST API or Python SDK gives you direct session control. If you just need a web surface, the iframe embed avoids browser-side credential handling entirely.

For implementation details, start with the docs and the relevant quickstart or integration repo, then wire it into your FastAPI call flow.