Introduction

Phone trees fail for the same reason most IVRs fail: they optimize for routing, not resolution. A caller who just wants to change a reservation, check an order, or ask about a billing issue ends up navigating menus, waiting on hold, and repeating context. A realtime voice agent fixes the conversation layer, but in practice you usually need one more piece: a visual face that moves in sync with the agent’s speech so the experience feels coherent in a web app or embedded support flow.

This post shows how to build a realtime AI avatar IVR in FastAPI, with the agent handling speech in the usual event-driven way and the avatar rendering the agent’s response as synchronized video. By the end, you should be able to wire up a voice agent, stream its audio to a browser or app, and attach a lip-synced avatar without exposing any secrets client-side.

What changes when IVR becomes conversational

Classic IVR is built around a state machine: play prompt, collect DTMF, branch, repeat. A voice agent is different. You still need a control plane, but the conversation itself becomes probabilistic and stateful. That has a few implications:

• Latency matters more than branching logic. If the agent takes too long to answer, the experience feels broken even if the answer is correct.

• Turn-taking is dynamic. The system should detect end-of-utterance and barge-in cleanly instead of waiting for the user to finish a long monologue.

• State belongs in the backend. Authentication, customer context, business rules, and escalation decisions should not live in the browser.

• Voice is not the whole UI anymore. If you want a usable customer-facing experience, a synced avatar can make the interaction legible and reduce the “talking to invisible software” problem.

In a FastAPI service, the easiest mental model is: your app owns orchestration, your voice stack handles streaming audio, and the avatar layer subscribes to the agent’s speech output. That keeps the IVR logic deterministic where it needs to be, while still giving the user a realtime conversational interface.

Architecture: FastAPI as the orchestration layer

FastAPI is a good fit because the HTTP surface is small, the async story is straightforward, and you can keep your realtime bits separated from your business logic. A typical deployment has three pieces:

1. Session bootstrap. Your backend creates or retrieves a call/session record, fetches customer context, and returns a short-lived token or connection payload.

2. Realtime voice path. The browser or telephony bridge connects to your agent runtime over WebRTC or an equivalent realtime transport. The agent receives user audio, runs ASR, LLM inference, and TTS, then streams the answer back.

3. Avatar rendering path. The avatar service subscribes to the agent’s synthesized speech and emits synchronized video frames so the face matches the audio timing.

You do not want the browser directly talking to every backend service. Keep the browser on a narrow diet: session creation, websocket/WebRTC signaling where needed, and rendering. Everything sensitive should stay server-side.

A minimal FastAPI session endpoint

The exact schema depends on your agent and avatar setup, but the pattern is consistent: create a session, return the connection details the client needs, and keep the authorization token short-lived.

from fastapi import FastAPI, Depends

}

from fastapi import FastAPI, Depends

}

from fastapi import FastAPI, Depends

}

In production, this endpoint should do the boring but important things: authenticate the user, enforce rate limits, attach audit metadata, and decide whether the caller should get the voice agent, a human handoff, or a fallback menu. The realtime layer should never be the place where access control happens.

Handling conversation state without creating a tangle

The biggest implementation mistake is trying to model the whole interaction as a giant prompt. That works for demos and collapses in production. A better pattern is to split state into three buckets:

• Ephemeral turn state: the last user utterance, current partial transcription, whether the agent is speaking, and interruption flags.

• Session state: caller identity, selected product, plan type, locale, and any policy constraints.

• Business state: order IDs, reservation records, payment status, or any external system data you fetched.

Your agent runtime should consume session state at the start of the call and update only the small amount of ephemeral state needed for turn-taking. That makes retries and recoverability much simpler. If the connection drops, you can reconstruct the call from durable session state instead of replaying the entire dialogue.

Realtime timing and the avatar layer

For a synced avatar, the important property is not “video” in the abstract; it is timing alignment. The face should start and stop with the agent’s speech, mouth movements should track phonemes closely enough to look natural, and the system should avoid noticeable drift if the user interrupts or the model streams tokens slowly.

That means the avatar renderer should subscribe to the same speech events driving TTS, not to the raw text output. If you try to animate from text alone, you lose the timing information that makes the face feel connected to the voice. If you try to animate from low-level audio after the fact, you add latency and complicate synchronization.

Practical gotchas:

• Keep the end-to-end path short. Extra proxy hops matter when you’re already doing ASR and TTS.

• Handle barge-in. If the user starts speaking while the agent is talking, stop the speech generation and cut the avatar cleanly.

• Expect partial transcripts. Many agents emit incremental ASR results; do not treat them as final intent signals.

• Design for degradation. If the avatar endpoint is unavailable, the voice agent should continue without it rather than failing the call.

Where Protoface fits

This is where Protoface is useful: it gives you a reusable avatar layer without forcing you to rebuild the synchronization logic yourself. If you are already running a voice agent in Python, the LiveKit plugin is the most direct integration point. You drop the avatar into the agent, and the agent keeps owning conversation flow while the avatar handles the synchronized visual surface.

For LiveKit-based stacks, the plugin is published as livekit-plugins-protoface. In practice, the integration is small: initialize the avatar service, attach it to the agent pipeline, and let the plugin follow the agent’s speech events. The exact constructor names and config fields are documented, so treat this as representative rather than copy-paste complete.

from livekit.agents import Agent

)

from livekit.agents import Agent

)

from livekit.agents import Agent

)

If you are not using the LiveKit plugin, the REST API and Python SDK are the right surfaces for session management and avatar lifecycle. The API is authenticated with Bearer API keys, which keeps sensitive operations server-side. That matters for anything production-facing: create the avatar or session on the backend, hand the browser only a short-lived connection artifact, and keep the key out of the client bundle.

import requests

print(resp.json())

import requests

print(resp.json())

import requests

print(resp.json())

For deeper reference, the public docs are the right place to confirm the current session and avatar fields: docs.protoface.com.

FastAPI deployment details that matter in production

Once the demo works, the hard part is keeping it reliable under real traffic. A few engineering details are worth getting right early:

• Use short-lived credentials. If a browser or iframe needs to connect, mint a narrow token and expire it quickly.

• Separate session creation from media transport. Your FastAPI app should not proxy audio/video streams unless you have a very specific reason.

• Log structured conversation events. Keep timestamps for turn start, ASR finalization, first token, TTS start, and turn end. That makes latency debugging tractable.

• Have a human escalation path. Any system that replaces phone trees should still allow a clean handoff when the agent cannot resolve the issue.

If you need an example repository to compare against, the quickstarts linked from the project README are useful starting points, especially if you want to see how a realtime agent is wired up end-to-end without inventing your own protocol glue.

Conclusion

Replacing a phone tree with a voice agent is mostly an orchestration problem: keep business logic in FastAPI, keep transport and media streaming narrow, and make sure your conversation state is explicit and recoverable. Once that’s in place, an avatar becomes a practical addition rather than a gimmick. It gives the agent a visible presence and helps the experience feel intentional instead of bolted together.

If you are building this now, start with a minimal backend session endpoint, wire up the agent loop, and then add the avatar layer once turn-taking is stable. The docs at docs.protoface.com cover the current API surface, and the GitHub examples are the fastest way to validate the integration path you’re using.