FastAPI Guide to Realtime Voice Avatar Routing for IVR and Call Flows
FastAPI guide to routing realtime voice avatars in IVR call flows with event-driven state, latency, and LiveKit integration.
Introduction
IVR systems and voice agents usually fail in the same place: the caller is talking to something that sounds intelligent, but feels opaque. A realtime avatar does not change the dialog manager, but it does change the experience boundary. Instead of a static hold message or a text-only widget, you can attach a synchronized talking face to the same realtime voice loop and route calls through it just like any other media endpoint.
This post is about the engineering side of that problem: how to route audio, state, and session control so a voice agent can drive a video face without adding noticeable latency or creating brittle media glue. By the end, you should be able to reason about the moving parts in a realtime voice-avatar flow, choose where the avatar should live in your architecture, and wire up a basic implementation with a clean separation between telephony, agent logic, and avatar rendering.
What “realtime voice avatar routing” actually means
In practice, you are connecting three pipelines:
1. Inbound and outbound audio. A caller speaks over PSTN, SIP, WebRTC, or a browser microphone. That audio is streamed into your agent stack, which may do ASR, turn-taking, tool calls, and TTS.
2. Agent state. The agent emits partial and final responses, interruption events, speaking state, and sometimes structured metadata like current intent or route.
3. Avatar rendering. The avatar receives the synthesized speech or speech timing information and produces a lip-synced video face that stays aligned with the audio stream.
The important part is that the avatar is not a separate “chat UI.” It is a realtime consumer of the same conversational events driving the voice agent. If you route those events cleanly, the face becomes just another session participant. If you route them poorly, you get the usual problems: speech/video drift, late interrupts, stale state, and bad handoffs.
Designing the call flow so the avatar stays in sync
For IVR and call-routing systems, the first design decision is where the routing authority lives. A useful default is:
The telephony or browser layer terminates the caller media stream.
Your agent service handles transcription, policy, and response generation.
The avatar session is attached to the agent session, not the telephony edge.
This keeps the avatar aligned with the actual conversational state instead of the transport layer. For example, if the caller interrupts mid-sentence, the agent can stop TTS, cancel the avatar’s current speaking segment, and immediately switch the face to listening mode. If you instead treat the avatar as an independent client, you will eventually fight race conditions between audio completion, visual completion, and state updates.
A few rules make this easier:
Route by conversation, not by request. Use one avatar session per active call or user session. Reuse session state only when the conversation itself is continuous.
Make interruption explicit. “Barge-in” should be a first-class event. When the user speaks, cancel pending output and update the avatar state immediately.
Keep latency budgets visible. Lip sync usually tolerates small jitter, but if your agent stack adds too much delay between TTS and avatar playback, the face will feel detached from the audio.
Separate routing from rendering. Your IVR decides what the agent says and where the call goes. The avatar service should only render the session state you send it.
In call flows, the common routing patterns are:
Front-door receptionist. The avatar greets the caller, qualifies intent, and then hands off to a specialist queue.
Contained support flow. The avatar handles an entire support interaction and only escalates when policy or confidence demands it.
Web-to-voice bridge. A browser conversation becomes a voice call or vice versa, with the avatar staying attached to the same session identity.
Implementation pattern: event-driven session orchestration
The cleanest implementation is usually event-driven. Your agent emits events; your orchestrator maps them to avatar updates. That avoids coupling your avatar rendering code to whichever ASR/TTS vendor you happen to be using this month.
At minimum, define a small session state machine:
idle — no active utterance
listening — caller is speaking or expected to speak
thinking — agent is processing
speaking — agent response is playing
handoff — transfer or escalation in progress
When a call starts, create an avatar session, bind it to the conversation ID, and keep the mapping in your application state. When the agent starts speaking, update the avatar to “speaking” and stream the output audio path that the avatar should track. When the caller interrupts, cancel both playback and the speaking state. When the conversation ends, close the avatar session and release any per-session resources.
The point of the example is not the exact method names; check the docs for the real shape of the SDK. The useful pattern is that your app owns the state machine, while the avatar session mirrors it.
How to think about latency, lip sync, and turn-taking
Realtime avatars expose timing bugs that text chat hides. In voice, a 300 ms delay is noticeable; in lip sync, a misaligned onset is even more visible. A few practical considerations matter more than vendor choice:
Preserve utterance boundaries. Do not batch unrelated agent responses into one long stream if you expect the avatar to animate turn-by-turn. Short, explicit utterances usually render more naturally than one continuously flowing block.
Stream early, but finalize cleanly. If your TTS or agent can stream partial audio, start playback early to reduce perceived latency. Just make sure you also send a definitive end-of-utterance event so the face can settle back into idle or listening state.
Handle barge-in at the media layer. Caller interruptions should stop audio playback, but they should also reset the avatar’s speaking animation immediately. Waiting for an async agent callback is often too slow.
Keep the source of truth singular. If the TTS engine, the telephony server, and the avatar renderer each infer “speaking” independently, they will drift. Pick one system as the authoritative session controller.
For IVR specifically, another common mistake is using the avatar as if it were a separate destination instead of a layer on top of the existing call flow. That creates duplicate routing logic and awkward transfers. Instead, let the IVR decide routing and use the avatar only for the live branch that actually benefits from a visible agent. The call may still transfer to a human, but the avatar session should be terminated or explicitly handed off at the same boundary.
Example: routing an avatar through a LiveKit voice agent
If your voice stack already uses LiveKit Agents, the simplest way to add a synchronized face is to drop in the Protoface plugin for that agent pipeline. That keeps the media graph intact and avoids building a separate video transport just for the avatar. The plugin is published on PyPI as livekit-plugins-protoface; the repo and examples are available on GitHub.
Conceptually, the flow is:
Your LiveKit agent receives caller audio.
The agent produces speech as usual.
The plugin attaches a realtime avatar session and keeps the face aligned with the agent’s speaking state.
If you are already using LiveKit for audio transport, this is the lowest-friction path because it keeps the realtime semantics in one graph. If you are not on LiveKit, the same design still applies: one session controller, one avatar session per conversation, and explicit state transitions for speaking, listening, and interruption. The transport changes; the orchestration model does not.
When the REST API is the right surface
For backends that create and manage avatars or sessions outside the agent runtime, the REST API is the right control plane. That is useful when your IVR orchestration runs in a separate service, or when you want to pre-provision avatars and attach them dynamically only when a call is connected.
A minimal request shape looks like this:
That separation is useful in production because it lets you keep API keys on the server, enforce your own routing policy, and audit session creation independently from the media path. If you need a programmatic wrapper, the Python SDK is the same idea with less JSON plumbing. The documentation at docs.protoface.com is the right place for the exact request and response fields.
Operational gotchas worth planning for
Two issues show up repeatedly in realtime voice-avatar systems.
First, session lifecycle leaks. If the call ends but the avatar session stays alive, you pay for stale state and eventually create confusing dashboards. Make session teardown part of the same code path as call hangup and error handling.
Second, inconsistent policy across surfaces. If your website embed, your agent backend, and your IVR all use different voice settings or instructions, the avatar becomes hard to reason about. Define a session template per use case and keep the routing logic in one place.
Also remember that realtime avatars do not replace good telephony engineering. You still need retries, idempotent session creation, explicit handoff semantics, and observability around latency, disconnects, and turn transitions. The avatar just makes those issues more visible.
Conclusion
The core idea is simple: treat the avatar as a realtime participant in the call, not as a separate UI artifact. If your agent owns the conversation state, and your avatar session mirrors that state through explicit events, you can route IVR calls and voice flows without fighting sync bugs or media spaghetti.
For a LiveKit-based agent, the plugin path is the fastest way to get a synchronized face. For backend orchestration, the REST API or Python SDK is the cleaner control plane. If you want to go deeper, the docs at docs.protoface.com and the examples in the GitHub organization are the most useful next stops.