Introduction

Adding a realtime AI voice avatar to a contact center IVR sounds simple until you run into the actual constraints: low-latency turn taking, clean audio transport, stateful call flows, and a video surface that stays synchronized with what the agent is saying. In practice, you are stitching together telephony, a voice agent, and a streaming avatar renderer while keeping the whole thing responsive enough that callers do not notice the seams.

This post shows one practical way to do it in Python. By the end, you should be able to wire a voice agent into an IVR flow, forward its audio into a realtime avatar session, and understand the latency and integration trade-offs that matter in production. We will keep the code focused on the moving parts that matter: agent lifecycle, media transport, and where the avatar session fits.

What “realtime avatar” means in an IVR context

An IVR is usually built around audio-only telephony: the call platform sends inbound audio to your app, your app returns synthesized speech, and the caller hears a voice. A realtime avatar adds a synchronized video face on top of that same voice interaction. The avatar is not a separate conversation layer; it is a visual presentation of the same agent turn.

That distinction matters. You do not want to “drive” the avatar independently from the agent. You want one source of truth for dialogue state and one low-latency media pipeline. If the voice agent pauses, the avatar should pause. If the agent is interrupted, the avatar should stop talking immediately. If the agent emits a short confirmation like “one moment,” the mouth movement should reflect exactly that utterance, not some delayed approximation.

In practice, the architecture looks like this:

• Telephony or RTC platform provides incoming audio.

• Your voice agent processes the audio, maintains conversational state, and emits a response.

• The response audio is streamed to the caller.

• The same response is sent to the avatar session so the face can lip-sync in realtime.

The hard part is keeping the audio and video paths aligned while avoiding extra buffering. Any unnecessary queueing becomes obvious as stale lip movement or a speaker who appears to talk after the words have already played.

Use a single agent loop and fan out the output

The safest pattern is to treat the agent as the canonical turn manager and fan out its output to both the telephony audio path and the avatar session. In other words: do not build two separate agents that independently generate speech. Generate once, then stream to multiple consumers.

In a Python service, that usually means an async task reading agent output events and forwarding them to the transport layers. The exact plumbing depends on your agent stack, but the shape is the same:

import asyncio

await avatar_session.stop()

import asyncio

await avatar_session.stop()

import asyncio

await avatar_session.stop()

There are a few important properties here:

• Backpressure must be visible. If the avatar side slows down, you do not want to stall the telephony path. If needed, decouple them with bounded queues and drop or coalesce noncritical frames.

• Interrupts should propagate immediately. In a real IVR, barge-in is normal. If the caller speaks over the agent, both the caller audio and avatar playback need to stop together.

• Audio chunking matters. Very large chunks improve throughput but increase latency. Very small chunks reduce latency but can increase overhead. Use a chunk size that your agent stack and media transport can sustain consistently.

Telephony integration: keep the media path boring

The fastest route to a stable deployment is to keep the telephony integration as conventional as possible. Terminate the call stream in your backend, normalize it to the format your voice agent expects, and then keep the call leg and avatar leg separate from that point onward.

That means watching for the same classes of issues you would solve in any realtime voice system:

• Sample rate conversion between telephony audio and your agent’s expected format.

• Voice activity detection and barge-in detection so the caller can interrupt naturally.

• Session lifecycle tied to the call lifecycle, not to an individual websocket reconnect.

• Timeouts and recovery so a transient avatar failure does not take down the voice agent.

If your IVR already uses a call control API, keep that integration narrow. Answer the call, open the media stream, and hand the call off to the agent loop. Do not let your business logic sprawl across the telephony gateway. The smaller the surface area, the easier it is to reason about latency and failure modes.

Python sketch: creating a session and attaching it to the agent

Protoface exposes a REST API and a Python SDK for managing avatars and realtime sessions. The exact request fields depend on the session model you choose, but the operational idea is straightforward: create a realtime session, then bind the session to the audio output produced by your voice agent.

Here is a minimal example using the Python SDK shape you would expect in a service that provisions a session at call start. Treat the names as illustrative and check the docs for the exact parameter names and return fields.

from protoface import Client

print(session.stream_url)

from protoface import Client

print(session.stream_url)

from protoface import Client

print(session.stream_url)

In a production IVR, you would typically create the avatar session when the call is answered, then keep the returned session handle in the call context. That handle is what your audio pipeline writes to while the call is active.

If you prefer to verify session creation outside of code, the REST API supports the same workflow. For example, a session request will look roughly like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

That separation is useful when debugging. If session creation works in isolation but the avatar is not moving in the call, the bug is usually in your media forwarding code, not in avatar provisioning.

Where the LiveKit plugin fits

If your voice agent already runs on LiveKit, the simplest integration is to drop the avatar directly into the agent using the LiveKit Agents plugin. The plugin handles the avatar-side wiring so the agent’s voice and the talking face stay synchronized without you writing custom media glue.

The operational model is the same as above: your agent still owns conversation state, turn detection, and interruption handling. The plugin just gives that agent a synchronized video face. That is the right abstraction boundary for contact center work, because you want the call logic to stay in your app and the visual rendering to stay in the avatar layer.

# Illustrative only; exact imports and constructor args are in the docs.

agent.add_plugin(avatar)

# Illustrative only; exact imports and constructor args are in the docs.

agent.add_plugin(avatar)

# Illustrative only; exact imports and constructor args are in the docs.

agent.add_plugin(avatar)

For teams already using LiveKit, this is usually the cleanest route because it reduces the number of custom streaming boundaries you have to maintain. You still need to think about barge-in, session cleanup, and failover, but you avoid rebuilding the avatar transport yourself.

Operational gotchas that matter in production

Realtime avatar work tends to fail in subtle ways. A few things are worth calling out explicitly:

• Keep session creation out of the hot path where possible. If your IVR has a predictable warm-up phase, provision the avatar session early enough that the first spoken turn is not blocked by setup latency.

• Handle partial failures. If the avatar stream drops but the voice path is healthy, decide whether to continue audio-only or to fail the call gracefully. Do not let the whole conversation collapse unless your product requires it.

• Use realistic rate limits. For customer-facing workloads, per-call limits and duration limits protect you from runaway sessions and accidental test traffic. This is especially relevant when sessions are created dynamically.

• Measure end-to-end latency. Track time from caller utterance to agent response, not just model inference time. The media pipeline often dominates the perceived delay.

You should also think about security boundaries. API keys belong server-side only. If you are creating sessions from a browser workflow, the safer pattern is an embed flow that keeps credentials off the client entirely. For an IVR backend, this is simpler: your service can talk directly to the API and keep secrets in your normal server-side secret manager.

Why this fits Protoface

Protoface is built for exactly this kind of integration: a voice agent that needs a synchronized talking face without forcing you to build a custom video pipeline. The REST API and Python SDK let you create and manage sessions programmatically, and the LiveKit plugin is the practical path if your contact center voice stack already lives there. That keeps the avatar layer close to the agent, which is what you want for low-latency turn taking.

If you need the exact session schema, auth details, or the supported integration patterns, the documentation is the right place to start: docs.protoface.com. The quickstart repo is also useful if you want a working reference instead of piecing together the plumbing from scratch.

Conclusion

To stream a realtime AI voice avatar into an IVR, model the problem as one agent with two synchronized outputs: telephony audio and avatar video. Keep the media path boring, propagate interrupts immediately, and avoid duplicating conversation logic across transport layers. If you are already on LiveKit, the plugin path is the shortest way to get there; if you are managing sessions directly, the REST API and Python SDK give you the control you need.

The practical next step is to prototype the call flow end to end with a single turn: answer a call, create a realtime avatar session, forward one assistant response, and verify that audio, lip sync, and interruption behavior all line up. Once that is solid, expand into your actual IVR branches and failure handling. For implementation details and current examples, start with the docs at docs.protoface.com.