Guide to Adding a Conversational AI Avatar to Your IVR Flow with SIP, STT, and TTS
Add a real-time AI avatar to IVR with SIP, streaming STT/TTS, latency budgeting, barge-in, and lip-sync sync tips.
Introduction
If you already have a voice IVR or agent pipeline, adding a talking face is mostly a systems integration problem: you need a low-latency path from telephony audio into speech-to-text, into an LLM or dialog manager, back out through text-to-speech, and then into a synchronized avatar renderer that can keep up with the audio stream. The hard part is not “making a face move.” It is keeping turn-taking, latency, and audio/video alignment stable enough that the experience feels conversational instead of stitched together.
In this post, I’ll walk through the practical architecture for adding a conversational AI avatar to an IVR flow, where SIP, STT, and TTS fit, and what to watch for when you move from a demo to a production call path. By the end, you should be able to map your existing voice stack onto an avatar-enabled flow and understand where to insert the avatar without breaking call quality.
The core call path: SIP to audio, audio to text, text to response, response to speech
An IVR conversation with an avatar typically looks like this:
A caller reaches your telephony edge over SIP.
The media server or voice gateway hands the caller audio to your voice agent.
Streaming STT converts partial audio into partial text.
Your dialog logic or LLM decides when the user has finished speaking and what to say next.
Streaming TTS generates audio as soon as the response is available.
The avatar renderer consumes that audio and generates a synchronized lip-synced video face.
The important design choice is that the avatar should follow the audio, not the other way around. In other words, do not wait for a complete transcript, a full LLM response, and a full TTS file before starting the visual output. That path adds avoidable latency. For a phone conversation, every extra few hundred milliseconds is noticeable, especially before the agent speaks.
Streaming is the key. STT should emit partial hypotheses so you can detect turn completion. TTS should begin as soon as the first sentence or clause is ready. The avatar renderer should accept an audio stream or tightly chunked audio frames so lip motion stays aligned with phonemes and timing.
Latency budget and turn-taking
For IVR, the avatar is only credible if the whole loop stays within a reasonable latency budget. In practice, the biggest delays usually come from:
Audio ingress/egress: SIP gateways, transcoding, jitter buffering.
Endpointing: deciding whether the caller is still speaking.
LLM think time: especially if you wait for full-turn completion.
TTS startup: some voices have a non-trivial first-byte delay.
Video generation/rendering: the avatar must keep pace with audio.
To keep the interaction natural, you usually want an interruption model that supports barge-in. If the caller starts speaking while the agent is talking, the system should stop or attenuate TTS playback, close the avatar mouth state quickly, and hand control back to STT. That means your media pipeline needs explicit interruption handling, not just a unidirectional “bot speaks after user stops” implementation.
Another operational detail: don’t treat the transcript as your source of truth for audio timing. STT is a control signal, not a playback clock. The avatar should be driven from the actual TTS audio stream or frame timestamps, because that is what determines mouth shape and expression timing.
Where the avatar fits in the architecture
There are two common ways to introduce a video face into an IVR system:
Sidecar to an existing voice agent: your current agent handles STT, dialog, and TTS; the avatar subscribes to the synthesized audio and renders synchronized video.
Integrated media pipeline: the voice agent, TTS, and avatar live in one realtime runtime, which reduces glue code and keeps stream timing tighter.
The sidecar model is usually the least disruptive if you already have a working call flow. You keep your telephony and agent logic unchanged, and you add the avatar as a consumer of the agent’s outbound audio. This minimizes risk and makes it easier to test the visual layer independently.
For production, the main gotchas are operational rather than conceptual:
Codec and resampling mismatches: telephony audio is often narrowband, while your avatar stack may expect a different sample rate.
Clock drift: if audio and video timestamps are not consistently derived from the same stream, lips will slip over time.
Session lifecycle: calls can reconnect, transfer, or end abruptly, so session cleanup must be deterministic.
Privacy boundaries: if you expose an avatar on the web, do not leak API keys into the browser.
Minimal integration example with a voice agent
If you are using a Python-based voice agent, the typical pattern is to load the avatar plugin into the agent runtime and pass through the realtime audio stream. The exact object names and fields vary by stack, so treat this as illustrative rather than copy-paste complete.
The important idea is not the exact API call; it is that the avatar should attach to the same session as the voice agent so the renderer sees the same conversational turns and the same audio timing. If your agent framework already supports realtime media hooks, the integration should be shallow. If it does not, you will likely need a thin adapter that forwards the TTS stream and turn state into the avatar runtime.
Managing sessions and avatars from the API
When you need to provision avatars or manage sessions programmatically, use the REST API or the Python SDK rather than baking these steps into your application startup. That gives you a cleaner separation between control plane and data plane: your service can create or select an avatar configuration, then hand off the live conversation to the media pipeline.
A simple API call looks like this:
And a Python SDK flow is similar in shape:
Use the API for lifecycle work: creating sessions, tracking usage, and cleaning up after calls. Keep long-lived secrets on the server side. If your deployment includes a web front end, avoid exposing keys in the browser; use server-issued session identifiers or a customer-managed embed flow instead.
Why this matters for IVR specifically
IVR systems are sensitive to small failures because callers are already constrained by telephony latency and limited input channels. A bad voice agent can still be tolerable if it is fast and accurate. A bad avatar can be worse if it introduces lag, desynchronization, or awkward visual pauses. So the implementation goal is not “show a face on every call.” It is “add a face without changing the call’s realtime characteristics.”
That means you should test three things before rollout:
Speech overlap handling: can the system interrupt cleanly when the caller barges in?
End-to-end delay: how long from caller speech end to avatar speech start?
Sync stability: does the mouth stay aligned through longer responses and multiple turns?
For most teams, the fastest path is to start with one agent flow, keep the avatar session scoped to a single call, and verify that the visual layer does not affect your existing telephony SLOs. Once that works, you can decide whether to extend the same model to web voice agents or other realtime surfaces.
How Protoface fits
This is exactly the kind of integration that Protoface is built for: you keep your voice stack, and attach a realtime avatar session to the audio stream. If you are already using LiveKit-based voice agents, the LiveKit-oriented quickstart is the most direct way to see the plugin path in practice, and the docs at docs.protoface.com cover the session and avatar lifecycle details.
The practical benefit is that you do not need to invent your own lip-sync layer or expose browser credentials. You can manage avatars and sessions from your backend, keep the media path realtime, and focus your engineering effort on dialog quality and call routing rather than video plumbing.
Conclusion
Adding a conversational avatar to an IVR flow is mostly about respecting realtime constraints: stream audio end to end, use STT for turn detection, use TTS as the source of truth for playback timing, and keep the avatar tightly coupled to that audio stream. If you preserve the audio timing, the face becomes a useful layer rather than a fragile one.
Start with one call path, verify latency and interruption behavior, then expand from there. For implementation details, the quickest next step is to read the docs at docs.protoface.com and adapt the quickstart that matches your agent stack.