Introduction

If you are replacing a legacy IVR with something conversational, the hard part is not “getting a model to talk.” The hard part is wiring together telephony, turn-taking, low-latency speech, state management, and a UI that gives the interaction some kind of presence. In practice, a voice agent without a face is fine for some use cases, but for many customer-facing workflows it feels like a phone bot. A realtime avatar changes that: the caller hears the agent and sees synchronized lip motion, which makes the interaction easier to follow and a bit more human without changing the underlying conversation architecture.

This post shows how to build that pattern in Flask with Twilio Voice, at the level you actually need to ship it: webhook plumbing, the audio/media path, session lifecycle, and where a realtime avatar fits. By the end, you should be able to stand up a basic IVR replacement that answers a Twilio call, hands audio to your agent stack, and drives a synchronized avatar for the caller-facing experience.

Start with the telephony and realtime constraints

Twilio Voice gives you the call control plane: inbound webhook requests, TwiML responses, and a media stream when you need raw audio. Flask is just the HTTP edge. The important design choice is that Twilio webhooks are synchronous and short-lived, while the actual conversation is long-lived and asynchronous. So your Flask app should do very little work in the request path: validate the call, return TwiML quickly, and hand off the realtime session setup to a worker or background task.

For a conversational IVR replacement, the usual flow is:

1. A caller dials your Twilio number.

2. Twilio hits your Flask webhook.

3. You return TwiML that either says a brief greeting or starts a media stream to your backend.

4. Your backend runs the voice agent loop: VAD or speech detection, STT, LLM turn generation, TTS, and interruption handling.

5. The avatar subscribes to the agent’s spoken output and renders a lip-synced video face for the caller or for an embedded web client.

The failure modes are usually around latency and state. If you block in the webhook, Twilio will time out or the caller will hear dead air. If your agent loop has high end-to-end latency, users will barge in or assume the system is broken. If you do not model session state explicitly, transfers, retries, and disconnects become messy fast.

Flask webhook: keep the request path thin

A minimal inbound webhook should authenticate the request, decide what kind of call this is, and return TwiML. If you are streaming media, the TwiML often just instructs Twilio to open a websocket to your media server. If you are using a vendor agent stack that manages the audio transport for you, the webhook may instead hand back a short greeting and then redirect the call into that stack.

from flask import Flask, request, Response

return Response(twiml, mimetype="application/xml")

from flask import Flask, request, Response

return Response(twiml, mimetype="application/xml")

from flask import Flask, request, Response

return Response(twiml, mimetype="application/xml")

That is intentionally not “the whole app.” The useful pattern is that the webhook only answers the phone. The realtime conversation lives elsewhere, where you can manage retries, queueing, and the agent lifecycle without racing Twilio’s HTTP timeout.

Build the agent loop around turn-taking, not just transcription

Once audio is streaming, the rest of the system is a realtime state machine. A common mistake is to think of the problem as “transcribe input, send to LLM, speak output.” That works for demos and fails for actual IVR replacement. In a call, turn-taking matters: the caller interrupts, changes their mind, or responds before the agent is done. Your pipeline needs to support partial speech, barge-in, and cancellation.

At a minimum, keep these states separate:

• Listening: accumulate caller audio and detect end-of-turn.

• Understanding: run STT and extract intent or generate a response.

• Speaking: stream TTS back to the call with low latency.

• Interrupted: stop generation immediately when the caller speaks.

For Twilio, the exact mechanics depend on your media transport. If you are using a websocket media stream, your backend receives audio frames and sends audio back on the same path or through a connected provider. If you already use a voice agent framework, keep Twilio at the edge and let the framework own the speech pipeline. The important part is that the conversation engine should emit structured events such as “agent started speaking,” “speech chunk generated,” and “agent stopped,” because the avatar needs those events to stay synchronized.

Drive the avatar from the same speaking events

The avatar should not be its own source of truth. It should subscribe to the agent’s output stream so the mouth movement tracks the actual audio being played. That means the video face is a rendering layer on top of the voice agent, not a parallel system.

Practically, this matters in three places:

1. Start time alignment: the avatar should begin motion when the agent actually starts outputting speech, not when text is generated.

2. Chunk continuity: if TTS is streamed in segments, the avatar should follow the entire spoken utterance across chunks.

3. Cancellation: if the caller interrupts, stop the audio and the avatar together so the lip sync does not lag behind the call.

If you are already on LiveKit for voice, this is where the avatar plugin model is useful. The agent owns the realtime media flow, and the avatar attaches to it as another participant or output surface. That avoids a second control plane in your Flask app and keeps the synchronization problem inside the agent runtime rather than in your web tier. The same principle applies if you run the conversation through another agent stack: the avatar should follow the voice events, not the other way around.

Handling state, transfers, and failure modes

Replacing an IVR means dealing with the boring parts that production systems always expose. You need a clean representation of the call session and clear boundaries between the telephony layer and the conversation layer.

Some implementation details that save pain later:

• Persist a call session record keyed by Twilio call SID, with agent/session identifiers, timestamps, and current state.

• Make initialization idempotent. Twilio can retry webhooks, and websocket reconnects happen. Session creation should be safe to repeat.

• Handle silence and fallback paths. If STT confidence is low or the caller is quiet too long, reprompt instead of letting the call stall.

• Support escalation. Real IVR replacement still needs a human handoff, voicemail path, or callback flow.

• Log the right events. You want call start/end, turn boundaries, interruptions, and transfer reasons, not just raw audio errors.

In Flask, that usually means the HTTP route stores metadata and enqueues work, while a worker process owns the long-lived conversation session. Do not try to keep the whole call loop inside the Flask request handler. It will work right up until traffic, retries, or network jitter show up.

Where Protoface fits

This is the point where a realtime avatar layer becomes useful instead of decorative. Protoface is built for the “voice agent with a face” part of the stack: you keep your telephony and agent logic, and attach a synchronized avatar to the speaking events. If your app already has a voice runtime, the most direct integration is through the LiveKit plugin or the Python SDK; if you are experimenting from scratch, the REST API can create and manage sessions programmatically.

A lightweight example using the Python SDK looks like this conceptually:

from protoface import ProtofaceClient

)

from protoface import ProtofaceClient

)

from protoface import ProtofaceClient

)

And if you want to inspect or automate session creation directly, the REST API is straightforward to use with a bearer token:

curl https://api.protoface.com/sessions \
}'

curl https://api.protoface.com/sessions \
}'

curl https://api.protoface.com/sessions \
}'

Exact request fields are in the docs, but the architectural takeaway is simple: keep the avatar session tied to the same conversational state as the agent. That makes synchronization and cleanup much easier than treating video as a separate subsystem.

Practical implementation notes

There are a few details that tend to matter more than the headline architecture:

• Latency budget: caller experience degrades quickly if turn completion takes too long. Keep STT, LLM, TTS, and avatar orchestration tight.

• Backpressure: if your audio output stalls, pause avatar updates rather than letting the session drift.

• Security: never put API keys in the browser; keep all session creation server-side.

• Observability: measure turn latency, interruption rate, fallback rate, and call drop-offs. Those numbers tell you whether the IVR replacement is actually better than the old menu tree.

If you are working through a concrete implementation, the quickstarts and reference material are worth keeping open while you wire things up: docs.protoface.com for API details and github.com/protoface-ai for integration examples. The fastest path is usually to get the agent audio loop stable first, then attach the avatar once the turn-taking is solid.

Conclusion

A realtime AI avatar IVR replacement is mostly a systems problem: Twilio handles call ingress, Flask handles the webhook edge, your agent stack handles speech and turn-taking, and the avatar subscribes to the same speaking events so the video stays aligned. If you structure it that way, the pieces stay decoupled and the call experience remains responsive enough to feel conversational.

Start small: answer a single Twilio number, stream audio into one agent session, log every turn, and attach an avatar only after the conversation loop is reliable. Once that works, the rest is iteration on prompts, routing, and failure handling. For implementation details and current API shapes, see the docs.