Introduction

Replacing an IVR is less about “adding AI” and more about changing the control plane for customer calls. A legacy phone tree is deterministic, cheap to run, and easy to reason about, but it fails the moment callers need to explain intent in their own words, recover from ambiguity, or hand off gracefully to a human. A realtime AI avatar phone agent addresses that by combining speech input, streamed model output, and a live video face that stays synchronized with the conversation.

By the end of this post, you should have a practical mental model for building a phone agent that answers inbound calls over Twilio, streams audio to a realtime agent over WebRTC, and renders a lip-synced avatar that makes the interaction feel like a coherent support experience instead of a voice bot bolted onto a call tree.

Why IVRs break down in practice

Traditional IVRs encode intent as menu depth. That works only if you can predict the caller’s problem and keep the menu short. In production, you usually get the opposite:

• Callers start with partial context: “I need to change the address on my account, and I already tried the website.”

• They interrupt prompts, speak over audio, or answer with details the IVR did not ask for.

• They need a stateful interaction, not a one-shot routing decision.

The technical limitation is that a phone tree is a fixed finite-state machine with poor coverage for natural language. A realtime agent is closer to a streaming pipeline: audio in, transcript/intent extraction, response generation, and audio/video out. The agent can still route, authenticate, or escalate, but it does so from a richer representation of the conversation.

What the runtime architecture actually looks like

At a high level, the telephony stack becomes a bridge between PSTN audio and a realtime media session:

1. Twilio answers the call and establishes a media stream or voice connection.

2. Your agent process receives audio frames continuously rather than waiting for complete utterances.

3. The agent runs ASR, dialog logic, and TTS in a low-latency loop.

4. A realtime avatar session consumes the generated speech and produces synchronized talking video.

5. The audio path and video path stay aligned so the face “tracks” the voice rather than drifting out of sync.

This is the critical distinction from a recorded video or a canned “video loop”: the avatar has to be driven by the same realtime turn-taking loop that drives speech synthesis. If the call audio is low-latency but the video lags by 800 ms, the experience feels broken immediately.

Designing for latency, barge-in, and handoff

In a phone agent, latency is not just a UX issue; it changes the turn-taking protocol. A caller will interrupt if the pause is too long, and the system must decide whether to cancel pending speech, continue listening, or switch state.

Three implementation details matter most:

• Streaming ASR: do not wait for end-of-speech if your dialog manager can act on partial results.

• Barge-in handling: when the caller starts talking, immediately stop or duck the agent’s speech output and advance the state machine into listening mode.

• Escalation path: if confidence drops, or the user requests a human, preserve call context and transfer cleanly instead of restarting the conversation.

For a support use case, you typically also want a small structured state layer alongside the model: account verification, reason-for-call, system lookup, and transfer eligibility. The model handles language; the state machine handles policy.

Twilio + WebRTC: the integration boundary

Twilio is responsible for the telephony edge: PSTN connectivity, call control, and a media stream into your application. WebRTC is the realtime media transport inside your application stack. The cleanest way to think about it is that Twilio terminates the phone call, and your agent terminates the interactive media session.

That separation helps in a few ways:

• You can keep call logic in your own process instead of pushing everything into Twilio Studio flows.

• You can swap speech providers or agent runtimes without changing the PSTN entrypoint.

• You can attach a video surface to the same conversation without inventing a second control channel.

A minimal webhook pattern is usually enough to start: answer the call, hand Twilio a media stream target, and then connect your worker to the realtime session. Exact Twilio configuration depends on whether you use TwiML, Media Streams, or a SIP/WebRTC bridge, but the architectural principle is the same: keep the call signaling separate from the agent’s media loop.

from fastapi import FastAPI, Request

return Response(content=twiml, media_type="application/xml")

from fastapi import FastAPI, Request

return Response(content=twiml, media_type="application/xml")

from fastapi import FastAPI, Request

return Response(content=twiml, media_type="application/xml")

Once you have a streaming audio source, the rest of the agent can run like any other realtime voice application: buffer frames, feed ASR, generate the next response, and emit audio back into the call leg. The avatar is just another synchronized output of that same response cycle.

A practical agent loop: state, speech, and video

The implementation pattern I recommend is to keep one conversation state object per call and drive it from events rather than from synchronous function calls. That makes barge-in, tool calls, and escalation much easier to reason about.

A simplified loop looks like this:

1. Receive audio frames from the telephony bridge.

2. Append frames to the active conversation buffer.

3. Stream them into ASR and update partial transcripts.

4. When the agent decides to speak, synthesize audio and simultaneously update the avatar session.

5. If the caller interrupts, cancel generation and transition back to listening.

The important part is that the avatar should not be managed as a separate “video feature.” It should be a consumer of the same response event that produces audio. If the agent says “I can help with that,” the mouth movement and the audio onset need to line up closely enough that users perceive one coherent speaker.

# Illustrative only; exact SDK fields are in the docs

# Illustrative only; exact SDK fields are in the docs

# Illustrative only; exact SDK fields are in the docs

For teams already using a Python voice-agent stack, this pattern is usually easier to insert than a full visual front-end rewrite. You keep your existing orchestration, then add the realtime avatar as a synchronized output channel.

Where Protoface fits

This is the kind of integration docs.protoface.com is meant for: a developer-facing realtime avatar layer that can be attached to a live voice agent without turning your application into a video project. In practice, you either create sessions through the REST API or instantiate the avatar from a voice-agent runtime. For example, the LiveKit plugin quickstart pattern drops a synchronized face into the agent process so the avatar follows the same turn-taking loop as the audio.

import os

print(resp.json())

import os

print(resp.json())

import os

print(resp.json())

If your team already owns the voice stack, this is usually the least disruptive path: keep Twilio for telephony, keep your agent runtime for dialog, and add the avatar session only where it belongs — at the output boundary of the conversation.

Operational concerns you should not ignore

These systems fail in production for familiar reasons: rate spikes, provider timeouts, prompt drift, and bad escalation behavior. A few things are worth designing up front:

• Session lifecycle: start and stop avatar sessions with the call, and clean up aggressively on disconnect.

• Auth and secrets: keep API keys server-side; never expose them to the browser if you also offer a web fallback.

• Observability: log turn boundaries, barge-ins, transfers, and timeout causes so you can separate model issues from telephony issues.

• Quality tiering: match avatar quality to call type. A high-value sales line can justify more fidelity than an internal helpdesk line.

If you need a web version of the same experience, customer-managed iframe embeds are useful because they let you ship an avatar without exposing backend credentials in the browser. For phone replacement work, though, the more relevant pieces are the realtime session model and the agent integration surfaces.

Conclusion

A legacy IVR can be replaced with a realtime AI avatar phone agent if you treat the system as a streaming media problem, not a menu problem. Twilio handles the phone edge, WebRTC carries the realtime media loop, and your agent process owns the conversation state, escalation policy, and latency-sensitive turn taking. The avatar is the visible part of that pipeline, not a separate product.

If you are building this for a support or sales line, start with one narrow call type, keep the state machine small, and measure end-to-end latency before you optimize prompts. From there, use the public docs and quickstarts to wire the specific agent runtime you already have into a realtime avatar session. The docs at docs.protoface.com are the right place to start, and the GitHub examples are useful if you want a concrete integration path rather than a conceptual one.