Introduction

Classic phone trees fail for the same reason most IVRs fail: they force callers to translate a goal into a menu path, and the menu path is usually wrong. If you want a better front door for a support line, internal help desk, or sales line, the simplest upgrade is not “more menu options.” It is a conversational layer that can listen, interpret intent, ask clarifying questions, and respond naturally with speech.

This post shows how to build that layer in Flask using speech-to-text for inbound audio, a language model or dialog policy for intent handling, and text-to-speech for the response. By the end, you should have a clear architecture for replacing DTMF menus with a low-latency, voice-driven loop that can route calls, collect information, and keep state across turns.

What changes when you replace a phone tree with conversation

A traditional IVR is a deterministic state machine driven by keypad input. That makes it easy to reason about, but brittle in practice:

• Callers phrase requests in their own words, not your menu taxonomy.

• Nested menus amplify abandonment when a caller chooses the wrong branch.

• Static prompts cannot adapt to context, account state, or prior turns.

A conversational system changes the control flow. Instead of “press 1 for billing,” you accept free-form speech, convert it to text, classify the intent, and generate the next action. In practice, the conversation loop looks like this:

1. Telephony provider streams audio into your backend.

2. Your backend runs speech-to-text on short chunks or a full utterance.

3. You decide whether to answer, gather more information, or transfer.

4. You synthesize a spoken response and stream it back to the caller.

The important engineering detail is that this is still a state machine, just a more flexible one. You should treat each turn as an explicit state transition, not as an unbounded chat log with no operational guardrails.

Flask as the orchestration layer

Flask is a reasonable choice when the web app already exists and you want to add voice behavior without moving to a separate real-time stack immediately. The Flask app should not do the heavy audio work in request handlers; it should orchestrate session state, dispatch work to STT/TTS services, and return the next action quickly enough to keep the call responsive.

A useful minimal separation is:

• Ingress: a webhook or media-stream callback from your telephony provider.

• Session store: per-call state, pending task, and conversation context.

• Turn handler: takes recognized text and returns structured next-step data.

• Speech layer: turns the response into audio.

Do not bury business logic inside audio callbacks. Keep your intent routing and domain decisions in a plain Python function so you can test it without telephony, STT, or TTS in the loop.

Core conversational loop: audio in, text out, audio back

At a high level, the loop is straightforward:

caller audio -> STT -> text -> intent/slot logic -> response text -> TTS -> audio

caller audio -> STT -> text -> intent/slot logic -> response text -> TTS -> audio

caller audio -> STT -> text -> intent/slot logic -> response text -> TTS -> audio

The implementation details matter more than the diagram.

Speech-to-text. For telephony audio, expect narrow bandwidth and imperfect recognition. If your STT provider can stream partial transcripts, use them for responsiveness, but only commit to a state transition on final text. If the provider only returns final utterances, that is fine; you just trade some latency for simplicity.

Intent handling. You do not need a full agent for every flow. For many support lines, a compact intent map is enough:

def route_intent(text: str) -> dict:
return {"intent": "fallback", "next": "clarify"}

def route_intent(text: str) -> dict:
return {"intent": "fallback", "next": "clarify"}

def route_intent(text: str) -> dict:
return {"intent": "fallback", "next": "clarify"}

Text-to-speech. TTS should be fast enough that the caller does not hear dead air after every turn. If your provider supports streaming synthesis, use it. If not, pre-generate short prompt variants for common paths and fall back to dynamic synthesis for the long tail.

Also be careful with barge-in: if a caller starts speaking while the system is still playing audio, your media pipeline should be able to stop playback and listen again. That is one of the main differences between a good conversational system and a frustrating one.

A practical Flask shape

Below is a simplified example of a Flask endpoint that receives recognized text, advances state, and returns the next prompt. This is intentionally not tied to any specific telephony or STT vendor.

from flask import Flask, request, jsonify

})

from flask import Flask, request, jsonify

})

from flask import Flask, request, jsonify

})

That handler should stay small. The real work belongs in handle_turn(), which can be tested with table-driven cases:

def handle_turn(state, transcript):

return {"step": "clarify"}, "Can you say that another way?", {"type": "listen"}

def handle_turn(state, transcript):

return {"step": "clarify"}, "Can you say that another way?", {"type": "listen"}

def handle_turn(state, transcript):

return {"step": "clarify"}, "Can you say that another way?", {"type": "listen"}

Once the logic is structured this way, swapping STT/TTS providers or introducing an LLM for a subset of turns becomes an implementation detail rather than a redesign.

Latency, turn-taking, and failure modes

Voice UX is extremely sensitive to delays. In a phone tree, a one-second pause is annoying; in a conversational flow, it can feel broken. A few practical constraints help keep it usable:

• Keep prompts short. Long system messages increase time-to-first-audio and make barge-in harder.

• Use timeouts deliberately. If you do not detect speech after a prompt, decide quickly whether to reprompt or escalate.

• Persist state every turn. Calls drop, webhooks retry, and media sessions reconnect.

• Normalize transcripts. “I want to pay my bill” and “billing” should land in the same branch.

The most common failure mode is over-automation: the system tries to answer everything with generated speech and never commits to a concrete action. For call handling, that is usually the wrong trade-off. The better pattern is to keep the dialog narrow and decisive: identify intent, collect missing slots, then either complete the task or transfer to a human with context.

Another failure mode is mixing streaming media concerns with business logic. If your Flask code is also parsing audio frames, building prompts, calling an LLM, and writing to your database, debugging will be painful. Split those responsibilities early.

Where Protoface fits

If your conversational flow should also show a live talking face on a website or inside a real-time agent, that is where Protoface is useful. Rather than treating the avatar as a separate frontend concern, you can attach a synchronized video face to the voice experience so the user sees lip-synced speech while the backend keeps doing the same STT/intent/TTS work.

For a Flask-based system, the relevant integration point is usually not the web app itself but the real-time agent or embed surface you connect to it. The LiveKit plugin is the cleanest fit if your voice stack already uses LiveKit agents; the Python SDK and REST API are useful if you want to create avatars or manage sessions programmatically from your backend. The exact request fields and session objects are documented in the public docs.

from protoface import Client

)

from protoface import Client

)

from protoface import Client

)

If you are wiring Protoface into a LiveKit voice agent, the plugin is published on PyPI and the examples in the repository are the fastest way to see the integration pattern in context: pipecat-protoface and the plugin repo. If you are using Pipecat, the integration guide is also directly relevant: Pipecat Protoface guide.

Operational notes that matter in production

There are a few details worth calling out before you ship:

• Authentication: keep API keys server-side. For direct backend calls, use standard bearer auth over HTTPS.

• Rate limits: protect the voice endpoint separately from your normal HTTP traffic. Voice retries can create bursts.

• Observability: log the transcript, intent, state transition, and latency for each turn. Do not log raw audio unless you have a strong reason.

• Fallback paths: always provide a clean human transfer or callback path when confidence is low or the user is stuck.

A useful mental model is that your Flask app is the control plane and the media pipeline is the data plane. Keep them loosely coupled so you can replace STT, TTS, or the avatar layer without rewriting the routing logic.

Conclusion

Replacing a phone tree with a conversational flow is mostly an exercise in disciplined state management: recognize speech, classify intent, advance a small state machine, and speak back quickly. Flask is perfectly adequate as the orchestration layer as long as you keep audio handling and business logic separate.

If you want the voice experience to have a synchronized face as well, Protoface fits naturally alongside the agent stack rather than inside the Flask request cycle. For implementation details, examples, and API specifics, start with the public docs at docs.protoface.com, then wire the smallest possible end-to-end flow before adding richer dialog logic.