Introduction

If you have ever tried replacing an IVR with a “real” conversational agent, you already know the hard part is not the LLM. The hard part is making the interaction feel live: low-latency audio, turn-taking, interruption handling, and a visual presence that does not look bolted on after the fact.

This post walks through a practical architecture for building a realtime AI avatar IVR replacement in Electron using WebRTC for media transport and Python for the backend agent/control plane. By the end, you should be able to wire up a desktop app that:

• captures microphone input and plays back synthesized audio in realtime,

• streams a synchronized talking avatar into an Electron window,

• maintains a clean separation between UI, media, and agent logic, and

• supports the same underlying agent for desktop, web, or call-center workflows.

The important framing here: the avatar is not the agent. The avatar is a realtime media surface that reflects the agent’s speech and state. If you keep that distinction clear, the system stays debuggable.

What an IVR replacement actually needs

Traditional IVRs fail mostly because they are slow and rigid. A useful replacement needs to behave like a streaming system, not a request/response API.

At a minimum, the stack needs four concurrent loops:

1. Audio ingress: microphone or telephony audio is chunked and sent upstream with low buffering.

2. Agent reasoning: ASR, dialog state, and LLM/tool logic run continuously, not after the user finishes an entire turn.

3. Audio egress: synthesized speech is streamed back as soon as it is ready.

4. Visual synchronization: the avatar’s mouth motion tracks the outgoing speech, and the UI reflects speaking/listening state with minimal delay.

In practice, WebRTC is the right transport for this kind of loop because it gives you low-latency bidirectional media, congestion handling, and a well-understood client model in Electron. Python is a good fit on the agent side because the orchestration layer usually involves SDKs, tool calls, and server-side session management rather than UI concerns.

The failure mode to avoid is treating the avatar like a video file. You want a live media session driven by the agent’s speech timing, not a pre-rendered animation.

Electron as the client shell

Electron is a reasonable choice when you want a desktop wrapper around a realtime agent UI without building separate native apps. The browser runtime gives you WebRTC support, Web Audio, and a flexible rendering layer for controls, transcripts, and state.

A clean Electron architecture usually looks like this:

• Main process: app lifecycle, privileged networking, token exchange if needed.

• Renderer process: WebRTC session UI, avatar view, transcripts, controls.

• Backend Python service: session creation, auth, agent orchestration, optional business logic.

The renderer should not know your long-lived API secrets. If it needs to connect to a realtime session, it should receive a short-lived session token or connection payload from your backend. Keep the Python service between the UI and any privileged API access.

One useful implementation detail: treat the avatar as just another remote media track in the renderer. That makes it easier to reconnect, show network state, and swap visual surfaces without changing agent logic.

WebRTC and the synchronization problem

WebRTC solves the transport, but not the application-level synchronization. A lip-synced avatar only looks correct if the visual stream is paced by the same speech timeline the client hears.

There are a few practical rules:

• Do not buffer too aggressively. Extra buffering increases stability at the cost of conversational latency, and users notice that immediately.

• Keep one authoritative speaking state. The agent should be the source of truth for when the avatar is speaking, when it is listening, and when barge-in is allowed.

• Handle interruption explicitly. If the user speaks over the agent, you need to stop or truncate synthesis, update the avatar state, and resume listening quickly.

• Separate transport from semantics. Network jitter might delay packets, but the dialog state should still reflect “agent is mid-turn” until the agent actually stops.

For desktop apps, the WebRTC peer connection usually lives in the renderer. That lets you attach incoming tracks directly to video elements and render the avatar as a normal video surface. If your backend also streams synthesized audio, make sure the audio and video are produced from the same session state so they stay aligned.

On the agent side, the core loop is usually:

mic audio -> ASR -> dialog state / LLM / tools -> TTS -> streamed audio + avatar sync

mic audio -> ASR -> dialog state / LLM / tools -> TTS -> streamed audio + avatar sync

mic audio -> ASR -> dialog state / LLM / tools -> TTS -> streamed audio + avatar sync

You want to avoid a design where the LLM is waiting for a full transcript before speaking. Partial hypotheses, endpointing, and streaming TTS are what make the experience feel interactive.

Python backend: create and manage sessions

The Python service is where you create sessions, generate short-lived credentials, and coordinate with your agent runtime. The exact session fields depend on the API surface, but the shape is straightforward: authenticate with your API key, create or look up an avatar, then start a realtime session bound to the frontend client.

Here is a minimal example using the Python SDK pattern. Adjust names and fields to match the current docs.

from protoface import Client

print(session.url)

from protoface import Client

print(session.url)

from protoface import Client

print(session.url)

If you prefer raw HTTP for debugging, the REST API is equally useful because it makes the network boundary explicit:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

In production, the backend should mint whatever ephemeral connection data the renderer needs, then discard it. That keeps your API key out of the client and gives you a clean place to enforce session policies, quotas, and per-user access control.

It also gives you a good place to log the things that matter operationally: session duration, reconnects, turn latency, and interruption rate. Those metrics are usually more useful than raw token counts when you are tuning conversational UX.

Where Protoface fits

This is the point where a developer-facing avatar layer matters. Protoface gives you a realtime avatar session surface that plugs into the agent stack without forcing you to invent your own lip-sync, media sync, or avatar session management.

For a voice-agent architecture, the most relevant integration is the LiveKit Agents plugin. If your backend already runs a LiveKit-based agent, the Protoface plugin for Pipecat is another practical route, and the corresponding Pipecat guide is here: docs. In both cases, the idea is the same: keep the agent logic in Python, and drop in a realtime avatar layer so speech and face stay synchronized.

A typical plugin-style integration looks roughly like this:

from livekit.plugins import protoface

)

from livekit.plugins import protoface

)

from livekit.plugins import protoface

)

The useful part is not the exact constructor shape; it is the division of responsibilities. Your agent continues handling ASR, tool calls, and turn-taking. The avatar layer handles presentation and sync. That separation makes it much easier to swap voices, change visual styles, or run the same conversational core in a different client.

Gotchas that matter in practice

There are a few issues that show up quickly once you put real users in front of the system:

• Latency budget creep: adding browser work, extra websocket hops, or unnecessary buffering can quietly push you from “realtime” to “responsive-ish.” Measure end-to-end turn latency early.

• State desynchronization: if the UI thinks the agent is speaking but the media track has already stopped, users will assume the system is broken.

• Interruptions: barge-in is not a nice-to-have. If the user cannot interrupt, the experience feels like a polished IVR, not a conversation.

• Retry semantics: reconnects happen. Design your session lifecycle so the renderer can reattach without restarting the whole agent.

• Security boundaries: never ship long-lived API keys in Electron. If you need browser-accessible integration, prefer an iframe model or mint ephemeral session data server-side.

It is also worth deciding early whether your “avatar” is a product feature or an implementation detail. If it is a feature, the UI should surface presence and intent clearly. If it is just a face for an agent, keep the visual layer understated so it does not distract from the conversation.

Conclusion

The clean mental model is simple: WebRTC moves realtime media, Python orchestrates the agent, and the avatar is a synchronized presentation layer attached to the speech stream. Electron gives you a practical desktop shell for that stack, but the design principles are the same whether you are shipping to desktop, web, or a call-center operator console.

If you build it this way, you get a system that is debuggable, reconnectable, and much easier to evolve than a monolithic “AI video call” prototype. Start by wiring up the media path, then add turn-taking and interruption handling, then layer the avatar on top once the conversation loop is stable.

For implementation details, session/auth behavior, and current SDK shapes, check the docs at docs.protoface.com. If you want a working starting point, the quickstarts in the repository linked from the docs are the fastest way to validate the end-to-end path before you customize the agent.