How to Connect STT, TTS, and a Realtime Avatar in Electron for Interactive IVR Calls
Build a low-latency Electron IVR with streaming STT, TTS, and a realtime avatar synced to the same voice session.
Introduction
If you want an interactive IVR experience that feels like a real conversation instead of a phone tree, you need three things working in lockstep: speech-to-text (STT) for incoming audio, text-to-speech (TTS) for responses, and a realtime avatar that stays visually synchronized with the generated speech. The hard part is not any one component; it’s keeping the media pipeline low-latency enough that turn-taking still feels natural.
This post walks through the practical shape of that system in Electron. By the end, you should be able to wire a browser-based desktop app to a voice agent, stream audio both ways, and render a talking avatar that tracks the agent’s responses closely enough for live calls and IVR-style interactions.
System shape: what actually needs to happen
An interactive call path usually looks like this:
The user speaks into a microphone or a telephony stream.
STT converts inbound audio to text, ideally with partial hypotheses for fast turn detection.
Your agent decides what to say next, possibly using a realtime LLM or a deterministic IVR flow.
TTS synthesizes the response audio.
A video avatar consumes the same text or audio timing so its lip sync and facial motion stay aligned.
Electron plays the audio and renders the avatar video in the app window.
The important implementation detail is that STT, TTS, and avatar rendering should be treated as one streaming system, not three disconnected requests. If you wait for full-text transcription before starting the agent, or wait for complete TTS output before handing anything to the avatar, latency balloons and the interaction starts to feel robotic.
Electron as the host for media, not the brain
Electron is a good shell for this because it gives you a desktop UI, access to camera/mic permissions, and a local rendering surface for video. But you generally do not want Electron to own the speech logic. Keep the actual agent loop in a worker process or backend service, and let the Electron app do three jobs:
capture microphone audio and play synthesized audio,
render the avatar video element or WebRTC stream,
bridge UI events and call state into the agent runtime.
That separation makes the system easier to debug and lets you swap STT/TTS vendors without rewriting the desktop app. It also keeps your UI responsive when transcription or synthesis stalls.
Build the voice loop as a streaming pipeline
The cleanest way to think about the call loop is as a set of streams:
Inbound audio stream from the microphone or telephony source.
Transcription stream from STT, preferably with partial results.
Token or text stream from the agent.
Audio stream from TTS.
Avatar sync stream driven by the same utterance timing.
In practice, the agent should start generating as soon as it has enough context, and TTS should stream audio as soon as the first tokens are available. The avatar should be attached to the same conversational turn, not to a separate playback timer you invented in the UI. If the avatar is fed from the same session that produces the speech audio, you avoid a common failure mode where the face continues moving after the audio clipped or vice versa.
Minimal Electron wiring: media in, avatar out
The exact implementation depends on your speech stack, but the host app usually looks like this:
In the renderer, you’ll typically attach a microphone stream to your STT client and a WebRTC or media element to the avatar surface. The key is to keep the avatar update path separate from your DOM update path so video doesn’t get blocked by UI work.
For the speech side, a streaming interface is what you want, even if your underlying vendor hides it behind callbacks. Conceptually:
That arrow from agent to avatar is the part many teams underbuild. If the avatar session is only updated after the full response is spoken, you’re no longer doing realtime animation; you’re just playing back a prerecorded face.
Common implementation pitfalls
There are a few gotchas that show up quickly in IVR-style apps:
Blocking turn detection: waiting for end-of-utterance too aggressively makes the system feel slow. Use partial STT and a sane silence threshold.
Audio drift: if the avatar and speaker are fed from different clocks or different sessions, they will drift apart over long turns.
Echo and feedback: in a desktop app, make sure the agent does not transcribe its own TTS output unless you explicitly want that behavior.
State duplication: keep one source of truth for call state. Electron UI state should mirror backend session state, not replace it.
Recovery: if the avatar stream drops, your call should continue. Conversely, if speech fails, surface a clear degraded mode rather than freezing the UI.
For IVR calls specifically, another subtle issue is barge-in. If the user starts speaking while TTS is still playing, your app needs to decide whether to stop playback immediately or wait for a controlled interruption point. In a realtime system, this is usually better handled by the agent layer than by the UI.
Where Protoface fits
This is the part where Protoface is useful: it gives you the avatar surface without forcing you to build the lip-sync and session management layer yourself. For an Electron-based voice app, the most relevant integration pattern is usually to create a realtime avatar session from your backend or agent process, then stream that session into the UI as part of the call. The public docs at docs.protoface.com cover the exact session fields and lifecycle.
If you are already running a voice agent on LiveKit, the GitHub org has a plugin path that drops a synced video face into the agent. That matters because it keeps the avatar tied to the same conversational turn that drives speech, instead of bolting video onto the UI after the fact. The practical benefit is less glue code and fewer timing bugs.
For direct API access, the REST API is straightforward to call from a backend service with an API key. A minimal request usually looks like this shape:
Use the exact request schema from the docs; the point here is simply that the avatar session should be created server-side, then surfaced to Electron through your own app logic.
A practical architecture that holds up
A setup that works well in production is:
Electron renders the UI and avatar video surface.
A backend owns API keys, call state, and session creation.
STT streams partial transcripts back to the backend.
The agent produces response text incrementally.
TTS streams audio chunks to the user.
The avatar session consumes the same turn metadata so the face stays in sync.
This architecture keeps the desktop client thin. It also makes it easier to test the agent loop without opening the app, which is useful when you are debugging timing issues at 2 a.m. and do not want to restart Electron every time you change a prompt.
Conclusion
Connecting STT, TTS, and a realtime avatar in Electron is mostly an exercise in disciplined streaming design. Keep the agent loop off the UI thread, stream partial results instead of waiting for complete turns, and make sure the avatar is attached to the same conversation session as the speech output. That is what keeps the experience feeling like a live call instead of a series of disconnected API calls.
If you want to implement this with less custom media plumbing, start with the docs at docs.protoface.com and the quickstarts linked from the Protoface repository. The main thing is to prototype early with real latency budgets, because the difference between “works” and “feels conversational” is usually a few hundred milliseconds of unnecessary buffering.