Introduction

If you are replacing an IVR with a realtime voice agent, the awkward part is not speech recognition or text-to-speech. It is the interface layer: users need a signal that the system is listening, thinking, and responding in a turn-based conversation. A plain voice-only bot can work, but it is easy to make it feel opaque and brittle. A synchronized talking face fixes a lot of that by making state visible: speaking, listening, latency, interruptions, and turn transitions all become easier to understand.

That is the problem this post addresses. By the end, you should be able to think clearly about the realtime pipeline for an AI voice avatar, understand where the video face fits in a voice agent architecture, and wire one into a Go-based IVR replacement without turning your backend into a streaming science project.

Start with the right architecture: audio is primary, video is derived

For IVR replacement, the voice agent remains the source of truth. The avatar should not drive the conversation; it should reflect it. That distinction matters because the lowest-latency path is typically:

caller audio → streaming ASR → LLM / dialog policy → streaming TTS → audio playback

The avatar sits alongside that pipeline and receives enough timing information to lip-sync the rendered face to the synthesized speech. In practice, you want the avatar to track the same turn boundaries and audio timeline as the TTS engine, not some separately inferred transcript. If the face lags behind the audio by even a few hundred milliseconds, the illusion breaks quickly.

There are a few implementation details worth getting right:

• Use streaming end-to-end. Chunked ASR, incremental LLM responses, and streaming TTS reduce perceived latency and make barge-in possible.

• Keep turn state explicit. The agent should know whether it is listening, thinking, speaking, or interrupted.

• Drive the avatar from audio events. Lip sync and facial animation should follow the actual audio frames or timestamps produced by TTS.

• Handle interruptions cleanly. If the caller talks over the agent, stop the current speech, cancel downstream generation, and reset the avatar state immediately.

If you have only built voice agents before, the main mental shift is that the avatar is not a separate conversational system. It is a realtime renderer attached to the same turn-taking state machine.

Build the call pipeline in Go without hiding the realtime parts

Go is a good fit for the orchestration layer because it handles concurrency well and makes the call state machine straightforward to reason about. In a typical IVR replacement, you will have at least three concurrent streams:

1. Incoming user audio from telephony or WebRTC.

2. Outgoing synthesized speech back to the caller.

3. Avatar session updates for the rendered face.

The important thing is not to over-engineer the boundary between them. Keep the agent logic deterministic where you can, and isolate network IO behind a few small components. A simplified Go layout looks like this:

type TurnState struct {

}

type TurnState struct {

}

type TurnState struct {

}

That code is intentionally minimal, because the real value is in how you structure the transitions:

• When audio starts from the caller, mark the agent as listening.

• When you receive a stable transcript segment, decide whether to continue listening or begin a response.

• When TTS starts, mark speaking and start the avatar session or speech event.

• When the utterance finishes, return to listening.

A common mistake is to wait for the full assistant response before starting TTS. That guarantees extra latency and makes the avatar feel detached. Prefer incremental generation and chunked synthesis where your speech stack supports it.

How the avatar stays synchronized

Realtime lip sync usually depends on one of two models. In the first, you pass the generated audio to the avatar system and it infers mouth movement from the stream. In the second, you pass both audio and metadata about timing or utterance boundaries. The second model is generally more reliable for voice agents because the renderer can align animation state with the exact speech timeline.

There are also a few operational concerns that are easy to miss:

• Clock skew matters. If audio is produced on one side and rendered on another, timestamps should be authoritative and in one direction.

• Jitter buffers are necessary. Real networks do not deliver smooth packets. Small buffers improve continuity, but too much buffering adds visible delay.

• Utterance boundaries are not transcript boundaries. The user hearing silence is not always the same as the agent being done speaking.

• Fallback behavior matters. If the avatar stream fails, the voice agent should continue operating without video rather than failing the entire session.

For IVR replacement, the face is best treated as an attachment to the agent session. If your telephony path is healthy and your avatar path is degraded, the conversation should still complete. That separation keeps video from becoming a single point of failure.

Using Protoface where the avatar belongs: the session boundary

This is where Protoface fits naturally. The platform is designed to add a synchronized talking face to an existing realtime agent rather than forcing you to rebuild your voice stack around video. For Go-based systems, the most common pattern is to keep your call logic in Go, then create or manage the avatar session through the API or SDK and attach it to the agent lifecycle.

If you want to do this manually, the REST API gives you explicit control over avatars and sessions. Authentication uses an API key in a bearer header, and the docs describe the exact payloads. A minimal session creation request looks like this:

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

The precise fields will depend on the session type and current API shape, so treat that as illustrative and check the docs before wiring it into production. The useful part is the boundary: create the avatar session when your agent starts, update it as the conversation context changes, and close it when the call ends.

If you prefer not to call the REST API directly from Go, the Python SDK is useful for scripting avatar/session lifecycle operations, and the developer dashboard is handy for inspecting session state and tuning behavior in the browser. For implementation details, use the public docs at docs.protoface.com.

Practical integration choices and gotchas

There are a few decisions that will save you time in production:

• Separate session management from conversation state. The agent can reconnect or restart without losing the avatar model of what is happening.

• Do not expose API keys to the browser. If you are building a web-based demo, use a customer-managed iframe embed rather than calling the backend from client-side code.

• Keep the avatar instructions short. Long prompts do not make the face better; they usually just make the behavior less predictable.

• Pick a quality tier intentionally. If your use case is support triage, you probably do not need the same render quality as a high-touch sales experience.

For a Go service, the cleanest pattern is usually: one goroutine for telephony/WebRTC input, one for the agent loop, one for the avatar/session control plane, and a small state machine that coordinates cancellation. That keeps the latency-sensitive path easy to profile.

If you are using LiveKit Agents, the avatar piece can be added as a plugin so the voice agent gets a synchronized talking video face with very little glue code. The integration examples in the plugin repository are the quickest way to see the expected lifecycle and event flow: https://github.com/protoface-ai/protoface-plugin-pipecat. If your stack is Pipecat-based, the integration guide is here: https://docs.pipecat.ai/api-reference/server/services/video/protoface.

Conclusion

Building a realtime AI voice avatar for IVR replacement is mostly an exercise in preserving timing. Keep the voice agent as the source of truth, treat the avatar as a synchronized rendering layer, and structure your backend so the session can fail independently without taking the call down.

If you are starting from scratch, begin with a small end-to-end path: one live call, one streaming agent loop, one avatar session, and clear state transitions for listening, speaking, and interruption. Then expand into routing, handoff, and recovery logic once the basic timing feels solid.

For the implementation details, docs, and examples, start at docs.protoface.com and the quickstarts linked from the Protoface GitHub organization.