Implementing an AI Avatar Front Door for Customer Support Calls with WebRTC and WebSocket
Build an AI avatar front door for support calls with WebRTC media, WebSocket signaling, turn-state sync, and low-latency barge-in handling.
Introduction
If you are building a customer support voice agent, the core problem is usually not speech recognition or LLM orchestration. It is the first few seconds of the call: you need to answer quickly, establish a presence, and keep the interaction coherent while audio, text, and video all stay in sync. An avatar “front door” is a practical pattern for that. The caller speaks to a voice agent, and the agent is rendered as a realtime talking face that lip-syncs to the generated speech.
This post walks through the architecture behind that pattern using WebRTC for realtime media and WebSocket-style signaling for coordination. By the end, you should know how to think about the transport split, how to wire avatar state to agent state, where latency typically comes from, and how to avoid the common mistakes that make these systems feel janky.
Start with the transport model: WebRTC for media, WebSocket for control
The first design decision is to separate media from control. WebRTC is a good fit for the audio and video streams because it handles jitter buffers, packet loss, adaptive codecs, and NAT traversal. That is what you want once a conversation starts. A WebSocket channel is better for low-volume signaling: session setup, status updates, text transcripts, turn state, cancellation, interruption, and avatar control messages.
In practice, the architecture looks like this:
The browser or support app opens a WebRTC connection for the live media path.
A WebSocket channel carries session metadata and realtime events.
Your voice agent produces audio and state transitions.
The avatar renderer consumes those events and emits synchronized video frames.
The key point is that the video face should not be treated like a static overlay. It is part of the conversation state machine. If the agent is speaking, the avatar needs mouth motion aligned to the generated audio. If the agent is interrupted, the video state should react immediately. If the agent is listening, the face should look idle but alive.
Design the conversation loop around turn state
Support calls are mostly about turn-taking. You are continuously shifting between: listening, transcribing, thinking, speaking, and sometimes barge-in recovery. The avatar should mirror that loop closely enough that callers can infer what the system is doing without being told.
A robust loop usually has these states:
Idle: connection open, avatar visible, agent waiting.
Listening: user is speaking, speech detection active.
Thinking: ASR finished, agent is generating a response.
Speaking: TTS audio is streaming, avatar is lip-syncing.
Interrupted: user barged in, stop generation, reset pose.
From an implementation standpoint, do not tie avatar updates directly to raw model tokens. That tends to create visual noise and uneven pacing. Instead, anchor the face to a few meaningful events:
speech start and stop
response generation start and cancel
audio playback start and end
interruption detected
This keeps the video coherent even when the backend is doing multiple things at once. If your voice stack supports streaming TTS, the avatar should begin movement only when audio is actually available to send downstream, not when the first token appears.
Latency matters more than perfect fidelity
For support calls, a slightly less expressive avatar that responds fast will usually outperform a more detailed one that lags. The user experience is dominated by perceived responsiveness. A few practical rules help:
Keep the first visible response under a second if possible.
Stream text and audio rather than buffering whole responses.
Use a transport path that survives browser network changes without requiring a full reload.
Avoid unnecessary transcoding between agent, audio service, and avatar renderer.
One subtle gotcha is synchronization drift. Audio can be emitted from one service, while video timing is generated from another. If they are not driven by the same speaking state, the mouth can begin moving too early, continue too long, or “snap shut” before the utterance finishes. The fix is to treat the spoken audio as the source of truth for speaking duration and use your realtime channel to announce those boundaries.
Another practical issue is barge-in. In a support setting, callers interrupt constantly. When that happens, you need to stop the current generation, stop any queued audio, and immediately transition the avatar out of speaking mode. If you only cancel the text generation but let the audio continue, the face will keep talking after the user has already taken the floor.
How to wire it into a voice agent
If you already have a realtime voice agent, the easiest implementation is to add the avatar at the agent layer rather than in the UI. That way the same speaking state that drives audio also drives the face. In Python, the basic pattern is to initialize the avatar session, attach it to the agent, and pass through the events that your voice stack already emits.
In an agent runtime, you would then bind that session to the speaking lifecycle. The important bit is not the exact method names here; it is the shape of the integration: create session, connect media, forward agent state, and cleanly tear down on hangup.
If you are working in a LiveKit-based stack, the cleanest option is usually to add the avatar as an agent plugin so the voice agent gains a synchronized talking face without re-architecting the call path. The quickstarts are useful here because they show the agent-side loop in a minimal form, and the plugin package is available on PyPI for the LiveKit path. The point is not to make the avatar a separate service you manually babysit; it should be another media participant attached to the same conversation lifecycle as the agent.
Signaling and browser integration with WebSocket
On the browser side, you need a place to receive state transitions that are not media. That is where the WebSocket channel fits. Typical messages include session ready, avatar speaking, transcription partials, error states, and end-of-call cleanup. The browser can then update indicators, transcript panels, or fallback UI without touching the media path.
Two design constraints matter here:
Do not let the browser become authoritative for conversation state. The backend should decide whether the agent is speaking, listening, or interrupted.
Keep the signaling messages idempotent. Reconnects happen. If the browser reconnects and receives the current session state again, it should be able to recover cleanly.
When you are debugging sync bugs, it helps to log state transitions, not just timestamps. A timeline like “user started speaking → agent canceled → avatar reset → transcript finalized” is much more actionable than a pile of raw audio events.
Where Protoface fits
This is exactly the kind of integration Protoface is built for: take an existing realtime voice agent and give it a synchronized video face without inventing your own avatar renderer and media pipeline. The developer surfaces are straightforward: a REST API for session creation and management, a Python SDK for programmatic control, and a LiveKit plugin path if your agent already lives there.
For a support-call front door, the most useful property is that you can keep the avatar lifecycle aligned with your voice agent lifecycle instead of bolting on a separate frontend animation system. You create a session, connect it to the agent, and let the conversation drive the face. If you are evaluating the integration path, start with the documentation and the relevant examples in the GitHub organization.
A minimal REST call looks like this in shape, even though the exact fields depend on your avatar setup:
The response typically gives you the session material you need to attach the avatar to the realtime call path. From there, your job is mostly lifecycle management: create, connect, monitor, and dispose. That is a much smaller problem than building and operating a custom avatar stack.
Conclusion
An AI avatar front door is really a realtime systems problem: keep media on WebRTC, keep coordination on WebSocket, drive the avatar from the same state machine as the voice agent, and optimize for low-latency turn-taking over perfect animation detail. If you build it that way, the avatar feels like part of the call rather than a separate widget.
If you want to implement this in your stack, start with a narrow prototype: one voice agent, one avatar session, one browser client, and explicit turn-state logging. Then expand once the latency and interruption behavior look correct. The docs at docs.protoface.com are the right place to map the concepts here onto the exact API and SDK details.