Introduction

Adding a realtime avatar to an outbound sales agent looks straightforward until you try to run it at production latency. The hard parts are not “make a face appear” or “play a TTS clip”; they are synchronization, turn-taking, interruption handling, latency budgeting, and not accidentally building a system that is creepy, brittle, or expensive to operate.

If you’re building this kind of flow, the goal is not just a talking head. It’s a stable conversational loop: speech-to-text, agent reasoning, text-to-speech, lip sync, and video delivery all staying aligned while the user can interrupt at any time. By the end of this post, you should have a clear sense of what to avoid, what to measure, and where realtime avatar integrations usually fail in practice.

Do not treat the avatar as a separate media toy

The first mistake is architectural: wiring the avatar up as an afterthought. For outbound sales calls, the avatar is part of the agent loop, not a decorative layer. If the video face lags behind the audio, keeps “talking” after the agent has been interrupted, or resets awkwardly between turns, users immediately notice. In a call, those failures read as incompetence or automation.

Technically, the avatar pipeline needs to subscribe to the same conversational state as the voice agent. That means it must know when the agent is speaking, when the user barges in, and when the turn has really ended. If the audio stack supports interruption, the avatar must stop mouth motion and facial speech cues at the same moment the TTS stream is cut off. If you don’t do that, you get a face that keeps moving while the audio has already changed, which is one of the fastest ways to make the system feel broken.

Do not model the avatar as “play this prerecorded MP4 when the model speaks.” Realtime sales calls are full of partial utterances, retries, and mid-sentence corrections. You need a streaming video face whose output is tied to the current audio segment, not to a fixed asset.

Do not optimize for visual polish before latency and turn-taking

It’s easy to spend time on avatar quality tiers, background styling, and pose variety before the interaction loop is reliable. That’s backwards. In outbound sales, the user experience is dominated by latency and conversational timing, not by whether the avatar has subtle eyebrow motion.

At a minimum, budget for:

• LLM response latency

• TTS startup latency

• video generation or synthesis latency

• network transport latency

• jitter and buffering on the client

If any one of these is inconsistent, the call feels off. A good rule is to measure the time from user end-of-speech to first audible agent audio, then separately measure time to first avatar frame that visually matches that audio. Those two numbers should be close and stable. If the face starts late, users see a dead-air gap even if the audio is only slightly delayed. If the face starts early, it looks like lip sync is broken.

Also, do not let your agent accumulate turn-length drift. If the avatar can only display a generic talking state and you never explicitly stop it, the face can remain “in speech mode” between turns. Production systems need explicit state transitions: idle, listening, thinking, speaking, interrupted, and ended.

Do not ignore interruption, barge-in, and partial utterances

Outbound sales calls are not monologues. People interrupt constantly: to ask what company is calling, to say “not interested,” to request a callback, or simply to stop the agent mid-pitch. The correct behavior is not to finish the sentence politely; it is to stop cleanly and hand the turn back to the user.

That has a few implications:

1. Cancel audio generation immediately. If TTS has already buffered several seconds of speech, don’t let it drain just because it exists.

2. Stop avatar lip sync on the same event. Audio and video should share the same interruption signal.

3. Preserve the conversational state. If the agent was in the middle of a qualification flow, resume from the right logical point, not from the last visible phrase.

4. Handle partial transcripts carefully. A user cutting in after “I’m calling because—” is not the same as a completed statement.

This is where people often discover that their speech pipeline is correct in isolation but wrong end-to-end. The STT system may emit partials, the LLM may start generating, and the TTS system may already be speaking before your barge-in detector has fired. In that case, the avatar is faithfully animating a sentence the user never asked to hear. That is a design failure, not just a timing bug.

Do not expose credentials or ship a browser-side backend by accident

If you’re embedding a realtime avatar on a website, do not put secret API keys in the browser and do not invent a client-side proxy just to hide them. That pattern is fragile and usually wrong. For production, the browser should receive only scoped, short-lived access to the interaction surface it actually needs.

For sales demos and customer-facing experiences, the clean pattern is often an embedded session with explicit restrictions: allowed origins, per-session voice or instructions, and rate limits that keep abuse under control. That way, you can give a customer a face to interact with without handing them your full backend credentials. It also makes it easier to rotate keys and reason about abuse detection.

If you are using the REST API directly, keep the auth boundary on the server:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

The exact fields depend on the endpoint you use, but the pattern is the important part: create or manage the session server-side, then pass only the minimum needed token or embed configuration to the client.

Do not assume all integrations have the same transport and runtime constraints

Realtime avatar delivery can sit on top of different stacks: a voice agent framework, a direct API integration, or an embedded iframe. Those are not interchangeable from an engineering standpoint.

For example, if you are already using LiveKit Agents, the avatar should attach to the existing audio pipeline rather than bypass it. The point is to keep the agent’s voice and the avatar’s mouth synchronized at the transport layer, not to bolt on a second media system. That is what the LiveKit-oriented quickstart examples are useful for: they show the integration shape without forcing you to redesign your whole agent stack. If you prefer a Python integration path, the Python SDK is the right place to manage avatars and sessions programmatically.

One common mistake here is to assume the same error handling works everywhere. It does not. A WebRTC-based voice session may recover from brief packet loss differently than a server-rendered or iframe-embedded experience. You need to understand where the media is originating, where state is authoritative, and what happens when the connection recovers after a dropout. If your reconnection path recreates the avatar session every time, users may see a visible reset even when the underlying voice call is still alive.

Protoface is useful when the avatar is part of the agent loop

This is the point where a purpose-built avatar layer helps. Protoface is designed for realtime, developer-facing avatar integration rather than offline video generation. The relevant bit for outbound sales is that it can sit inside your agent flow as a synchronized talking face, so you keep one conversation state instead of gluing together a separate video system.

If you are wiring this into a voice agent, the LiveKit plugin path is the most direct way to keep audio and visual output aligned. If you are building your own orchestration, the REST API and Python SDK give you a server-side way to create avatars and sessions without leaking secrets to the browser. And if your use case is a web demo or customer-facing landing page, the iframe embed model avoids exposing API keys in the client at all. For implementation details, start with the documentation and the relevant quickstart or SDK repo for your stack.

A minimal Python flow looks like this conceptually:

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

The exact method and field names may differ, but that is the shape you want: create a session on the server, bind it to your agent, and let the media stack handle synchronized speech and facial animation.

What to measure before you ship

Before you launch outbound sales calls with a realtime avatar, instrument the parts that matter:

• First response time: end of user speech to first agent audio

• Avatar sync offset: difference between audio onset and visible mouth motion

• Interrupt latency: time to stop both voice and video after barge-in

• Session failure rate: reconnects, dropped media, and forced restarts

• Cost per minute: especially if you use different quality tiers

If you cannot measure those numbers, you cannot really improve the experience. And if you can measure them, you’ll usually find that the biggest wins come from reducing unnecessary state changes, avoiding extra hops, and making interruption semantics explicit.

Conclusion

The main thing not to do is treat a realtime avatar as a visual garnish. In an outbound sales system, the avatar is part of the conversation engine, which means it needs the same attention you’d give to audio transport or turn-taking logic. Keep the media synchronized, handle interruption cleanly, avoid browser-side secrets, and test the unhappy paths early.

If you want to implement this with a minimal amount of glue code, read the docs, start from the relevant integration repo, and build one narrow end-to-end flow before you generalize it. The fastest way to avoid production surprises is to make the first version small, measurable, and honest about latency.