Introduction

If you are adding a realtime avatar to a voice agent, the hard part is usually not the avatar rendering itself. It is keeping three streams in sync: the model’s text, the synthesized audio, and the visual mouth/face animation. With Vapi, that gets even more interesting because the agent is already handling turn-taking, interruption, and telephony or browser audio. The goal of a multi-voice setup is to make that system feel deliberate: the right speaker voice for the right conversational role, with a single avatar that tracks the active speaker cleanly.

This post walks through the integration pattern I recommend when pairing Vapi with a realtime avatar service: how to think about voice assignment, how to route speech so the avatar stays synchronized, and where to put the glue code so the setup stays maintainable. By the end, you should be able to wire a multi-voice agent into a realtime avatar, avoid common timing mistakes, and know where the backend boundaries should live.

What “multi-voice” actually means in practice

In most agent stacks, “multi-voice” does not mean multiple avatars talking at once. It usually means the agent can speak with different synthetic voices depending on role, intent, or conversation state. For example:

• a neutral default voice for normal replies

• a warmer voice for support or onboarding

• a distinct voice for system prompts, confirmations, or sub-agents

For a realtime avatar, the important constraint is that one speaking stream should map to one visible face at a time. If you mix voices without coordinating the playback boundary, lip sync will drift or the avatar will “snap” between visemes mid-utterance. The visual layer should follow the same audio stream the user hears, not a separately reconstructed approximation.

That means the practical integration question is: where do you choose the voice, and where do you emit the resulting audio so the avatar can stay in lockstep? In a Vapi-based stack, the right place is usually the agent orchestration layer, not the browser. The browser should receive already-decided media events, not infer them.

Architect the conversation around a single audio authority

The cleanest pattern is to let your voice agent own the turn-taking and voice selection logic, then attach the avatar to the same output path the user hears. The avatar should not independently decide when to speak. It should consume the agent’s audio output as the source of truth.

That gives you a few practical benefits:

• Deterministic lip sync — the avatar follows the exact audio track used for speech.

• Better interruption handling — when the agent is cut off, both audio and face stop together.

• Cleaner state management — the avatar is a rendering concern, not a second conversational brain.

For multi-voice routing, a simple model is enough:

1. Vapi produces the next assistant turn.

2. Your orchestration layer chooses the voice profile for that turn.

3. The selected TTS output is streamed to the avatar session.

4. The avatar renders the speaking face from the same stream.

In other words, if the voice changes, it should change at turn boundaries whenever possible. Mid-turn voice switching is technically possible, but it is rarely worth the visual artifacts unless you are intentionally doing character work or multi-speaker roleplay.

Keep the avatar session and the speech stream tightly coupled

Realtime avatars are sensitive to latency and discontinuities. If you create the avatar session too late, the first few tokens may render without a face. If you buffer audio too aggressively, the avatar will appear to react slowly. If you attempt to “reconstruct” speech from transcripts after the fact, lip sync will look off even when the words are correct.

For that reason, create the avatar session before the conversation starts and keep it alive for the duration of the exchange. Then stream the agent’s audio directly into that session. The session boundary should roughly match the lifetime of the call or browser interaction.

There are a few implementation details that matter:

• Use the final audio form — the avatar should receive the same synthesized audio that would otherwise be played to the user.

• Preserve timing metadata if available — any TTS timing or chunk boundaries help the renderer stay aligned.

• Handle barge-in as a first-class event — stop avatar playback immediately when the agent is interrupted.

• Do not fan out independently — one stream in, one avatar out.

If your Vapi configuration can emit conversation events or allows you to hook into the assistant’s audio lifecycle, that is where you connect your avatar pipeline. Treat the avatar as a downstream consumer of the agent’s speech, not a peer.

Practical voice selection patterns for multi-voice agents

There are a few voice-selection strategies that work well in real systems:

• Per role: each assistant persona gets a fixed voice. Good for support + sales + concierge flows.

• Per state: the same agent uses different voices for greeting, troubleshooting, escalation, or closing.

• Per tenant: each customer configures a brand voice while the avatar stays constant.

The main trade-off is consistency versus expressiveness. Per-role voices are easier for users to recognize and make debugging simpler. Per-state voices give you more control over tone but can feel unnatural if you switch too often. Per-tenant voices are operationally nice if you are building a platform, because the voice choice becomes part of deployment config rather than application code.

A good rule: keep the avatar visual style stable and let the voice vary sparingly. The more frequently you change voices, the more carefully you should test turn transitions and interruption behavior.

Example: route a selected voice into the avatar session

The exact fields depend on your stack, but the integration shape is usually straightforward. Below is a Python sketch showing how you might create an avatar session and then send the agent’s chosen speech stream into it. The SDK method names and payload fields may differ, so treat this as a pattern rather than copy-paste code.

from protoface import Client

)

from protoface import Client

)

from protoface import Client

)

If you are using a Vapi workflow, the important part is not the specific SDK call. It is that the same voice decision made for the assistant turn is used to create the audio that reaches the avatar. Keep that mapping explicit in your code, ideally in one place, so voice policy does not leak into UI components or prompt handlers.

Where the Protoface integration fits

This is the point where Protoface fits cleanly into the architecture: create and manage the avatar session through the REST API or Python SDK, then feed it the audio from your voice agent. The API is authenticated with bearer keys, so keep it server-side. A minimal REST request to create or manage sessions looks like this:

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"avt_123","voice":"alloy"}'

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"avt_123","voice":"alloy"}'

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"avt_123","voice":"alloy"}'

If you want the avatar embedded in a product surface without exposing backend credentials, the customer-managed iframe flow is the safer option. But for a Vapi integration specifically, you generally want the backend to own the session and the avatar to follow the agent’s outbound audio path. That keeps the turn logic, voice selection, and media state in one control plane.

If you are using LiveKit as part of your voice stack, there is also a plugin surface for dropping a Protoface avatar into the agent pipeline, which is useful when you already have a LiveKit-based media graph. For Vapi, the same principle applies: one authoritative assistant audio stream, one synchronized avatar session.

Gotchas that matter in production

A few problems show up repeatedly in real deployments:

• Voice changes mid-utterance: avoid this unless you want audible and visible artifacts.

• Double playback: do not play the same synthesized audio locally and also route it into the avatar unless you intentionally need both paths.

• Session churn: creating a new avatar session on every turn adds latency and increases failure modes.

• Unbounded browser trust: never expose API keys to the client if the browser is initiating the interaction.

• Interrupted turns: make sure stop/cancel events propagate to both speech and avatar rendering immediately.

Testing should include at least three scenarios: normal back-and-forth, user barge-in during a long answer, and rapid voice changes between turns. If the avatar remains visually stable in those cases, you have probably wired the pipeline correctly.

Operational guidance for teams shipping this

From an engineering standpoint, the safest implementation is to centralize the following responsibilities in one backend service:

• Vapi agent configuration and turn policy

• voice selection logic

• avatar session lifecycle

• audio stream forwarding

• cleanup on disconnect, timeout, or cancellation

That service can be thin. It does not need to render video or understand low-level avatar animation. It just needs to ensure that the same conversation event that triggers speech also drives the avatar session. If you keep that boundary crisp, you can swap voices, models, or even providers later without redesigning the whole stack.

Conclusion

A Vapi multi-voice setup works best when you treat the agent’s audio as the source of truth and make the avatar a synchronized downstream consumer. Pick voices at turn boundaries, keep session state server-side, propagate interruptions immediately, and avoid duplicating media paths. That gives you predictable lip sync and makes the system much easier to reason about under load.

If you want implementation details for sessions, authentication, or SDK usage, start with the docs and the quickstarts linked from the project repository. The main thing is to keep the architecture simple: one conversation, one authoritative speech stream, one avatar that follows it.