Introduction

If you are building a voice agent, the hard part is no longer “can I generate text?” It is making the interaction feel coherent: the avatar should look like the product, speak in the right voice, and behave with a persona that matches the use case. That means coordinating three layers at once: presentation, audio generation, and conversational policy.

This post shows the practical shape of that problem and how to wire it up with Protoface. By the end, you should understand how to customize an avatar’s visual appearance, select or switch voices, and inject persona/instructional context so the realtime conversation stays consistent across turns. The examples use the OpenAI Realtime API as the model-facing layer, then show where Protoface fits as the avatar layer.

Start with the architecture: model, voice, avatar

In a realtime avatar stack, the LLM is not directly “the avatar.” The avatar is a rendering and synchronization problem sitting on top of the conversational loop.

A typical flow looks like this:

1. The user speaks or types into your client.

2. The agent receives audio or text and streams tokens from the realtime model.

3. Text is turned into speech by the TTS component, or the model emits audio directly if your stack supports it.

4. The avatar layer consumes that audio and visual state, then lip-syncs the face video to the speech stream.

The important detail is that “voice” and “persona” are mostly model-side concerns, while “look” is avatar-side. If you mix those up, you end up trying to solve layout, rendering, and prompt engineering in the same place.

Customizing the look: treat it as a session-level visual contract

The avatar’s look should be selected before the conversation starts and then held stable for the session. In practice, that means choosing a face asset or avatar configuration that represents the brand or character, and passing that configuration when you create the realtime session.

For developers, the useful mental model is:

• Avatar identity: what face is rendered.

• Avatar state: how that face behaves during the session, including speaking, idle, and transition states.

• Session constraints: duration, allowed origin, rate limits, and any per-session customization.

Keep visual customization coarse-grained. If you need a different look for different products or brands, create different avatar configs rather than trying to mutate a single avatar on every turn. That makes session behavior deterministic and reduces the chance of visual mismatch after reconnects.

For a website embed or a browser-based experience, it is especially important that the avatar configuration is bound to the session, not the client. That avoids exposing secrets and prevents a user from swapping faces in the browser by editing JS state.

Customizing the voice: pick a voice that fits the interaction, not just the brand

Voice selection matters more than most teams expect. A voice that sounds impressive in isolation can be exhausting in a support flow, and a friendly support voice can sound untrustworthy in a sales motion. In a realtime agent, the right voice is the one that stays intelligible under streaming latency and does not fight the visual timing of the avatar.

With the OpenAI Realtime API, you generally configure the voice and behavioral instructions as part of the session setup. The exact fields vary by API shape and SDK version, but the intent is consistent: establish the model’s speaking style before the conversation begins.

A minimal session configuration usually includes:

• model selection

• voice choice

• system or developer instructions

• output modality or audio settings

Keep the voice stable for a given avatar unless you have a good product reason to change it. Sudden voice changes during a live conversation are jarring and can break the perceived continuity of the agent.

Customizing the persona: use instructions, not ad hoc prompt fragments

The persona is the part developers often under-specify. A good realtime agent needs more than “be helpful.” It needs operational rules: what it can promise, how it should respond to uncertainty, when to ask clarifying questions, and how terse it should be under load.

Write the persona as a concise instruction block that is stable across the session. For example:

{
}

{
}

{
}

That is the right level of specificity for a realtime avatar. It gives the model a behavioral boundary without forcing a scripted script. In practice, you want the persona to control tone and policy, while the conversation history controls content.

A few useful rules:

• Keep instructions short enough to be remembered across long sessions.

• Put hard constraints first: safety, policy, and unsupported actions.

• Make style guidance secondary: tone, verbosity, and pacing.

• Avoid contradictory instructions across the agent stack.

OpenAI Realtime API: session setup and streaming considerations

The realtime API is useful here because it can maintain low-latency turn-taking while the conversation is still unfolding. That is especially important when the output is both audio and lip-synced video, because any delay between model response and animation becomes visible.

There are a few practical implications:

• Token streaming affects perceived personality. Short, early tokens make the agent feel responsive.

• Audio timing matters more than raw model latency. A 200 ms delay can be acceptable if it is consistent; an irregular 200 ms delay looks broken.

• Interruptions must be handled cleanly. If the user talks over the agent, stop playback and reset the avatar state immediately.

If you are assembling the stack yourself, you usually keep the model connection and the avatar connection logically separate: the model emits response text/audio, and the avatar client subscribes to the audio stream and state updates. That separation makes it easier to swap models or voices later without rewriting the visual layer.

Illustrative session initialization might look like this, with fields simplified to show the pattern rather than the exact schema:

{
}

{
}

{
}

If your application uses text-to-speech outside the model, the same principle applies: the model generates the content, your TTS service generates the waveform, and the avatar consumes the waveform. The critical part is preserving alignment between text timing and audio timing so the face looks like it is actually speaking the generated words.

How Protoface fits: a realtime avatar layer on top of your agent

For teams using LiveKit-based voice agents, the simplest integration path is the LiveKit plugin published as livekit-plugins-protoface. It drops a synchronized talking video face into the existing agent pipeline, so you can keep your model, transport, and audio logic where they already are and add the avatar as a separate surface. If you are using Pipecat instead, the plugin and guide at this integration reference show the equivalent pattern.

The main operational advantage is that the avatar session becomes a managed artifact, not a pile of client-side glue. You can create avatars and realtime sessions through the REST API, authenticate with API keys, and manage them from the dashboard or from the Python SDK. For browser embeds, the customer-managed iframe model is useful because the API key never touches the client, and you can constrain parent origins and session limits centrally.

A simple REST-style session creation call is enough to show the shape:

curl -X POST https://api.protoface.com/v1/realtime-sessions \
}'

curl -X POST https://api.protoface.com/v1/realtime-sessions \
}'

curl -X POST https://api.protoface.com/v1/realtime-sessions \
}'

The exact request fields and response shape are documented in the docs, but the important thing is the separation of concerns: create the avatar once, create sessions per interaction, and keep persona/voice/session policies attached to the session rather than hardcoding them in the browser.

Python and LiveKit examples: keep the integration thin

If you prefer to wire things up from Python, the SDK is the easiest place to start because it keeps the session lifecycle explicit. You can create or fetch avatar records, then open a realtime session with the look, voice, and instructions you want for that run.

from protoface import Client

print(session.url)

from protoface import Client

print(session.url)

from protoface import Client

print(session.url)

For LiveKit, the plugin approach is similarly thin: you keep your existing voice agent and add the avatar transport. The key operational question is whether you want the avatar to mirror the agent exactly or whether you need a distinct persona layer for the visual agent. In most production systems, matching those two layers is the safest default.

One practical gotcha: don’t let the avatar layer become stateful in ways the model does not know about. If the avatar is showing a “thinking” or “listening” state, make sure that state is derived from the conversation pipeline, not from a separate timer. Otherwise lip-sync and turn-taking drift apart.

Trade-offs and gotchas

Realtime avatars are a synchronization problem first and a rendering problem second. The common failure modes are not visual fidelity issues; they are consistency issues.

• Voice drift: the model’s response style does not match the selected voice.

• Persona drift: the instructions are too vague, so the agent changes tone between turns.

• Timing drift: audio starts before the avatar state changes, or vice versa.

• Security mistakes: API keys or privileged session creation logic leak into the browser.

Design for the narrowest possible client surface. The browser should render the avatar and forward user input. The server should own session creation, policy, and billing-sensitive operations. That separation is especially important if you are embedding the avatar on third-party pages or shipping it as part of a customer-facing product.

Conclusion

To customize a realtime avatar well, think in three layers: the look is an avatar configuration, the voice is a model or TTS choice, and the persona is a stable instruction set. Keep those responsibilities separate, create sessions explicitly, and let the transport handle low-latency streaming while the avatar layer handles synchronized presentation.

If you want to implement this in production, start by reading the documentation, then choose the integration path that matches your stack: REST API for explicit session control, the Python SDK for backend orchestration, or the LiveKit plugin for voice agents that already exist. If you want the fastest way to validate the pattern, use one of the quickstarts linked from the Protoface repository and adapt the avatar, voice, and persona settings from there.