What the OpenAI Realtime API Can and Can’t Control About Avatar Customization
OpenAI Realtime API controls conversation, timing, and tools—not avatar rendering, identity, policy, or lip-sync transport.
Introduction
When developers ask what the OpenAI Realtime API can “control” about an avatar, the most important answer is usually: not as much as they think. Realtime models are good at driving conversation state, turn-taking, speech content, and low-latency responses. They are not, by themselves, an avatar control plane.
If you are building a talking-face experience, the actual avatar system has to decide how a face is rendered, how audio is aligned to video, what expressions are allowed, what latency budget you can tolerate, and which session settings are mutable at runtime. That separation matters. The LLM can influence behavior through text, tool calls, and instructions, but it does not directly own the media pipeline.
By the end of this post, you should be able to distinguish model-level control from avatar-level control, know which settings belong in your app versus your avatar service, and avoid common integration mistakes when wiring a realtime voice agent to a live video face.
What the realtime model actually controls
The OpenAI Realtime API is best thought of as the conversational brain and event source. It can produce spoken responses, accept streaming input, and maintain a low-latency dialogue loop. In an avatar application, that means it can influence:
Textual behavior: what the agent says, style, tone, persona, and task-specific instructions.
Turn-taking: when the agent starts and stops speaking, interrupt handling, and conversational pacing.
Tool usage: if your agent is designed to emit structured actions or function calls.
Audio output timing: indirectly, because the avatar can only lip-sync once the speech stream exists.
What it does not control is the avatar renderer itself: camera framing, face identity, preset expression sets, video quality tier, session duration limits, embed authorization, or whether a particular avatar asset can be swapped at runtime. Those are platform concerns.
In practical terms, the model can say, “speak cheerfully and briefly,” but your avatar system decides whether “cheerfully” maps to a smile blendshape, a head-nod animation, or nothing at all. If you don’t define that mapping in your application, you will get a conversational agent, but not necessarily a well-behaved talking face.
Model instructions are not avatar configuration
A common failure mode is trying to pass avatar preferences through prompt text and expecting the renderer to infer them. That works only for behaviors the avatar service explicitly exposes as instruction-driven. For example, custom instructions can shape speaking style in a session, but they do not replace configuration that belongs to the media layer.
Think in layers:
Conversation layer: prompt, tools, turn-taking, and response generation.
Session layer: authenticated realtime session, time limits, voice selection, and per-session custom instructions.
Avatar layer: identity, asset selection, lip-sync rendering, quality tier, and allowed presentation options.
Transport layer: WebRTC or similar streaming path, audio/video timing, reconnect behavior, and browser delivery.
Only the first layer is meaningfully controlled by the model. The rest must be configured and enforced by your app or your avatar service.
What the model cannot safely control
There are several things you should not expect the realtime model to manage, even if it can mention them in a response:
Persistent avatar identity: who the avatar is, what it looks like, and which asset bundle it uses.
Browser trust boundaries: whether an end user can open the embed directly, or whether access must be origin-restricted.
API secrets: never expose bearer tokens or long-lived credentials in client-side code.
Operational limits: rate limits, session duration caps, and quality-tier billing constraints.
Media policy: whether a session may publish camera-like video, audio only, or a specific quality mode.
This is especially important for realtime avatar systems because the “agent” is often distributed across multiple services. The model generates language, but the avatar service has to remain authoritative about anything that affects safety, billing, or media transport.
Where control should live in a realtime avatar app
If you are designing the system from scratch, use the following rule: put deterministic control in code, not in prompts.
Examples:
Use application code to choose which avatar a user gets.
Use backend-authenticated session creation to attach custom instructions or a voice preset.
Use browser or server policy to decide who can open an embed.
Use the model prompt only for things that are genuinely semantic: tone, task, and dialogue constraints.
That separation makes your system easier to reason about and much easier to debug. If the avatar looks wrong, you inspect avatar/session config. If the agent says the wrong thing, you inspect prompt, tools, and conversation state. If the video freezes, you inspect the transport path.
Transport and synchronization are the real constraints
Realtime avatar UX is dominated by latency and synchronization. A face that is 250 ms ahead of the voice looks uncanny; a face that trails too far behind feels broken. That means the avatar renderer needs access to the same speech stream as the audio sink, or at least a tightly synchronized derived signal.
In practice, you are balancing three clocks:
Model latency: how long the assistant takes to decide what to say.
TTS or audio pipeline latency: how long it takes to generate speech.
Video/avatar latency: how quickly the face can animate to match that speech.
Once you see the problem this way, the question changes from “can the model control the avatar?” to “how do I keep the media path and the language path aligned?” The answer is mostly about session orchestration, not prompt engineering.
How Protoface fits into this
Protoface is the layer that turns a voice agent into a realtime talking face without making the model responsible for avatar rendering. In a LiveKit-based agent, the OpenAI Realtime quickstart shows the general pattern: the agent handles the conversation, while the avatar plugin handles synchronized video output. The point is not to “let the model control the face”; the point is to expose the right surface for each concern.
For a LiveKit agent, the integration is usually concise. The plugin inserts the avatar into the media pipeline so the agent’s speech has a synchronized video face:
That code is intentionally schematic. The exact parameters depend on the SDK and quickstart you are using, but the architectural point stays the same: the agent produces speech; the plugin renders the face; neither one pretends to own the other’s responsibilities.
If you need to create or manage avatars and sessions directly, use the REST API with backend-authenticated requests. That is the right place for deterministic state like “create session,” “attach instructions,” or “start an avatar instance”:
And if you are building a web embed, the safest path is a customer-managed iframe. That keeps API keys off the client entirely, while still letting you scope access with parent-origin allowlists and per-embed session rules. In other words, the browser gets a constrained UI surface, not the authority to mint or manage avatars.
Practical gotchas
There are a few recurring mistakes worth calling out:
Assuming prompt text can change rendering. It usually cannot. If it matters visually, configure it explicitly.
Putting secrets in the browser. Do not hand out API keys to client-side code; use a backend or a managed embed.
Ignoring sync drift. Test interruptions, long responses, and network jitter; those are where lip-sync problems show up.
Mixing policy with persona. “Be formal” belongs in instructions; “only accessible from this origin” belongs in embed/session policy.
If you are operating at scale, also watch usage billing and quality tier selection. Video quality is not just a UX choice; it affects cost and can affect latency, so treat it as part of your product envelope, not as a cosmetic toggle.
Conclusion
The OpenAI Realtime API can drive a conversation, but it cannot be treated as the control plane for avatar customization. It can shape speech, timing, and dialogue behavior; it cannot safely own avatar identity, session policy, browser trust, or media transport. Those responsibilities belong to your app and to the avatar platform you wire in.
If you keep that boundary clear, integration gets much simpler: use the model for language, use code for policy, and use an avatar service for synchronized rendering. For the exact session fields, embed options, and SDK usage patterns, see docs.protoface.com. If you want to see the realtime agent pattern end to end, the quickstarts linked from the repository are the fastest way to get oriented.