Introduction

If you are adding a realtime avatar to a voice agent or web experience, the “face” is only half the problem. The practical question is how to customize that avatar so it matches the product context: different hair styles, outfits, skin tones, accessories, or brand-specific presentation, without breaking realtime behavior or introducing fragile client-side state.

This post focuses on the engineering side of that problem: how avatar appearance is typically modeled, where those settings should live, how to keep them consistent across sessions, and how to update them safely in a web app. By the end, you should have a clear mental model for implementing appearance customization in a way that fits realtime streaming constraints, works cleanly with voice agents, and keeps secrets out of the browser.

What “customization” actually means in a realtime avatar system

For a developer, customizing an avatar usually means changing a small set of stable, declarative attributes rather than “editing pixels” in the browser. In practice, that often includes:

• Face: identity, expression style, age range, lighting, camera framing, or a selected avatar model.

• Hair: length, color, style, parting, and whether the style should be conservative or more expressive.

• Outfit: shirt/jacket type, colors, formality level, and whether the look should fit a support agent, sales rep, game character, etc.

The important point is that these are not usually runtime CSS tweaks. They are configuration values that drive avatar generation or selection. In a realtime pipeline, the avatar must stay synchronized with speech, so the appearance profile has to be stable enough that the media layer can render consistently while audio and lip sync are changing frame by frame.

That has a few consequences:

1. Appearance settings should be versioned or at least treated as immutable inputs for a session.

2. Client apps should not derive appearance from ad hoc UI state alone; store the canonical config server-side.

3. If you want different looks for different tenants or experiences, map those experiences to preset profiles rather than letting users free-form everything.

Model appearance as a profile, not a one-off request

The cleanest pattern is to define an avatar appearance profile in your application database. Think of it as the payload you send to your avatar provider when creating or updating an avatar. Keep it small, explicit, and serializable:

{
}

{
}

{
}

You do not need a giant schema on day one. In fact, a small enum-based model is better because it is easier to validate and less likely to produce inconsistent renders across clients. If the backend only supports a fixed set of styles, map your internal names to the provider’s accepted values in one place.

A few implementation details matter:

• Validate on write: reject invalid combinations before they reach the avatar service.

• Normalize style values: use canonical identifiers such as short_side_part instead of free-text labels.

• Separate appearance from behavior: face/hair/outfit belong in the profile; voice, instructions, and tools belong elsewhere.

• Preserve session consistency: once a realtime session starts, its appearance should not change unless the platform explicitly supports that kind of update.

Update avatars through the backend, not the browser

Whether you expose customization in a web app or an admin dashboard, the browser should generally not talk directly to the avatar API with long-lived credentials. The exception is a managed embed flow designed for zero backend exposure. For everything else, keep the control plane on the server.

The basic pattern is:

1. User selects a face/hair/outfit preset in your app.

2. Your backend validates the selection and stores it.

3. Your backend calls the avatar API to create or update the avatar definition.

4. When a session starts, the backend binds the session to that avatar.

Here is a minimal example using the REST API shape you would expect for avatar creation. The exact field names are documented in the API reference, so treat this as illustrative rather than copy-paste final:

curl -X POST https://api.protoface.com/avatars \
}'

curl -X POST https://api.protoface.com/avatars \
}'

curl -X POST https://api.protoface.com/avatars \
}'

From an application-design standpoint, the key trade-off is whether you create one avatar per appearance preset or one avatar per end user. For most products, preset-based avatars are simpler:

• They reduce the number of managed objects.

• They make QA easier because the rendered look is deterministic.

• They avoid letting users create visually inconsistent or unsupported combinations.

If you do need user-specific avatars, still keep the customization surface constrained. Freeform face/hair/outfit selection sounds flexible, but the operational cost goes up quickly if you need to moderate, validate, and support every combination.

Realtime behavior: keep appearance separate from session state

Realtime avatars sit on top of streaming audio/video transport, typically WebRTC-based. That means your app is dealing with two timelines at once: a control timeline for avatar/session configuration, and a media timeline for audio frames, video frames, and lip-sync updates. Do not conflate them.

When a session starts, the avatar appearance should already be known. The streaming pipeline then uses that config while it renders the talking face in sync with the agent’s speech. If your app allows the user to switch looks mid-conversation, the safest design is usually to start a new session with the new avatar profile rather than mutating the existing one in place.

That avoids a class of bugs that are common in realtime apps:

• Visual state updates arriving after media state has already advanced.

• Client reconnections replaying stale customization state.

• Race conditions between “update avatar” and “start speaking” actions.

In a web app, I would treat appearance like any other immutable session input: compute it once, persist it, and attach it when initiating the avatar session. If you need a user-facing editor, let the editor change the stored profile, then explicitly relaunch or rebind the session.

Example: programmatic control from Python

If your backend is Python, using a dedicated SDK is usually cleaner than hand-rolling every request. A small server-side service can own avatar definitions, session creation, and any appearance updates. The following example shows the shape of that flow:

from protoface import Client

})

from protoface import Client

})

from protoface import Client

})

Again, treat the field names as illustrative unless you are following the exact SDK reference. The important part is the architectural shape: your app owns the customization logic, and the SDK is just the transport to the avatar service.

Where Protoface fits: server-managed avatars for web apps and voice agents

One practical place this matters is when you are embedding a realtime avatar into a site that already has a voice agent or conversational UI. With Protoface, you can manage avatar appearance server-side through the REST API or Python SDK, then attach that avatar to a realtime session. That lets you keep face, hair, and outfit configuration out of the browser while still giving product teams a straightforward way to expose customization in the app.

If you are building on the LiveKit stack, the same general idea applies through the LiveKit integration path: your agent handles audio, and the avatar layer renders the talking face in sync with speech. The main operational lesson is unchanged regardless of surface: define appearance on the backend, bind it to the session, and keep session state deterministic.

For implementation details, start with the docs at docs.protoface.com. If you want a working voice-agent setup that includes a face, the quickstart repositories linked from the docs are a better starting point than trying to wire everything manually on day one.

Practical gotchas when exposing customization in a web app

There are a few issues that come up repeatedly in production:

• Do not expose API keys in the browser: if your app needs direct control, route through your backend. Managed iframe embeds exist precisely for no-backend use cases.

• Make presets explicit: if the UI shows “casual,” “professional,” and “friendly,” map those to known avatar profiles rather than generating arbitrary config from labels.

• Keep rate limits in mind: if users can reconfigure avatars often, add debounce and server-side throttling so you are not creating sessions on every keystroke.

• Measure render stability: test hair/outfit combinations under the same lighting and camera framing you expect in production.

Also, do not assume appearance changes are cheap. Even if the control plane call is fast, a new session or model render may incur latency and cost. That is why quality tier and usage planning matter when avatar sessions are part of a larger product workflow.

Conclusion

Customizing a realtime avatar’s face, hair, and outfit is mainly a backend design problem. The right approach is to model appearance as a small, validated profile; keep it server-side; attach it to a session; and treat it as immutable for the lifetime of that session. That keeps your realtime pipeline predictable and avoids the class of bugs that show up when presentation state and media state drift apart.

If you are implementing this now, start with a narrow set of presets, wire the control plane through your backend, and test how new profiles behave under actual voice and video playback. From there, expand only as far as your product really needs. The API docs at docs.protoface.com are the right place to map this architecture onto the exact request and SDK fields.