Introduction

When you add an avatar to an AI app, one of the first architecture decisions is where customization lives. Do you let the browser mutate the avatar’s appearance and behavior directly, or do you keep avatar state on the server and push only the minimum necessary data to the client?

That choice matters more for avatars than for ordinary UI state because realtime video faces are usually coupled to streaming audio, lip-sync timing, session identity, and model-driven behavior. In practice, the wrong split can lead to inconsistent renders across devices, insecure exposure of keys or instructions, and hard-to-debug latency spikes.

This post compares client-side and server-side avatar customization in an Agora-based AI app. By the end, you should be able to decide which parts of customization belong in the browser, which should stay on the backend, and how to wire the result into a realtime voice agent without turning your app into a state synchronization problem.

What “avatar customization” actually means in a realtime app

For a talking avatar, customization is broader than a profile picture. It usually includes:

• Visual identity: face style, skin tone, hair, clothing, framing, background.

• Behavioral identity: voice selection, speaking style, custom instructions, personality constraints.

• Session-scoped state: language, tenant, user profile, conversation context, moderation policy.

• Presentation state: camera crop, layout, overlays, and whether the avatar is active, listening, or silent.

In a realtime system, some of this state is static per avatar, some is dynamic per session, and some should be ephemeral per frame or per utterance. The implementation detail that matters is not just where the state is stored, but who is allowed to modify it and how quickly those changes need to propagate.

If you are building on Agora for transport and media, the same design question still applies: your audio path may be low-latency and stream-oriented, but customization metadata is usually control-plane data. Treating those as separate concerns avoids coupling UI changes to the media pipeline.

Client-side customization: fast iteration, weaker trust boundaries

Client-side customization means the browser decides what the avatar should look like or how it should be parameterized, then sends those choices to the app backend or directly to a realtime session initializer. This is attractive because it is easy to prototype and gives the user immediate feedback.

Typical examples:

• A UI slider changes the avatar’s appearance before the session starts.

• A dropdown swaps between preconfigured voice/personality presets.

• The page reads local user settings and applies them to the avatar embed.

The upside is obvious: low friction and fast UI loops. The downside is that the browser is not a trust boundary. Anything sent from the browser can be inspected, modified, replayed, or scripted. If customization includes per-tenant constraints, paid feature gates, or hidden system instructions, client-side control becomes a liability.

There is also a practical reliability issue. The browser is a poor source of truth for anything that must survive refreshes, reconnects, or multiple tabs. If the avatar session is created from client state and the page rehydrates inconsistently, you can end up with a UI that claims one configuration while the live session is running another.

Client-side customization works best when the data is:

• Purely presentational.

• Non-sensitive.

• Disposable if the page reloads.

• Safe to expose to the end user.

That usually means color palettes, layout preferences, or selecting from a small set of public presets. It does not mean API keys, hidden voice instructions, tenant-scoped rules, or session policies.

Server-side customization: better control, better auditability

Server-side customization means the backend decides the canonical avatar configuration and emits a session token, signed request, or session record that the realtime layer consumes. The browser may still choose from allowed options, but the backend validates and persists the actual result.

This is the right default for most production systems. The reasons are straightforward:

• Security: secrets and privileged instructions stay out of the browser.

• Consistency: every client sees the same authoritative avatar state.

• Auditability: you can log who changed what and when.

• Policy enforcement: billing tier, moderation, and tenant rules are enforced centrally.

In a realtime avatar system, server-side control is especially important for anything that affects session behavior. Think of the backend as the place where you decide:

1. Which avatar a given user is allowed to use.

2. Which voice/instructions are allowed for the current tenant.

3. What session duration and rate limits apply.

4. Whether a customization request should create a new session or update an existing one.

The operational pattern is simple: the client submits an intent, the server validates it, and the server either stores it or forwards it to the realtime provider. This keeps the media path lean and the control path explicit.

Here is the rough shape of the backend-first flow with an HTTP API:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

The exact fields and session schema are documented in the API docs, but the pattern is the important part: the browser never sees the secret that authorizes the session.

Where the split usually lands in practice

A useful mental model is to divide customization into three buckets:

• Client-owned: harmless presentation toggles, local preview settings, transient UI state.

• Server-owned: avatar identity, session policy, instructions, keys, entitlements.

• Negotiated: user-selectable options that the server validates against a whitelist.

For example, let users pick from three approved avatars in the browser, but have the server map that selection to a vetted avatar ID and create the session. That gives you a responsive UI without making the browser authoritative.

Another practical boundary is persistence. If you expect a customization choice to matter after refresh, after reconnect, or across devices, persist it server-side. If it only affects the current page render, keep it local.

Finally, separate customization from media streaming. Audio and video may traverse Agora or another transport with tight latency constraints, but the user’s selected avatar style, personality, or compliance settings should be resolved before the stream starts, or through a controlled server-side update path. Avoid trying to “patch” the media stream with client-side mutations that actually belong in session metadata.

How this looks in a LiveKit-style voice agent integration

If you are using a voice-agent framework, the cleanest implementation is usually to keep the agent and its avatar synchronized in the server process. For a LiveKit-based agent, the plugin repository includes a drop-in integration that attaches a realtime talking face to the agent. That means the agent’s turn-taking, speech output, and avatar animation can remain coordinated without pushing avatar logic into the browser.

A minimal Python sketch looks like this:

from livekit.agents import WorkerOptions, cli

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

from livekit.agents import WorkerOptions, cli

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

from livekit.agents import WorkerOptions, cli

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

The point is not the exact API surface; it is the deployment shape. The avatar is controlled from the same trusted backend process that already owns the agent session, which is usually the right place for policy, observability, and retry logic. If you are integrating with a Pipecat pipeline instead, the same principle applies; the integration guide in the docs and the repository examples show how to keep avatar state in the server-side media graph rather than the client.

How Protoface fits without forcing a bad architecture

Protoface supports both models, but it is most useful when you keep the sensitive parts server-side. The REST API and Python SDK are the obvious fit when you want your backend to create avatars and sessions, validate per-tenant policy, and pass only a short-lived session handle or embed URL to the browser. That keeps API keys and custom instructions off the client and gives you a single source of truth for the avatar lifecycle.

For web embeds, the iframe path is even stricter: the browser hosts an interactive avatar without ever seeing an API key, while the parent-origin allowlist and per-embed limits keep control where it belongs. That is a good default if you want a customer-facing web experience and do not need the browser to directly manipulate session internals.

If you want to review the integration surface or compare quickstarts, start with the docs at https://docs.protoface.com and the relevant examples in the GitHub organization. The main design takeaway is unchanged: let the browser choose among safe options, but let the server own the authoritative avatar state.

Conclusion

For realtime avatars, client-side customization is best for low-risk, ephemeral UI preferences. Server-side customization is the safer and more scalable choice for anything that affects session identity, voice, instructions, authorization, or billing. In an Agora-based AI app, that usually means the media path stays realtime and optimized, while the control plane stays server-authoritative.

If you are implementing this now, start by drawing a hard line between presentation state and session state. Put secrets, policies, and avatar identity on the backend. Keep browser state limited to safe choices and transient previews. Then wire those decisions into your agent or embed flow once, rather than trying to reconcile them after the session is already live.

For implementation details, SDK usage, and current examples, check the documentation at docs.protoface.com.