How to Add Avatar Skin Tone, Hairstyle, and Clothing Options to a LiveKit Voice Agent
Learn to add skin tone, hairstyle, and clothing options to a LiveKit voice agent with validated avatar session config.
Introduction
If you’re adding a talking avatar to a voice agent, the first version usually works with a generic face and a fixed look. That’s enough to prove the pipeline, but it’s rarely enough for production. In practice, teams want the avatar to match a user profile, a brand character, or a specific conversational context: different skin tones, hairstyles, clothing, and sometimes a few constrained variants for A/B testing or audience segmentation.
This post covers the practical part: how to model avatar appearance options, how to expose them cleanly in your application, and how those choices flow through a realtime voice-agent stack without breaking synchronization between audio and video. By the end, you should be able to decide where appearance data belongs, how to pass it into a session, and how to keep the implementation maintainable as you add more variants.
What “avatar customization” really means in a realtime agent
For a voice agent with a live video face, appearance is not just a static UI setting. It is part of the runtime state that influences how the avatar session is created and rendered. The important distinction is:
Identity: who the avatar is supposed to represent.
Appearance: skin tone, hairstyle, clothing, and other visual attributes.
Session state: the active realtime instance that is speaking, listening, and animating.
You generally want to keep identity and appearance as structured inputs to your backend, then translate them into whatever your avatar service expects at session creation time. That way, your app can treat “finance advisor with navy blazer and short black hair” as a record, not as a pile of hard-coded frontend props.
In a realtime system, the avatar is usually driven by streamed audio from your agent and rendered as a synchronized video face. The visual variant must be selected before or at session start so the generated face stays consistent across the whole conversation. If you try to swap appearance mid-session, you need to be careful: not every rendering stack supports seamless morphing, and abrupt changes can look like a cut rather than a continuous avatar.
Model appearance as data, not UI state
The cleanest implementation is to define a small appearance schema in your application and store it alongside the avatar profile. Keep the schema narrow and explicit. For example:
That schema should be validated on the backend, not trusted from the browser. The frontend can render a selector, but the backend should decide what is allowed for a given tenant, user tier, or brand package.
A few implementation details matter here:
Use enumerations, not free text. “Blazer” is better than “dark business jacket with subtle lapels,” because downstream systems need stable values.
Separate canonical IDs from display labels. Your app can show “Short curly hair” while sending
short_curly.Keep appearance options versioned. If you later rename or retire a style, existing sessions should still resolve deterministically.
Support fallbacks. If a requested combination is unsupported, choose a safe default rather than failing the entire session.
A simple backend mapping layer is often enough:
That keeps your app code stable even if the avatar provider later expands the underlying catalog.
Where to validate combinations and enforce product rules
Not every combination should be exposed to every user. In a production product, the rules are usually business-driven:
Some hair styles might only be available on certain quality tiers.
Some clothing variants might be reserved for specific brands or roles.
Skin tone choices may need to be broad and inclusive, but still constrained to a fixed supported set so the rendering pipeline remains predictable.
Do the validation in the backend service that creates or configures the avatar session. That gives you one place to enforce policy, log the chosen variant, and reject unsupported combinations before a session is opened.
In a LiveKit-based voice agent, this usually means your agent setup phase decides the avatar configuration, then creates the voice session with that configuration attached. The agent runtime should not need to know about your product rules; it only needs the resulting, validated avatar settings.
From an operational perspective, the most common mistake is to let the browser decide appearance and then forward that straight to the agent. That makes it easy to spoof unsupported variants and hard to audit what actually launched. Treat avatar appearance the same way you’d treat pricing tier or authorization scope: resolve it server-side.
Passing the configuration into a LiveKit voice agent
Once you have a validated appearance object, the next step is wiring it into your voice agent flow. The exact integration depends on your stack, but the pattern is consistent: initialize the realtime avatar session with a structured payload, then let the agent stream audio and receive synchronized video output.
If you are using the LiveKit plugin, this is typically the least invasive path because it drops the avatar into an existing LiveKit Agents setup. For a developer-facing reference implementation, see the [plugin repo](https://github.com/protoface-ai/protoface-plugin-pipecat). The same conceptual model applies even if your application uses a different agent framework: create the avatar session with the selected appearance, then connect it to the audio stream.
Here is an illustrative Python example showing the shape of the data you might pass into session creation. The exact field names depend on the SDK or API version, so treat this as a pattern, not copy-paste code:
The important part is the separation of concerns: avatar creation describes what should be rendered; session creation binds that avatar to the live agent and its audio stream.
If you are operating closer to the transport layer, the same idea shows up as a REST call. For example, a session creation request may look like this:
The response shape and exact field names are defined in the docs, but the workflow is the same: authenticate with an API key, create or reuse an avatar, then start a realtime session with the desired appearance attached. For the authoritative parameter list, use the documentation.
Synchronization, latency, and why appearance changes are a session concern
Realtime avatars live and die on synchronization. The agent is streaming or generating speech, the avatar renderer is lip-syncing, and the client is receiving video in near real time. Appearance should therefore be treated as a mostly static property of the session. That avoids two classes of problems:
Temporal inconsistency: the audio stream starts with one look and the video switches to another mid-utterance.
Cache churn: every appearance change forces the rendering pipeline to reconfigure assets or models.
There are valid cases for swapping appearance during a conversation, but if you need that, make it an explicit transition rather than an implicit update. For example, a role-play app might start a generic onboarding avatar and later switch to a branded product specialist. In that case, end the current session cleanly and start a new one with the new visual configuration.
Also consider how you persist choices. If users can pick an avatar style from a web app, store the selected identifiers on your backend and replay them when reconnecting or restoring a session. Don’t rely on client state alone; reconnects, refreshes, and multi-tab usage will eventually expose that assumption.
When Protoface fits this problem
This is exactly the sort of workflow Protoface is built for: a developer-facing realtime avatar API that lets you add a synchronized talking face to a voice agent without hand-rolling the rendering stack. In practice, you can keep your appearance options in your app, validate them server-side, and then pass the resulting avatar/session configuration into the LiveKit plugin or the REST API.
If you want to wire this up quickly, start from the public docs and the quickstarts in the GitHub org, then adapt the avatar configuration step to your own appearance schema. The main thing to preserve is the boundary: your product owns the allowed skin tone, hairstyle, and clothing options; the avatar service owns realtime rendering and synchronization.
Practical pitfalls to avoid
Three mistakes show up repeatedly:
Overloading the frontend: letting the browser decide which visual variants exist instead of asking the backend.
Using ad hoc strings: storing “blue suit v2” in one place and “navy_blazer” in another.
Mixing appearance with runtime state: treating visual style as if it were a live, mutable UI prop instead of a session attribute.
If you avoid those, the implementation stays straightforward. The backend resolves a valid appearance, the agent session starts with that configuration, and the video face stays consistent while the voice agent talks.
Conclusion
Adding skin tone, hairstyle, and clothing options to a live voice agent is mostly a data-modeling and session-management problem. Define a small, validated appearance schema; resolve it server-side; attach it to the avatar session; and keep the runtime avatar stable for the duration of the conversation.
If you are using LiveKit, the cleanest path is to wire the configuration into your agent startup flow and let the avatar plugin handle the synchronized video face. For exact API fields, session lifecycle details, and supported avatar options, check docs.protoface.com. If you want a working starting point, the GitHub quickstarts are the fastest way to see the pieces connected end to end.