Introduction

If you are building a voice agent, a support bot, or a game NPC, the text-to-speech part is only half the product. Users understand and trust the interaction faster when the agent has a synchronized face: lip motion, gaze, and a visible “speaking” presence that updates in real time. The hard part is not generating a video file; it is keeping the avatar aligned with the live audio stream, with low latency and predictable failure modes.

This post shows how to wire a conversational AI avatar into a Django application using LiveKit as the realtime transport. By the end, you should understand the moving parts: how audio and video are synchronized, where Django fits, what you need to manage server-side, and where Protoface fits into the stack without exposing secrets in the browser.

What “realtime avatar” actually means

In practice, a conversational avatar is a live media pipeline:

• The user speaks into a microphone or types a message.

• Your agent processes the input, decides on a response, and streams audio back.

• A video face is driven by the same response timing so the mouth motion stays aligned with speech.

• The client receives both tracks over a low-latency transport, typically WebRTC through LiveKit.

The key engineering constraint is that the video cannot be treated as a separate asset. If speech arrives in chunks, the avatar has to follow those chunks closely enough that the mouth shape, pauses, and turn-taking feel natural. That means the avatar service must live inside the same realtime session as the voice agent, or at least consume the same timing signals.

For Django developers, that usually means Django does not stream media itself. It handles control plane concerns: authentication, session creation, issuing tokens or session metadata, persisting avatar configuration, and exposing endpoints your frontend or agent runtime can call. The actual media path stays on LiveKit/WebRTC.

Django as the control plane

Keep your Django app focused on orchestration. A clean split is:

1. Django authenticates the user and decides whether they can start a session.

2. Your backend calls the avatar/session API to create a realtime session.

3. Django returns the session details to the frontend or agent runtime.

4. The client joins the LiveKit room and subscribes to the audio/video tracks.

This avoids putting API keys in the browser and keeps your session logic auditable. It also lets you enforce app-specific rules such as tenant limits, per-user quotas, and conversation metadata.

A minimal Django view that creates a session might look like this:

import os

return JsonResponse(resp.json())

import os

return JsonResponse(resp.json())

import os

return JsonResponse(resp.json())

The exact request fields depend on your avatar and session model; the point is that Django should call the REST API server-side and return only the non-secret session data needed by the client or agent runtime.

Using LiveKit for the media path

LiveKit is a good fit here because it already solves the hard realtime bits: room management, WebRTC transport, track subscription, NAT traversal, and low-latency media delivery. Your avatar becomes just another participant or track source in the room.

In a typical voice-agent setup, your agent runtime produces audio and may also receive transcripts or events. The avatar service listens to that audio stream and generates a synchronized talking face. The browser then renders the video track alongside the agent’s audio. From the app’s perspective, you are not “playing a video”; you are subscribing to a live track whose frames are generated in response to speech.

That distinction matters for debugging:

• If the audio is delayed, the avatar will look delayed.

• If the transcript arrives but audio is clipped, lip sync will drift.

• If network jitter increases, you may see the avatar hold a pose briefly and then catch up.

When things go wrong, inspect the media timeline before blaming the model. A lot of “bad avatar quality” is really transport latency, packet loss, or bad turn segmentation.

Plugin-based integration with a LiveKit voice agent

If you already have a LiveKit agent, the cleanest path is to drop the avatar into the agent runtime with the LiveKit plugin. The plugin is designed to attach a Protoface avatar to the agent so the agent’s speech is mirrored by a synchronized talking video face. This keeps the integration close to the source of truth: the runtime that already owns the agent’s audio.

At a high level, the setup looks like this:

from livekit.agents import AgentSession

session.run()

from livekit.agents import AgentSession

session.run()

from livekit.agents import AgentSession

session.run()

Use this as a shape, not copy-paste production code. The plugin’s actual constructors and event hooks are documented in the repo and the package docs. The important idea is that the plugin handles the avatar sidecar behavior so your agent code stays focused on turn-taking, tool use, and response generation.

If you are using a different voice-agent stack, the same pattern still applies: create a session server-side, join the media room from the runtime that emits speech, and ensure the avatar service receives the same timing and interruption signals as the spoken response.

How to think about session lifecycle and latency

The operational details are where most integrations become fragile. A few rules help:

• Create sessions just-in-time. Do not keep stale avatar sessions around unless you need them.

• Separate avatar configuration from session state. The avatar definition changes less often than the live conversation.

• Expire aggressively. Realtime sessions should have a bounded lifetime and server-side cleanup.

• Measure end-to-end latency. Track from user utterance to first audio frame and first visible video frame.

For conversational UIs, the visible response threshold is often more important than raw throughput. A 200 ms improvement in time-to-first-avatar-frame is easy to notice. A 200 ms improvement in model throughput may not be, if the user still sees the avatar frozen while waiting for the first speaking chunk.

Also pay attention to how your agent segments speech. Long, monolithic responses can create awkward avatar behavior if the avatar service only gets one large burst of audio. Smaller, semantically sensible chunks tend to produce more natural pacing, especially when the model is allowed to pause for emphasis.

Practical REST flow from Django

If you prefer keeping the integration explicit, the REST API gives you a straightforward server-side flow:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

From Django, you can wrap that in a view or service object and persist the returned session identifiers in your own database. That is useful if you need to correlate avatar usage with authenticated users, tickets, or conversation logs.

If you want to prototype quickly, the developer dashboard is useful for inspecting sessions and iterating on avatar settings in an interactive way. For code-first work, the public docs are the better reference for exact request and response shapes.

Web app delivery without exposing secrets

One common failure mode is trying to do too much from the browser. For a realtime avatar, the frontend should generally receive only ephemeral session details, room access data, and whatever display metadata it needs. It should not see your API key or be responsible for creating privileged sessions directly.

That is where a backend-managed flow is valuable, whether you are using LiveKit directly or embedding a ready-made experience. The browser can join the session, render the avatar, and handle user interaction, but secret-bearing operations stay server-side. That keeps your security story simple and your rate limiting enforceable.

When to use the other surfaces

If you are already committed to a LiveKit voice agent, the plugin is usually the fastest path. If you need to manage avatars and sessions directly from Django, the REST API is the most explicit surface. If you want to script setup, test with fixture data, or build internal tooling, the Python SDK is more ergonomic than raw HTTP. And if you need to drop an avatar into a site without shipping any backend at all, an iframe embed is the simplest option.

For plugin-specific usage and examples, start with the relevant repository and the integration guide in the docs: https://github.com/protoface-ai/protoface-plugin-pipecat and https://docs.protoface.com.

Conclusion

The architecture is straightforward once you separate responsibilities. Django should own auth, session orchestration, and persistence. LiveKit should own the realtime media path. The avatar service should consume the same speech timing as the agent so the face and voice stay aligned. If you keep those boundaries clean, adding a conversational avatar is mostly a matter of wiring rather than inventing a media stack from scratch.

For exact API fields, SDK usage, and current quickstarts, start with the documentation at https://docs.protoface.com and the examples in the GitHub repos linked above. Build the control plane in Django, keep secrets server-side, and measure end-to-end latency early; that will save you more time than any UI polish later.