Building a Django Realtime Voice Agent with Lip-Synced Avatar Video
Build a Django realtime voice agent with lip-synced avatar video using LiveKit, REST APIs, and server-side session control.
Introduction
Realtime voice agents are straightforward until you add a face. At that point you have a three-way synchronization problem: the model is producing text or speech incrementally, the audio stream has to stay low-latency, and the avatar video has to track phonemes well enough to look believable instead of delayed or uncanny. If you get it wrong, the result is a voice agent that talks over itself, drifts out of sync, or visibly lags behind the conversation.
This post walks through the practical architecture for building a Django-based realtime voice agent with a lip-synced avatar video layer. By the end, you should understand how to wire a web frontend to a Django backend, keep audio and video in sync, and choose the right integration surface for an avatar system without exposing secrets in the browser.
What “realtime” actually means in this stack
For a voice agent, “realtime” usually means the system is streaming at least one of these continuously:
Mic audio from the browser to your backend or agent runtime.
Partial ASR transcripts and intermediate model outputs.
Generated speech audio back to the browser.
Avatar frames or a WebRTC video track that lip-syncs to that speech.
The key constraint is latency budget. Human conversation is tolerant of short pauses, but once you get much beyond a second of perceived delay, turn-taking gets awkward. That means your video face cannot be a post-processing step after the speech is complete. It needs to be driven from the same streaming event timeline as the audio.
In practice, the architecture looks like this:
The browser captures microphone audio and sends it over a realtime channel, usually WebRTC.
Your agent stack performs ASR, reasoning, and TTS in a streaming pipeline.
The generated speech audio is forwarded to an avatar renderer that produces a synchronized talking face.
The browser plays audio and video as a coordinated media session.
Django is not usually the media transport itself; it is the control plane. Use it for auth, session creation, webhooks, database state, and your application logic. Let the realtime layer handle the media plane.
Django as the control plane
A clean design is to keep Django responsible for user/session lifecycle and delegate the voice session to a dedicated realtime service or agent worker. That lets you keep your existing Django app, admin, permissions, and database models while avoiding the mistakes of trying to push media streams through request/response handlers.
A minimal data model usually tracks:
the authenticated user
a voice-agent session record
the avatar or persona configuration
state transitions such as created, active, ended, failed
When the user clicks “start,” Django creates a session row and returns the browser a short-lived token or session handle. The browser then connects to the realtime media service. If you need server-side orchestration, Django can call your agent backend directly or via a queue.
Where the lip sync happens
Lip-syncing is not the same as simply overlaying video on audio. The avatar system needs a speech-driven animation model that can align mouth shapes with phonetic timing, usually from the speech audio stream or from intermediate speech tokens. The important part for developers is that the avatar must stay on the same timeline as the audio, so the animation engine can adjust frame generation or visemes as speech emerges.
There are a few consequences worth keeping in mind:
Buffering matters. If you buffer too much audio before starting video, the face will look frozen at the beginning of each response.
Chunk size matters. Very small audio chunks reduce latency but can increase overhead and jitter if your pipeline is inefficient.
Interruptions matter. If the user barges in, you need to stop or fade the current speech and reset the avatar state immediately.
Turn boundaries matter. A good agent exposes clear “start speaking,” “stop speaking,” and “speech ended” events.
If you are building your own renderer, the coordination problem becomes surprisingly subtle. In most teams, it is more cost-effective to use a system that already handles the media synchronization for you and focus on the agent logic.
A practical Django flow
For a useful implementation, keep your Django endpoint thin. It should authenticate the user, create a voice session, and hand the browser a session descriptor. The browser then establishes the realtime connection. Below is an illustrative example; exact field names depend on the SDK or API version you use.
On the frontend, you would connect the mic, play back the generated audio, and render the avatar video in the same session. If your agent runtime is built around LiveKit, the cleanest path is usually to let the voice agent continue to own the audio pipeline and add the avatar as a media participant or plugin.
Using a LiveKit agent with a synchronized avatar
If your stack already uses LiveKit Agents, the avatar integration is the least invasive route. The idea is simple: keep your existing agent, and attach a video avatar so the agent’s generated speech is mirrored as lip-synced video. That preserves your current ASR/TTS/LLM design while adding the visual layer.
The GitHub repo linked here is the best place to inspect examples if you are evaluating the broader integration pattern, and the general API surface is documented in the vendor docs. The actual plugin package for LiveKit is published on PyPI as livekit-plugins-protoface.
A stripped-down worker might look like this:
The important operational detail is that the avatar should be tied to the same speech output event stream as TTS. If your agent can interrupt itself, the avatar needs to stop animating immediately when the audio is canceled, not after the next TTS chunk completes.
REST API and server-side session creation
When you need direct control from Django, the REST API is the straightforward option. Keep your API key on the server only and create avatars or sessions there, then pass a short-lived client artifact to the browser if needed.
For example, a server-side request to create a session might look like this:
Then Django can return the resulting session metadata to the client. This pattern keeps secrets out of the browser and gives you one place to enforce user permissions, quota logic, and audit logging. It also makes it easier to reconcile usage with your own application events.
For Python server code, the SDK is useful when you want to manage avatars and sessions without hand-rolling HTTP calls. The exact method names are documented, but the pattern is the same: initialize the client with your API key on the backend, create or fetch an avatar, then create a realtime session and hand the browser only what it needs.
Trade-offs and gotchas
The biggest mistake teams make is mixing responsibilities. Django should not become your media server, and your realtime layer should not become your business logic store. Keep the separation clean.
A few practical gotchas:
Authentication: never ship long-lived API keys to the browser.
Session cleanup: end stale sessions and release media resources when the user disconnects.
Network variability: handle jitter and reconnection without resetting the entire conversation state unless you intend to.
Voice interruption: support barge-in so the user can cut off the agent naturally.
Metrics: track end-to-end latency separately from ASR, LLM, TTS, and avatar render time.
If you are debugging a “laggy face” complaint, inspect where delay is introduced. Often the culprit is not rendering itself; it is an over-buffered audio pipeline, late TTS flushes, or a session setup path that takes too long before the first response begins.
Where Protoface fits
Protoface is a good fit when you want the avatar layer to be a solved problem instead of a custom graphics project. In this architecture, the relevant surfaces are the LiveKit Agents plugin, the REST API, and the Python SDK. That means you can keep your Django app focused on auth and orchestration while the avatar session runs through a dedicated integration layer.
For teams already using LiveKit, the plugin is especially practical: you can drop a talking face into an existing voice agent without rewriting the agent itself. If you prefer to manage the session lifecycle from Python, the SDK and REST API give you a server-side path to create avatars and sessions, then hand the browser a minimal connection payload. The docs at docs.protoface.com cover the exact request and object shapes.
Conclusion
The main design principle is simple: keep Django as the control plane, keep the voice stack realtime, and make the avatar follow the same speech timeline as the audio. Once you do that, adding a face to a voice agent becomes an integration problem instead of a graphics project.
If you are starting from scratch, build the session lifecycle first, then wire the voice agent, then attach the lip-synced avatar. Validate interruption handling and latency early, because those are the issues that usually surface in production. For concrete setup details, implementation examples, and current API shapes, start with the docs at docs.protoface.com.