How to Connect STT, TTS, and a Realtime Avatar in a Django Support Workflow
Learn to wire streaming STT, TTS, and a realtime avatar into a Django support workflow with secure session orchestration.
Introduction
Support workflows are a good stress test for realtime AI systems because they combine three moving parts: speech-to-text (STT) to capture the customer’s utterance, a language model or rules engine to decide what to say next, and text-to-speech (TTS) to render the answer with low latency. If you want to make that experience feel natural, a static voice assistant is only half the story. You also need a synchronized visual presence: lip movement, expression timing, and a video transport that keeps up with speech.
This post shows how to connect STT, TTS, and a realtime avatar in a Django-based support workflow. By the end, you should be able to reason about the control flow, separate the latency-sensitive pieces from the web app, and wire a voice agent to a live avatar without exposing credentials in the browser.
Start with the workflow, not the avatar
The cleanest way to think about this is as a pipeline:
audio input → STT → agent logic → TTS → avatar/video transport
Django is usually not the realtime media plane here. It is better suited to session orchestration, authentication, persistence, and admin workflows. The actual audio/video exchange should happen over a realtime transport such as WebRTC, usually mediated by a voice-agent framework or a dedicated service.
A few practical implications follow:
STT and TTS should be low-latency and streaming, not batch APIs. You want partial transcripts and incremental audio so the agent can start reacting before the user finishes a long sentence.
The avatar should subscribe to the agent’s speech stream. The video face is not “talking independently”; it is synchronized to the same audio timeline that the user hears.
Django should issue session state, not shuttle audio frames. Use it to create support tickets, authorize access, store conversation metadata, and hand out short-lived session identifiers or embed settings.
If you keep those boundaries clear, the implementation stays predictable and debuggable.
Django as the control plane
In a support workflow, Django typically owns the business transaction around the conversation: user authentication, ticket lookup, routing, and audit logging. The realtime agent can then consume context such as account tier, recent incidents, or open cases.
A common pattern is:
User opens a support page or authenticated dashboard.
Django creates a conversation record and fetches any relevant context.
Django returns a token or session payload to the frontend.
The frontend connects to the voice agent/media service.
The agent handles STT, reasoning, TTS, and avatar sync in realtime.
The important design choice is that Django should not try to own the media loop. Once the session is established, latency and jitter are mostly transport problems, not web-framework problems.
Streaming STT and TTS: what matters in practice
Most failures in conversational support systems come from mismatched timing, not bad model quality. If STT only produces final transcripts after a long pause, the agent feels sluggish. If TTS chunks are too large, the avatar visibly lags behind the audio. If turn detection is poor, the agent talks over the user or cuts off too early.
In practice, you want streaming semantics at both ends:
Streaming STT to emit partial hypotheses as audio arrives.
Streaming TTS to begin playback before the full response is synthesized.
Turn detection / interruption handling so the user can barge in and the assistant can stop cleanly.
For support scenarios, a few guardrails are worth adding:
Cap response length unless the user explicitly asks for detail.
Use deterministic prompts for account-sensitive flows.
Log transcript segments and timestamps, not just the final answer.
Keep the agent’s state machine simple: greeting, identify issue, ask clarifying questions, resolve or escalate.
That state machine matters because a support bot is not a generic chat toy. It should know when to collect identifiers, when to summarize, and when to hand off.
How the avatar stays in sync with speech
A realtime avatar does not need to understand the conversation content to stay convincing. It needs accurate audio timing. The avatar system typically consumes the same synthesized speech stream that is being played to the user, then generates a video face whose mouth movements and expression changes are aligned to that audio.
That alignment has two consequences:
Audio is the source of truth. If you delay audio delivery, the avatar will also lag.
Start/stop boundaries matter. The avatar needs clean speech segment boundaries to avoid odd lip-sync artifacts during silence or interruption.
In a support workflow, this means you should decide where the “voice agent” lives. If the agent already runs in a realtime voice framework, the avatar should attach there. If you are starting from a browser or Django app, use a transport that can bridge the speech stream into an avatar session without round-tripping through your server for every frame.
A minimal Django pattern for session orchestration
Here is a pragmatic backend shape: Django creates a support session, stores metadata, and returns the client with enough information to join the realtime interaction. The exact session fields depend on your chosen media layer, but the structure is stable.
This endpoint should be fast and side-effect free enough to call when the user clicks “Talk to support.” The heavy lifting happens after the browser has the session details.
Where Protoface fits
This is the point where Protoface is useful: it gives the voice agent a synchronized video face without forcing you to build the avatar layer yourself. In a LiveKit-based stack, the LiveKit plugin is the most direct integration path, because it lets you drop the avatar into an existing realtime agent and keep the audio/video relationship aligned.
Conceptually, the plugin sits beside your voice agent, not inside Django. Your Django app still handles auth and support state, while the agent and avatar handle the low-latency speech loop.
If you are wiring this up for production, read the integration notes in the docs and the plugin examples in the repository. The critical thing is that the avatar should consume the same streamed speech that your agent emits, rather than a separate synthesized copy.
REST and Python SDK flows for backend-managed sessions
There are cases where you want Django or a background worker to create or manage sessions directly: for example, pre-provisioning an avatar experience for an authenticated support queue, or recording usage metadata before a customer joins. In that case, the REST API and Python SDK are the natural surfaces.
The REST API is authenticated with an API key in the usual bearer form:
The exact endpoint paths and request bodies are documented in the product docs, but the operational pattern is straightforward: keep the API key server-side, create or configure avatars and sessions from trusted backend code, and hand only short-lived session data to the client.
If you prefer Python, the SDK is a better fit for Django services or Celery tasks that need to set up infrastructure before a support call starts. The code below is deliberately schematic; use the package’s documented classes and fields for your specific workflow.
The key operational rule is simple: never leak the API key to the browser. Keep anything that can create, delete, or reconfigure avatar sessions on the server side.
Browser embedding versus backend-managed voice agents
For pure web support experiences, an iframe embed can be the fastest route because it avoids shipping your own backend media layer. The browser gets an embedded avatar experience, while the API key stays hidden. That is a different trade-off from the Django + voice-agent path: it reduces integration work, but it is less flexible if you need to tightly couple the conversation to your internal support systems or existing voice stack.
Use iframe-style embeds when you want a customer-facing experience with minimal infrastructure. Use backend-managed sessions when the support flow depends on your own agent logic, ticketing, or existing realtime stack.
In both cases, the same engineering principle applies: isolate the real-time media path from the web app, and keep session authorization explicit.
Operational gotchas worth planning for
A few failure modes show up repeatedly in production support agents:
Clocking and buffering issues: if TTS buffers too aggressively, the avatar may appear to “wake up” late.
Interruptions: implement barge-in so a user can stop a verbose answer mid-sentence.
Session cleanup: terminate abandoned realtime sessions to avoid wasted usage and zombie media rooms.
Escalation paths: make it easy to hand off to a human when confidence drops or the user requests it.
Rate limiting: especially for public-facing embeds, enforce duration and per-IP limits server-side or through the embed configuration.
Also remember that “good enough” lip sync is still sensitive to bad audio. No avatar layer can hide clipped TTS, long synthesis stalls, or poor turn detection.
Conclusion
The practical shape of a realtime support system is: Django owns orchestration, STT turns audio into partial text, the agent decides what to say, TTS streams the answer, and the avatar mirrors that speech in realtime. If you keep the media path separate from the web app and treat timing as a first-class concern, the whole stack becomes much easier to reason about.
For implementation details, start with the documentation and the relevant quickstart or plugin repository for your stack. If you are already using LiveKit, the plugin route is usually the shortest path. If you need backend-managed sessions, use the REST API or Python SDK and keep your API keys on the server. From there, the remaining work is mostly product engineering: prompt design, support-state modeling, and careful handling of interruptions and escalation.