From Audio Stream to Avatar Reply: A Quickstart for Realtime ASR Integration
Quickstart for realtime ASR-to-avatar integration: stream audio, handle latency, sync TTS lip-sync, and use Protoface SDK/REST/LiveKit.
Introduction
If you already have speech-to-text, a voice agent, or a streaming inference pipeline, the last mile is usually what feels awkward: the user hears a response, but the UI still looks static. Adding a realtime avatar changes that. The hard part is not “making a face move” in isolation; it’s keeping audio, transcription, response generation, and lip-sync aligned under realtime constraints.
This post walks through the practical integration path: how audio moves from a stream into ASR, how an agent turns that transcript into a reply, and how that reply becomes a synchronized talking face. By the end, you should understand the latency budget, the control points, and the simplest ways to wire an avatar into a voice application without inventing a custom media stack.
Start with the streaming model, not the avatar
For realtime ASR integration, the key design choice is whether your system processes audio in discrete requests or as a continuous stream. For conversational UX, the stream is usually the right model. The client captures microphone audio, sends frames over WebRTC or a similar realtime transport, ASR emits partial and final transcripts, and the agent consumes those transcripts incrementally.
That matters because the avatar should not wait for the entire answer before it starts moving. If your agent is generating speech incrementally, the avatar can begin lip-syncing as soon as the first phonemes are available. If your stack only exposes full text after the model finishes, you’ll get a noticeable delay between hearing and seeing the response.
The practical pipeline looks like this:
Capture audio from the user.
Stream audio frames to ASR.
Emit partial transcripts for turn detection and early reasoning.
Generate the agent’s response, ideally in a streaming form.
Convert the response into audio, then drive the avatar animation from that audio stream.
In other words, the avatar is downstream of the agent’s speaking path, not the source of truth. The avatar should reflect the audio timeline you’re already producing.
Understand the latency budget
Realtime conversational systems usually fail for the same reason: each individual step is “fast enough,” but the sum is not. A typical end-to-end turn includes mic capture, network transit, VAD or turn detection, ASR, agent reasoning, TTS, video rendering, and playback buffering. If each step adds even 100–200 ms, the user experiences a sluggish, over-mediated conversation.
There are two useful consequences of this:
Keep the media path continuous. Avoid round-tripping audio through multiple encoding stages unless necessary. Every unnecessary transcode and buffer is extra delay.
Stream partials wherever possible. Partial ASR and incremental agent output let the system react before a sentence is complete. This is especially useful for barge-in and interruption handling.
Also be explicit about who owns turn-taking. If the client, ASR layer, and agent all try to decide when the user is done speaking, you’ll end up with duplicated logic and inconsistent behavior. Pick one place to do final turn detection, and let the rest of the stack consume that signal.
Wire ASR to the agent, then the agent to the avatar
The cleanest implementation pattern is: ASR produces text, the agent produces text or audio, and the avatar follows the agent’s speaking output. In a voice app, that usually means the avatar subscribes to the same audio stream the user would hear. If the agent speaks through TTS, the avatar lip-syncs against that TTS audio. If your agent produces audio directly, the avatar follows that output instead.
Here’s a minimal sketch of the data flow in Python terms. The exact class names and fields depend on your SDK or orchestration layer, so treat this as structural, not copy-paste production code:
The important bit is not the syntax; it’s the contract. Your application should create a session, bind it to a specific avatar, and feed the speaking stream into that session so video and audio stay in lockstep. If your architecture already has an event bus, this is often just another subscriber.
What to watch for in production
Realtime avatars are sensitive to the same edge cases that affect live voice agents:
Backpressure: if the video side can’t keep up, don’t let old audio pile up indefinitely. Drop stale frames or reset the turn cleanly.
Interruption handling: when the user barges in, stop or fade the current speaking stream immediately, then transition to listening mode.
Clock drift: if audio and video are generated on different threads or services, ensure they share a consistent timeline. Lip-sync quality degrades quickly when timestamps diverge.
Prompt discipline: long, rambling responses are more expensive to render and harder for users to follow. Short answers with explicit acknowledgements tend to feel more responsive.
One subtle issue is that ASR confidence and turn detection are often conflated. Low-confidence partial transcripts should not necessarily trigger a full retry, but they also shouldn’t be treated as final user intent. In practice, keep partials for responsiveness and final transcripts for state changes.
Protoface in the middle, where it belongs
This is the part Protoface is built for: once your voice agent has speech output, you can attach a synchronized avatar without building a separate media pipeline. For Python-based integrations, the SDK gives you a programmatic way to create sessions and manage avatars; if you’re already using a LiveKit voice agent, the quickstart repo and the LiveKit plugin path are the fastest way to drop in a talking video face with minimal glue. The plugin package is published on PyPI as livekit-plugins-protoface, so it fits naturally into an existing LiveKit agent loop.
A typical LiveKit-side setup is conceptually simple:
The useful part here is architectural separation. You keep ASR, NLU, and TTS in the voice stack you already trust, and Protoface handles the avatar side as a synchronized output surface. That reduces the chance you end up with a bespoke video renderer tightly coupled to a single agent implementation.
REST API and session lifecycle basics
If you need to create avatars or manage sessions from your own backend, the REST API is the right control plane. It is authenticated with API keys, so the key stays on the server. A basic request pattern looks like this:
Use the API when your application needs to provision sessions dynamically, store metadata, or coordinate avatars across multiple clients. The developer dashboard at docs.protoface.com is the best place to check the exact request/response shapes, rate limits, and quality-tier behavior before wiring this into production. The general rule is the same as with any other media service: keep secrets server-side, and create session artifacts only when the user actually needs them.
Conclusion
If you’re adding realtime ASR to an avatar-driven experience, the implementation challenge is mostly about respecting the media pipeline: stream audio continuously, preserve timing, stream the agent’s output, and let the avatar follow that output with minimal buffering. Once you treat the avatar as a synchronized consumer of the speech stream, the system becomes much simpler to reason about.
From there, Protoface gives you a few pragmatic entry points: use the Python SDK or REST API for backend-managed sessions, or plug into an existing LiveKit voice agent with the plugin path. For implementation details, field names, and current quickstarts, start with docs.protoface.com and the example repos linked there. If you build against the quickstarts first, you’ll usually end up with a cleaner integration than if you start by wiring everything manually.