How to Stream STT/ASR Audio to the Protoface REST API for a Realtime AI Avatar
Learn how to stream STT/ASR partial and final transcripts to Protoface REST API sessions for low-latency realtime AI avatar sync.
Introduction
If you already have speech-to-text (STT/ASR) output in your stack, the remaining problem is usually not “how do I get text?” but “how do I keep the avatar synchronized with the user’s speech in real time?” That means taking a stream of recognized words, preserving timing, handling partial hypotheses, and forwarding enough signal to drive a lip-synced talking face without adding noticeable lag.
This post shows the practical shape of that pipeline. By the end, you should know how to:
decide where ASR fits in a realtime avatar architecture,
stream transcripts or partial results to an avatar session without fighting latency,
avoid common timing and state bugs, and
wire the pieces together using either the REST API or a voice-agent integration.
What “streaming STT to an avatar” actually means
There are two distinct streams in a conversational avatar system:
Audio stream: microphone or telephony audio flowing into STT.
Semantic stream: the text hypothesis produced by STT, often as partial and final transcripts.
For a voice agent with a talking face, the avatar typically does not need raw ASR audio if your agent already has another audio path. What it does need is a low-latency signal that says, “the user is speaking now” and “here is the current text / utterance boundary.” That signal lets the system coordinate:
when to animate the avatar as listening versus speaking,
when to interrupt or barge in,
when to start generating a reply, and
how to align mouth movement with synthesized or recorded speech.
In practice, your STT layer emits partials like “I need to change my…” and then a final result like “I need to change my shipping address.” The key is to forward those updates as a stream, not as a single lump of text after the user finishes talking. A realtime avatar system is mostly about minimizing the gap between user intent and visible response.
Designing the pipeline: audio in, transcript out, avatar state update
A useful mental model is:
Capture audio from the browser, agent runtime, or telephony bridge.
Send it to your STT provider.
Emit partial transcript events as they arrive.
Commit a final transcript when the utterance ends.
Use that utterance boundary to trigger the next step in your dialog manager.
There are a few implementation details that matter more than the rest:
Preserve ordering. A partial transcript can arrive after a newer one if you fan out work incorrectly. Sequence numbers or monotonic timestamps help.
Track utterance state explicitly. Don’t infer “user finished speaking” from the absence of audio alone. Use VAD or the STT provider’s finalization event if available.
Separate “display text” from “control events.” The avatar should get speech state transitions, while your app may also want the raw text for logging, analytics, or downstream LLM context.
Budget for latency. Every extra hop matters: microphone → STT → app logic → avatar session → video render. Keep the control path as thin as possible.
Minimal REST flow: create a session, then drive it with transcript events
If you’re building your own orchestration layer, the cleanest pattern is to create a realtime session first, then feed it user events as your STT stream evolves. The exact request/response schema depends on the API surface you’re using, so treat this as structural pseudocode and check the docs for the current fields.
Once you have a session, your app can forward transcript updates into whatever control endpoint or SDK call the API exposes for that session. In a typical implementation, you’d send:
partial transcript events as the STT engine updates its hypothesis,
a final transcript event when the utterance is committed, and
speaker-state changes such as start-speaking / stop-speaking if your dialog manager needs them.
The important thing is not the exact wire format; it’s that the avatar session sees the same event boundaries your voice agent sees. If your STT provider offers timestamps per token or per segment, keep them. They are useful for debugging awkward lip-sync or delayed interruptions.
Handling partial transcripts, finals, and barge-in
Most bugs in realtime voice UX come from treating partial transcripts like final ones. Partial STT results are inherently unstable. The user may say “I need a refund” and the engine may first emit “I need a roof…” before correcting itself. That is normal.
For avatar control, a few rules help:
Do not trigger downstream business logic on partials unless you are intentionally doing speculative UX.
Use partials to show “listening” state or a lightweight live caption, not to commit actions.
Commit on final only for dialog turns, intent classification, or LLM input.
Support barge-in by stopping the avatar’s current speech when fresh user audio is detected. If the agent keeps talking over the user, the experience degrades fast.
In a well-behaved system, the audio stream and transcript stream are coordinated but independent. If the user interrupts the agent, your VAD or STT endpointer should flip the session into listening mode immediately, even if the previous assistant utterance is still playing out on the client.
Example: wiring STT events into a Python control loop
Here’s a compact example of how you might structure the orchestration in Python. This uses the Python SDK for session management, but the important part is the event flow, not the specific method names.
This pattern scales well because the UI, dialog manager, and avatar session all consume the same turn state. You do not want separate “truths” about whether the user is speaking. Keep one source of turn coordination in your app.
Where Protoface fits: avatar session control without rebuilding the media layer
If you want the talking face without building your own video rendering stack, this is where Protoface is useful. You create and manage realtime avatar sessions through the REST API or the Python SDK, then your app feeds the session from your existing voice pipeline.
For developers already running a voice-agent stack, the integration path is straightforward: use your current STT provider, keep your transcript stream, and attach the avatar session to the same turn lifecycle. If you’re working from Python, the SDK is the fastest route. If you’re in a realtime voice agent framework, the LiveKit plugin is the lower-friction option because it drops the avatar into the agent runtime and keeps the speech/video synchronization in one place.
For implementation details, session fields, and current API shapes, use the documentation. For agent integrations and examples, the relevant repositories are usually the best starting point because they show the control flow in context.
Practical gotchas
A few issues show up repeatedly in production:
Double-finalization. Some STT systems emit a final result and then a correction. Make your turn state idempotent.
Clock drift. If you rely on local timestamps across multiple workers, your logs will lie to you. Prefer provider timestamps and monotonic sequence numbers.
Over-eager response generation. If you start an LLM response on partials, you’ll get fast but wrong behavior. Speculation should be a deliberate choice.
Blocking the event loop. Realtime systems fail when you put synchronous work on the critical path. Keep audio ingestion, transcript handling, and avatar updates asynchronous.
Browser exposure. If you’re shipping a web client, keep API keys off the frontend. Use a backend or a managed embed pattern instead of pushing secrets into the browser.
If you need an interactive avatar on a website with no backend, a customer-managed iframe embed is usually the better fit than hand-rolling browser logic. But for STT-driven voice agents, the core architectural advice stays the same: preserve the realtime event stream and keep your turn boundaries explicit.
Conclusion
Streaming STT/ASR to a realtime avatar is mostly an exercise in clean event handling. Treat partial transcripts as unstable hints, commit only on final utterances, and keep the avatar session aligned with the same turn state your agent uses. That gives you low-latency captions, correct barge-in behavior, and a face that stays in sync with the conversation instead of lagging behind it.
If you want to implement this against Protoface, start with the REST API or the Python SDK, then move to the LiveKit plugin if your stack is already running there. The docs at docs.protoface.com are the right place to verify request shapes and current integration details.