Building a Realtime Avatar App with Protoface REST API Speech-to-Text Streaming in Python
Build a Python realtime avatar app with Protoface REST API, streaming STT, event-driven sessions, and low-latency lip sync.
Introduction
If you are building a voice agent, support bot, or conversational demo, the missing piece is often not speech recognition or text generation—it is presence. A realtime avatar gives the agent a face that can track the conversation with low latency lip sync and a usable visual channel on top of the voice pipeline.
This post shows how to wire up a realtime avatar app in Python using streaming speech-to-text as the input side of the loop, then feeding recognized text into an agent or avatar session. By the end, you should understand the moving parts: audio capture, low-latency transcription, event-driven session control, and where the avatar service sits in the architecture.
Architectural shape: audio in, transcript events out, avatar session in the middle
At a high level, the system has four stages:
Client audio capture from a mic, browser, or telephony bridge.
Streaming speech-to-text that emits partial and final transcripts as the user speaks.
Application logic that turns transcripts into prompts, tool calls, or agent turns.
Realtime avatar playback that animates a face in sync with the generated response.
The important implementation detail is that STT is not a batch step. You want partial transcripts as soon as they are stable enough to be useful, while still being able to revise your application state when the final transcript arrives. That means your code should be written around events, not synchronous request/response calls.
Streaming speech-to-text in Python: keep the loop event-driven
For a realtime voice UX, the transcription layer typically receives a continuous audio stream and yields incremental results. The exact client depends on your STT provider, but the shape is the same: you push audio frames in, subscribe to transcript events, and decide what to do with each update.
Here is a minimal pattern in Python that shows how to structure the loop. The STT client here is illustrative; the point is the control flow.
Two practical notes matter here:
Partial transcripts are volatile. Do not treat them as committed state unless your UX can tolerate small corrections.
Final transcripts are the boundary for agent actions. If you are triggering tool calls, updating CRM records, or generating avatar replies, do it on finalized text unless you have a very strong reason not to.
From transcript to response: reduce latency without making the app fragile
The next design choice is when to start generating a reply. There are two common patterns:
Final-only. Wait for the user to stop speaking, then generate a response. This is simpler and usually safer.
Incremental. Start planning on partial transcripts, then confirm or revise when the final transcript arrives. This can feel faster, but it is easier to get wrong.
For most avatar apps, final-only is the right default. The user experience is still responsive if your transcription latency is low and your avatar begins moving as soon as the response is ready. If you later optimize for interruption handling or barge-in, you can layer incremental logic on top.
One subtle issue is turn detection. In practice you need some combination of:
speech end detection from the STT layer,
explicit push-to-talk or VAD gating, and
application-level cancellation when the user interrupts.
If you skip this, your avatar may continue “listening” or “speaking” after the user has already taken back the floor.
Putting a realtime avatar on the response path
This is where a service like Protoface fits naturally. The avatar is not your speech recognizer and not your LLM; it is the video face that sits on top of the conversational loop and renders a synchronized response once your agent has something to say.
There are two common ways developers integrate it:
REST API for creating and managing avatars and realtime sessions from your backend.
Python SDK when you want the session lifecycle to live in application code instead of shelling out to raw HTTP.
A typical REST flow is: authenticate with your API key, create or select an avatar, open a session, then hand the resulting session information to your realtime pipeline. The exact request shape depends on the endpoint, so use the docs for field names and response objects. The important part is that the API key stays on the server.
In Python, the same idea becomes easier to compose with your transcription and agent code. The SDK is useful when you want to create sessions dynamically or manage avatar state inside the same process that handles audio and transcript events.
Once the session exists, your app can attach it to the conversation pipeline. The practical effect is that your agent emits spoken output and the avatar renders the corresponding face movement and lip sync, rather than leaving the user with only audio.
WebRTC, latency, and why session boundaries matter
Realtime avatars are usually only convincing when the end-to-end loop stays tight. That means you should treat the following as first-class performance constraints:
Audio capture latency: how quickly mic frames reach your backend.
STT latency: how quickly transcripts stabilize.
Agent latency: how quickly you generate the next response.
Avatar render latency: how quickly motion and lip sync appear onscreen.
In practice, a browser or WebRTC-based media path usually gives the smoothest result because it avoids a lot of buffering and transcoding overhead. But the exact transport is less important than keeping the session boundaries clean. A session should represent one conversational context with a predictable lifecycle: start, active exchange, and teardown.
That lifecycle matters operationally too. If you do not explicitly close sessions, you can end up with dangling resources, confusing usage data, and stale state when the same user reconnects.
Practical implementation pattern in Python
A good production shape is to separate responsibilities into three coroutines or services:
Audio ingestion that normalizes frames and feeds STT.
Transcript handling that decides when to trigger the agent.
Session control that creates, updates, and terminates the avatar session.
That separation keeps the code testable and makes interruption handling much easier. For example, when a new final transcript arrives, you can cancel the in-flight response, update the session state, and start a new turn without rewriting your audio pipeline.
If you are integrating with a broader voice stack, the LiveKit path is also worth knowing about. The livekit-plugins-protoface plugin can drop a Protoface avatar into a LiveKit voice agent so the agent gains a synchronized talking video face. If your existing app is already built around LiveKit Agents, that is the shortest path from “voice-only agent” to “voice plus face.”
Common gotchas
A few implementation mistakes show up repeatedly in realtime avatar projects:
Trying to do everything in the browser. Keep API keys server-side unless you are using a customer-managed embed flow designed for browser use.
Using final transcripts too late. If your STT endpoint is laggy, the avatar will feel detached from the conversation.
Ignoring interruptions. Users will talk over the agent; your app should recover cleanly.
Not bounding session lifetime. Long-running sessions need explicit cleanup and usage tracking.
Overfitting to one transcript provider. Keep your app logic provider-agnostic so you can swap STT vendors later.
Also remember that quality tier affects cost. If you are prototyping, start with the lowest tier that meets your latency and visual quality requirements, then measure before you optimize upward.
Conclusion
The basic recipe for a useful realtime avatar app is straightforward: stream audio into STT, treat transcript events as the control plane for your conversation logic, and attach a synchronized avatar session on the response path. The engineering work is mostly about latency management, state transitions, and keeping secrets off the client.
If you want to implement this with Protoface, start with the API and SDK docs at docs.protoface.com, then pick the integration surface that matches your stack. If you are already on LiveKit, the plugin route is the fastest way to add a face. If you are building your own backend, use the REST API or Python SDK and keep the session lifecycle explicit.
For examples and starting points, the quickstarts linked from the project repo are a good next stop, and the developer dashboard is useful once you want to inspect sessions, keys, and usage while you iterate.