Introduction

Streaming speech-to-text (STT) for a realtime avatar is less about “transcribing audio” and more about keeping a low-latency control loop stable: audio frames arrive continuously, partial transcripts update incrementally, the agent decides when a user has finished an utterance, and the avatar has to stay visually synchronized while the conversation is still in flight. If any part of that loop is too slow or too brittle, the experience feels laggy or unnatural.

In this quickstart, you’ll build the STT side of that loop in Python and wire it into a realtime voice agent that can drive an animated face. By the end, you should understand how to stream microphone audio, consume partial and final transcripts, handle turn boundaries, and keep the rest of the agent pipeline responsive enough for a lip-synced avatar.

What “streaming STT” means in practice

For developers, streaming STT usually means you are not waiting for an entire recording to finish before sending audio to the recognizer. Instead, you push small chunks of PCM or Opus audio as they arrive, and the STT service emits:

• Partial transcripts while the user is still speaking.

• Final transcripts once a segment is considered complete.

• Timing or segment events that help you detect end-of-utterance and drive downstream behavior.

This matters for avatars because the agent’s response pipeline needs to be triggered at the right time. If you wait for a long, delayed final transcript, the face appears to “listen” too long. If you react to every partial too aggressively, you can interrupt the user mid-sentence. The core engineering problem is balancing latency, segmentation, and confidence.

There are three implementation details to get right early:

1. Audio format consistency — know the sample rate, channels, and sample width expected by your STT provider.

2. Backpressure — don’t let your audio capture loop outrun the network or inference pipeline.

3. Utterance segmentation — define when partials become “good enough” to hand to the LLM or dialog manager.

Streaming audio from Python without introducing latency

For a realtime agent, your microphone or media source should feed a small, steady stream of frames into the STT client. The exact implementation depends on your audio source, but the pattern is consistent: capture 20–40 ms chunks, convert them to the format your STT provider expects, and send them immediately.

At a high level, your loop looks like this:

import asyncio

await stt_client.finish()

import asyncio

await stt_client.finish()

import asyncio

await stt_client.finish()

That simple loop hides a few practical concerns:

• Chunk size: smaller chunks reduce latency but increase overhead. In practice, ~20 ms is a common starting point.

• Jitter: if your capture timing is inconsistent, STT quality and endpointing both suffer.

• Resampling: many microphones do not match the model’s preferred sample rate, so plan to resample once, not repeatedly in the hot path.

If you are using WebRTC, LiveKit, or another realtime transport, you often receive decoded audio frames from the media layer rather than raw microphone buffers. The same rule still applies: preserve timing, keep the frames small, and avoid blocking the capture callback with any heavy work.

Consuming partials and finals correctly

The transcript stream is usually event-driven. Treat partial results as speculative UI state, and finals as committed state. That distinction is important for both user experience and agent logic.

async def handle_transcripts(transcript_stream):

print(f"final: {event.text}")

async def handle_transcripts(transcript_stream):

print(f"final: {event.text}")

async def handle_transcripts(transcript_stream):

print(f"final: {event.text}")

For conversational systems, a final transcript usually means one of two things:

• You can pass the user’s utterance to the LLM or dialog manager.

• You can reset the agent’s “listening” state and prepare for response generation.

Do not use partials as if they were final input unless you are intentionally building incremental understanding. Most production agents need a debounce or endpointing policy so they do not trigger responses too early. A common pattern is:

• Show partial text in the UI as it arrives.

• Wait for a final transcript or silence-based endpoint.

• Only then enqueue the turn for generation.

That pattern also keeps avatar animation coherent. The face can remain in a listening state while partials are arriving, then transition to speaking once generation begins.

Turn detection, latency, and avatar synchronization

Once you have streaming transcripts, the next problem is deciding when the user has finished speaking. This is where most realtime systems get tricky. Endpointing too aggressively causes interruptions; endpointing too conservatively makes the agent feel slow.

A practical setup usually combines three signals:

• Voice activity: audio energy indicates the user is speaking.

• STT finalization: the recognizer thinks a segment is complete.

• Short silence window: a few hundred milliseconds of silence can be enough to confirm the turn.

For avatars, this is not just a transcription concern. If the avatar starts speaking before the user is clearly done, the visual overlap feels broken. If it waits too long, the video face looks unresponsive. In practice, the cleanest architecture is to let STT drive turn boundaries, then hand the resulting text to the response generator, and only switch the avatar into “speaking” once TTS or response audio is ready to play.

If you are implementing your own pipeline, keep the STT, LLM, and avatar layers loosely coupled. That gives you room to swap providers and tune latency independently. It also makes failure handling saner: if transcript delivery stalls, you can hold the avatar in a listening state rather than producing a confusing visual transition.

Minimal Python structure for the realtime loop

A useful way to think about the whole flow is as three concurrent tasks:

1. Capture and stream audio into STT.

2. Consume transcript events and detect turn completion.

3. Generate and play the response while updating the avatar state.

In Python, that often looks like one task per coroutine, with a queue between transcription and response generation:

import asyncio

await agent.play(reply)

import asyncio

await agent.play(reply)

import asyncio

await agent.play(reply)

This structure is intentionally simple, but it captures the main constraint: don’t block the audio ingest path while waiting for language generation or video updates. Realtime avatar systems tend to fail when a single synchronous callback does too much work.

Two practical gotchas worth calling out:

• Threading vs. async: many audio APIs are callback-based, while STT and agent clients are async. Bridge them with a queue instead of doing network calls inside the callback.

• Cancellation: when a user interrupts the agent, you need to cancel pending speech and reset the turn state immediately.

Where Protoface fits in this pipeline

This is exactly the kind of control loop Protoface is designed to sit beside: you keep your STT and agent logic in Python, then attach the avatar layer through the surface that matches your stack. If you are already using LiveKit for voice, the LiveKit-oriented plugin and quickstarts are the shortest path to a synchronized talking face. If you want direct programmatic control instead, the Python SDK and REST API let you create avatars and manage realtime sessions from code, with authentication handled by API keys in the usual bearer-token pattern.

import requests

resp.raise_for_status()

import requests

resp.raise_for_status()

import requests

resp.raise_for_status()

I’m keeping that example deliberately schematic because the exact request shape depends on the API object you are creating. The important point is that session creation and avatar management are exposed as normal API operations, so they fit cleanly into the same backend that is already orchestrating your transcription and agent logic. For concrete endpoints and fields, use the documentation.

Testing and debugging tips

Streaming STT bugs are usually timing bugs, not parsing bugs. When something feels off, inspect the full event timeline instead of just the final transcript.

• Log audio frame timestamps so you can see whether capture jitter is the real issue.

• Record partial and final events separately; partial churn often reveals endpointing problems.

• Measure end-to-end latency from user speech start to transcript finalization, then from finalization to avatar response start.

• Test interruption behavior by speaking over the agent and verifying that cancellation works cleanly.

If you are using a browser-based flow, also make sure the UI reflects the same state machine as the backend. A user should see “listening,” “processing,” and “speaking” transitions that line up with actual transcript and playback events. That consistency matters more than any individual model choice.

Conclusion

Streaming STT for a realtime avatar is mostly an exercise in systems design: keep audio chunks small, treat partial transcripts as provisional, finalize turns with a clear endpointing policy, and keep response generation off the ingest path. Once those pieces are in place, the avatar layer becomes much easier to reason about because it is driven by clean state transitions rather than noisy ad hoc callbacks.

If you want a concrete implementation path, start with the docs at docs.protoface.com, then choose the integration surface that matches your stack. For LiveKit voice agents, the plugin route is usually the fastest. For backend-controlled session management, use the REST API or Python SDK. Either way, the main goal is the same: make transcription, turn detection, and avatar playback behave like one realtime pipeline rather than three unrelated subsystems.