Introduction

Streaming live speech recognition into a realtime voice avatar is mostly a systems problem: you need low-latency audio capture, incremental transcription, sentence-level turn handling, and a video face that stays synchronized with the spoken output. The common failure mode is building these pieces independently and then discovering they drift: the transcript arrives late, the avatar starts talking before the agent has finished thinking, or the mouth motion lags behind the generated audio.

By the end of this post, you should understand the end-to-end pipeline well enough to wire speech recognition into a voice agent, decide where to keep state, and avoid the timing mistakes that make realtime avatars look uncanny instead of responsive.

1. The pipeline: audio in, partial text out, synchronized face out

At a high level, a realtime voice avatar system has four moving parts:

1. Audio capture and transport from the user, usually over WebRTC or a similar low-latency media path.

2. Streaming speech recognition that emits partial transcripts before the user finishes speaking.

3. Turn logic that decides when the user is done, when to send text to an LLM, and when to interrupt or continue.

4. Speech synthesis plus avatar rendering so the generated response is spoken and the avatar mouth matches the audio in realtime.

The important detail is that transcription is not the same thing as turn completion. A streaming recognizer may emit words continuously, but your agent should usually wait for endpointing or a pause heuristic before it produces a response. If you trigger generation on every partial transcript, you get a chattery agent that interrupts itself.

For avatar quality, the lip-sync source of truth is the audio stream being played back, not the transcript. The transcript drives language understanding; the synthesized audio drives facial motion. Keep those concerns separate and you avoid a lot of debugging pain.

2. Speech recognition: use partials, but treat them as unstable

Modern streaming ASR services typically give you a sequence of interim hypotheses and then a final transcript. Interim results are useful for responsiveness: you can show live captions, update intent detection, or prefetch downstream context. But they are inherently unstable, especially near clause boundaries, proper nouns, and noisy audio.

A practical integration usually looks like this:

async for event in asr_stream(audio_frames):
await agent.respond(transcript_buffer.final_text())

async for event in asr_stream(audio_frames):
await agent.respond(transcript_buffer.final_text())

async for event in asr_stream(audio_frames):
await agent.respond(transcript_buffer.final_text())

There are three implementation details worth getting right:

• Chunk size matters. Sending tiny audio frames reduces latency but increases overhead. Very large frames add recognition delay. In practice, 20–100 ms frames are a reasonable starting point.

• Endpointing should be explicit. Relying only on a final ASR result can make the assistant feel sluggish. Use silence detection, voice activity detection, or your ASR provider’s endpoint events if available.

• Do not overwrite committed text with partials. Maintain separate buffers for interim and final text so the agent never reasons over unstable text as if it were final.

If you are building a browser-based client, the browser should capture microphone audio and stream it to your backend or agent runtime. Avoid round-tripping through full file uploads; speech recognition wants continuous audio, not blobs.

3. Turn-taking and interruption handling are where most systems fail

The hard part is not transcription; it is deciding who has the floor. In a voice conversation, the user may pause mid-sentence, continue after a beat, or interrupt while the avatar is speaking. Your system needs a policy for all three cases.

A sane default is:

• Buffer interim transcript updates while the user is speaking.

• Mark the user turn complete after a short silence threshold or explicit endpoint signal.

• Send the final text to the agent/LLM only once per completed turn.

• If the user starts speaking while the avatar is talking, stop synthesis playback or mark the response as interrupted.

That last point matters for lip sync. If your synthesis keeps playing after you have already accepted a new user turn, the avatar will keep moving its mouth for a response that the conversation has effectively abandoned.

Here is a simplified control flow:

state = {

state["assistant_speaking"] = False

state = {

state["assistant_speaking"] = False

state = {

state["assistant_speaking"] = False

In production, you will likely add a timeout, a conversation memory store, and explicit cancellation tokens for any in-flight LLM or TTS work. Without cancellation, you end up synthesizing answers that nobody will hear.

4. Syncing avatar motion with speech output

For a realtime avatar, the cleanest model is that the avatar follows the audio stream being rendered to the user. That gives you deterministic timing: if the audio plays, the lips move with it; if the audio is interrupted, the face stops accordingly.

This is why it is usually better to send synthesized audio into the avatar pipeline rather than trying to animate purely from text. Text-to-viseme mapping is useful for offline animation, but in a live system it introduces a second timing model and another place for drift.

To keep the experience stable:

• Preserve ordering. The response text, the TTS output, and the avatar playback event should all belong to the same turn.

• Keep latency bounded. If generation takes too long, the avatar will sit motionless after the user finishes speaking. Prewarm models and keep transports persistent.

• Handle partial responses carefully. If you stream assistant text incrementally, make sure your TTS and avatar stack can either stream the audio incrementally too or buffer until the utterance boundary is known.

In practice, the avatar layer should not care whether the upstream text came from an LLM, a rules engine, or a human operator. It only needs the audio and enough session state to stay synchronized.

How Protoface fits in

This is the part where Protoface is useful: it gives you a developer-facing realtime avatar layer you can drop into an existing voice agent without having to build the face transport, session plumbing, or lip-sync handling yourself. If you are already using LiveKit, the quickstart examples are a good reference for wiring an agent pipeline into a synchronized video face.

If your stack is Python-first, the LiveKit plugin approach is usually the shortest path. The idea is simple: your agent keeps doing speech recognition, reasoning, and synthesis, while the plugin attaches a Protoface avatar to the voice session so the response is rendered as a talking face.

# illustrative only; exact setup depends on your LiveKit agent stack

agent.attach_avatar(avatar)

# illustrative only; exact setup depends on your LiveKit agent stack

agent.attach_avatar(avatar)

# illustrative only; exact setup depends on your LiveKit agent stack

agent.attach_avatar(avatar)

If you are managing avatars or sessions programmatically, the REST API and Python SDK are the right surfaces. You would typically create an avatar in advance, start a realtime session when the call begins, and keep the session lifecycle aligned with your agent turn state. The exact request/response fields are documented in the API reference, so treat this as shape-only example code:

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"ava_123","quality":"standard"}'

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"ava_123","quality":"standard"}'

curl -X POST https://api.protoface.com/sessions \
-d '{"avatar_id":"ava_123","quality":"standard"}'

Two practical notes:

• Keep API keys on the server. Use bearer auth from your backend only; do not expose keys in the browser.

• Choose quality tiers intentionally. Higher quality can cost more and may have different latency characteristics, so match the tier to the interaction.

For implementation details, the docs are the best place to start: docs.protoface.com.

Implementation checklist and common gotchas

If you are integrating live speech recognition into a voice avatar for the first time, this checklist will save time:

• Use streaming audio transport, not file-based uploads.

• Separate interim ASR state from committed transcript state.

• Endpoint on silence or explicit ASR completion, not just on the last partial transcript you happened to receive.

• Cancel in-flight assistant generation when the user barges in.

• Drive lip sync from the synthesized audio stream, not from raw text.

• Persist session state across the full conversation turn so avatar playback and agent response stay paired.

One subtle bug is buffering too aggressively to “improve accuracy.” If you add several hundred milliseconds of buffering before ASR or TTS, the system may become more accurate but feel dead. For interactive avatars, perceived latency matters as much as model quality.

Conclusion

Streaming live speech recognition into a realtime voice avatar is a coordination problem: capture audio continuously, treat partial transcripts as provisional, detect turn boundaries carefully, and keep avatar playback synchronized with the actual synthesized audio. Once those pieces are explicit in your architecture, the rest is mostly integration work.

If you want to avoid building the avatar transport and lip-sync layer from scratch, start with the relevant Protoface surface for your stack: the LiveKit plugin, the REST API, or the Python SDK. The docs at docs.protoface.com and the quickstarts on GitHub are the fastest path from prototype to working system.