Introduction

Continuous speech recognition is the difference between a voice agent that feels responsive and one that feels like a push-to-talk demo. In a live avatar workflow, it also determines whether the face can stay synchronized with the speaker’s intent: partial transcripts drive turn-taking, barge-in, and visual feedback before the user finishes a sentence.

This post walks through the practical pieces you need to wire up continuous ASR in front of a realtime avatar. By the end, you should be able to reason about the audio stream, stream partial hypotheses into your agent loop, handle end-of-utterance cleanly, and connect the result to a video avatar without adding latency you can’t explain.

What “continuous” means in practice

Continuous speech recognition is not just “transcribe a long recording.” In an interactive system, audio arrives incrementally, and the recognizer emits a sequence of partial and final results while the user is still speaking. Your application typically consumes three kinds of events:

• Partial transcript: low-latency, unstable text used for UI and early intent detection.

• Final transcript: committed text for agent logic and logging.

• Endpoint / turn detected: signal that the user likely stopped speaking.

The important design constraint is that these signals are probabilistic. Partial text can change. Endpoint detection can fire too early. If you treat either one as ground truth, you’ll create awkward interruptions or missed turns.

For a live avatar, the pipeline usually looks like this:

1. Microphone audio is captured in small frames, often 20–30 ms.

2. Frames are encoded and sent over a realtime transport such as WebRTC.

3. ASR processes the stream and yields incremental hypotheses.

4. The agent decides when to respond, interrupt, or keep listening.

5. Avatar rendering receives the agent’s spoken output and lip-syncs it.

The only way this feels natural is if recognition latency, agent latency, and speech synthesis latency are all controlled together. Don’t think of ASR as a standalone feature; it is part of the turn-taking system.

Building the audio pipeline cleanly

If you are implementing this yourself, start by separating three concerns: capture, recognition, and orchestration.

Capture should produce a stable stream of PCM frames at the sample rate your ASR expects. Avoid buffering large chunks “for efficiency”; that just raises your floor latency. Small frames are better, as long as you don’t starve the recognizer.

Recognition should be a streaming service, not a batch job. You want incremental results and endpointing. Many stacks expose a concept like voice activity detection (VAD) or a speech endpointer. That is useful, but you should treat it as a hint rather than a hard boundary.

Orchestration should own the conversation state. This is where you decide whether a partial transcript is enough to start intent classification, whether to prefetch a tool call, or whether to wait for a final hypothesis before speaking. A practical pattern is:

• Use partials for UI updates and speculative classification.

• Use finals for authoritative downstream actions.

• Use endpoint detection to start the response timer, not to finalize the meaning of the turn.

Minimal streaming ASR loop in Python

The exact SDK calls depend on your ASR provider and transport, but the structure is always similar. Here is a compact example of the control flow you want in a streaming agent:

async for event in asr.stream(audio_frames):

maybe_start_response_timer()

async for event in asr.stream(audio_frames):

maybe_start_response_timer()

async for event in asr.stream(audio_frames):

maybe_start_response_timer()

There are two gotchas here:

• Do not overwrite your committed user message with every partial. Keep partial state separate.

• Do not assume an endpoint means the user is done. In live speech, short pauses happen inside a single turn.

If you are adding barge-in, the agent should be able to stop speaking when new user audio is detected. That requires the ASR/VAD side to emit a reliable “user started talking again” event, and your speech synthesis path must be interruptible.

Latency, endpointing, and transcript stability

Continuous recognition feels good only when the unstable parts are contained. Three tuning knobs matter most:

• Chunk size: smaller chunks lower latency but increase overhead.

• Endpoint threshold: too aggressive and you cut users off; too lax and the agent lags.

• Stability policy: some recognizers emit “final-ish” partials before committing; use those carefully.

In a real conversation, “end of utterance” is a UX decision as much as a signal-processing decision. For example, a user saying “well, I guess…” may pause for 400 ms and then continue. If you respond immediately, you’ll feel impatient. A common approach is to combine endpoint detection with a short grace window and only finalize once either:

• the silence threshold is exceeded, or

• the next action is cheap and reversible, so you can speculate safely.

Also remember that recognition accuracy and latency trade off. Higher quality tiers usually mean more compute or larger models, which can improve transcripts but may increase turnaround time. For a voice avatar, the best configuration is the one that keeps turn latency low enough that the face and the spoken response feel coupled.

Where the avatar fits: streaming speech into a realtime face

This is the point where many teams get tangled up: they solve ASR, then discover the avatar needs a slightly different notion of “speech start,” “speech end,” and “interrupt.” A realtime avatar should not be driven only by final transcripts. It should be driven by the same stream that powers the agent’s turns.

One practical way to avoid glue code is to keep the voice agent in a realtime transport and attach the avatar at the agent layer. With the LiveKit agent path, that means the avatar sees the same conversational timing as the agent rather than a replayed text feed. The benefit is simple: the face can start talking when the agent starts speaking, stop when the audio is interrupted, and remain synchronized with the live session instead of lagging behind transcript events.

If you are building with LiveKit, the livekit-plugins-protoface plugin is the relevant integration point. You install it into your agent process and let the agent manage the transport, while the avatar handles the synchronized video face. For setup details and examples, see the plugin repository and the public docs at docs.protoface.com.

# illustrative only; exact config depends on your agent

)

# illustrative only; exact config depends on your agent

)

# illustrative only; exact config depends on your agent

)

The main thing to preserve is event ordering: user audio in, partials out, final transcript committed, agent response generated, avatar speech rendered. If you add extra queues between those stages, keep them bounded and observable. Unbounded buffering is how realtime systems become “eventually realtime.”

REST and session management when you need control

If your workflow involves provisioning avatars, creating sessions, or managing access server-side, the REST API is the right place to do it. The API is authenticated with bearer keys, so keep calls on the backend and never expose credentials to the browser.

curl https://api.protoface.com/<endpoint> \
-d '{"name":"support-avatar"}'

curl https://api.protoface.com/<endpoint> \
-d '{"name":"support-avatar"}'

curl https://api.protoface.com/<endpoint> \
-d '{"name":"support-avatar"}'

I’m intentionally leaving the endpoint path generic here; use the exact resource names from the docs because they may vary by object type and release. The useful pattern is the same regardless: create the avatar/session on the server, store the identifiers in your app, and let the frontend connect only to the ephemeral session state it needs.

If you prefer Python, the SDK gives you a cleaner place to put that provisioning logic than raw HTTP calls. For teams with their own control plane, that tends to be the best split: REST or SDK for setup, realtime transport for the conversation itself.

from protoface_sdk import ProtofaceClient

session = client.sessions.create(avatar_id=avatar.id)

from protoface_sdk import ProtofaceClient

session = client.sessions.create(avatar_id=avatar.id)

from protoface_sdk import ProtofaceClient

session = client.sessions.create(avatar_id=avatar.id)

Common implementation mistakes

A few failure modes show up repeatedly in voice-avatar systems:

• Feeding transcripts directly into rendering: text and speech timing are not the same thing.

• Finalizing too early: endpoint detection is not certainty.

• Ignoring barge-in: users will interrupt, especially when the agent is wrong.

• Letting buffers grow: realtime systems degrade quietly when queues back up.

• Mixing UI state and conversation state: partial transcripts should not mutate committed history.

The best debugging tool is observability. Log partials, finals, endpoint events, response start, response stop, and interruption causes with timestamps. Once you can see those markers, latency problems become much easier to localize.

Conclusion

Continuous speech recognition in a live avatar workflow is mostly about respecting stream semantics: partials are provisional, finals are committed, and endpointing is advisory. If you keep those roles separate, your agent can respond quickly without sounding jumpy, and your avatar can stay synchronized with the conversation rather than chasing it.

For implementation details, follow the public docs at docs.protoface.com and use the relevant integration surface for your stack: the LiveKit plugin for agent-driven realtime video faces, the REST API for provisioning, or the Python SDK for server-side orchestration. If you want to see the available quickstarts and patterns in one place, start from the linked repositories in the project README and adapt the one that matches your voice stack.