Introduction

Realtime avatars are only useful if their mouth movement, timing, and perceived attention match the conversation closely enough that users stop noticing the transport. That means your text-to-speech or ASR stack cannot be treated as a loose sidecar. In a voice-agent pipeline, automatic speech recognition (ASR/STT) is part of the control loop: it turns incoming audio into text quickly enough for the agent to respond, while the avatar layer uses that same timing to stay synchronized with the speaker or the agent’s generated speech.

This post explains where STT belongs in a realtime avatar architecture, what “streaming” actually means in practice, and how to think about latency, chunking, partial results, and turn boundaries. By the end, you should be able to reason about a pipeline that combines audio transport, ASR, LLM inference, and avatar rendering without introducing awkward lag or lip-sync drift.

What ASR does in a realtime avatar pipeline

In a standard conversational system, ASR has one job: convert microphone audio into text. In a realtime avatar pipeline, it has at least three jobs:

• Transcription: produce words the LLM or dialog manager can consume.

• Turn detection: help decide when the user has finished speaking, especially when endpointing is based on silence or partial-confidence signals.

• Timing signal: provide enough low-latency structure that downstream systems can start acting before the utterance is fully complete.

The key point is that ASR is not just a “preprocessing” step. If you wait for a final transcript before doing anything else, you add avoidable latency. In a realtime avatar, that extra delay is visible on the face.

A practical pipeline looks like this:

microphone audio
-> client playback

microphone audio
-> client playback

microphone audio
-> client playback

Whether your avatar is a 2D talking face, a generated video face, or a composited headshot, the timing constraints are similar. The avatar must track the spoken audio stream, not a delayed transcript. The transcript mainly drives the agent’s reasoning; the audio stream drives the mouth.

Streaming ASR: partials, finals, and endpointing

“Streaming ASR” usually means the recognizer emits results while audio is still arriving. That sounds obvious, but there are a few implementation details that matter a lot.

Partial transcripts are useful but unstable

Partial results are speculative. Early words may change as the recognizer sees more context. That is normal. The system consuming them should treat partials as hints, not truth. Use them for:

• Low-latency UI feedback, such as “hearing…” indicators.

• Early intent classification when your application can tolerate revision.

• Prefetching or warming downstream components.

Do not use partials as durable conversation state unless your application is designed for correction.

Final transcripts are the commit point

Final results are the point at which the ASR engine believes an utterance segment is complete. In many systems, finalization is driven by endpointing: silence detection, max segment length, or a combination of acoustic and language-model signals. Once a final transcript arrives, your orchestration layer can safely hand the text to the LLM or apply business logic.

Endpointing is where latency is won or lost

For conversational avatars, endpointing is often more important than raw recognition accuracy. If the ASR waits too long to finalize, the user sees an agent “thinking” after they stop speaking. If it finalizes too early, it cuts off the user and creates turn-taking errors. The right threshold depends on the use case:

• Support bots: favor reliable finalization over aggressiveness.

• Sales or demo agents: favor quicker turn detection so the interaction feels responsive.

• Interruptible assistants: need very careful handling of barge-in and resume behavior.

A good mental model is to separate the audio stream into three layers:

1. Transport moves frames reliably and with low jitter.

2. Recognition converts frames into text incrementally.

3. Conversation control decides when an utterance is “done enough” to act on.

Confusing those layers is a common source of latency bugs.

How ASR affects lip sync and avatar behavior

For a realtime avatar, the transcript itself does not drive the mouth. The audio that will be spoken does. But ASR still shapes what the avatar looks like because it controls the timing of the response.

Here is the practical sequence for a user speaking to an agent:

1. The client streams microphone audio.

2. ASR emits partials quickly, then a final transcript.

3. The agent decides on the response.

4. TTS generates response audio.

5. The avatar layer renders the face and aligns mouth movement to that audio.

If ASR is slow, the avatar waits longer before speaking, which users interpret as sluggishness. If ASR is too eager and the agent interrupts too soon, the avatar appears to cut off the user. Either way, the visible artifact is on the face, even though the root cause is the recognizer.

There are a few gotchas worth calling out:

• Audio chunk size matters. Smaller chunks reduce latency but increase protocol overhead and sensitivity to jitter.

• Clock drift matters. If your transport and renderer do not agree on timing, you can get subtle desynchronization over longer sessions.

• Token timing is not mouth timing. Do not confuse token emission with phoneme timing; the avatar should track audio playback, not transcript arrival.

• Interruptions need explicit handling. If the user barges in, you usually need to stop or duck TTS and clear any pending avatar speech state.

Implementation pattern: stream audio, don’t batch it

In practice, the right design is to push audio frames continuously and let the ASR service return incremental events. Even if your upstream UI only exposes “recording started” and “recording stopped,” the backend should still process audio as a stream.

That usually means:

• Using a WebRTC or similar low-latency transport for live sessions.

• Forwarding frames to your ASR engine as they arrive.

• Handling partial transcripts asynchronously.

• Starting response generation as soon as the final segment is available.

If your stack supports intermediate endpointing, you can sometimes begin planning a response before the final transcript lands. That is useful for agentic workflows, but it should be treated as an optimization, not a correctness requirement.

A minimal Python example for a realtime session

If you are managing sessions programmatically, the basic shape is: create an avatar/session, keep an authenticated server-side call, and wire the returned session into your realtime stack. The exact fields depend on your configuration, but the request flow is straightforward.

import requests

print(session)

import requests

print(session)

import requests

print(session)

Use this pattern when your backend is responsible for orchestration, provisioning, or session lifecycle management. Keep the API key server-side; do not ship it into the browser.

Where the LiveKit plugin fits

If your voice agent already runs on LiveKit, the cleanest way to add a talking face is to attach the avatar at the agent layer rather than bolting on a separate rendering path. The quickstarts and the LiveKit plugin approach are useful because they keep the audio path and the avatar path in the same realtime session.

Conceptually, the agent still does the same three things: listens to audio, gets a transcript, and produces speech. The plugin’s job is to synchronize the avatar with that speech so the face moves when the agent talks and stays quiet when it listens.

# illustrative only; exact import names and args are in the docs

)

# illustrative only; exact import names and args are in the docs

)

# illustrative only; exact import names and args are in the docs

)

That integration is useful when you want a voice agent with a visible presence and you already trust LiveKit for your media transport. If you want the gory details, the relevant docs are on docs.protoface.com and the plugin examples are in the integration repo.

Operational concerns: auth, rate limits, and observability

Once you move from a local demo to a production avatar pipeline, a few non-obvious concerns show up immediately.

• Authentication: keep API keys on the server. For REST calls, use bearer auth and rotate keys like any other secret.

• Session lifecycle: make sure sessions are explicitly created, tracked, and cleaned up. Zombie sessions can burn quota and confuse analytics.

• Latency measurement: track ASR partial latency, finalization latency, TTS start latency, and end-to-end turn time separately.

• Rate limiting: especially for public-facing embeds, you need per-origin or per-IP controls so abuse does not turn into cost spikes.

For debugging, instrument the pipeline with timestamps at the boundaries between transport, ASR, response generation, and playback. A transcript arriving 250 ms faster is only meaningful if you can see whether the avatar actually started speaking sooner.

How Protoface fits without changing the architecture

The main reason to use Protoface here is not that it replaces ASR. It does not. The value is that it gives you a clean avatar surface to plug into an existing voice stack: via REST for session management, via the Python SDK for backend orchestration, or via a LiveKit plugin when you already have realtime voice infrastructure.

In other words, ASR still belongs to the speech pipeline you already run or choose separately. Protoface sits at the avatar layer and consumes the realtime state that comes out of that pipeline. That separation is healthy: it keeps recognition concerns, media transport, and avatar rendering decoupled enough that you can swap ASR vendors or turn-taking policies without rewriting your visual layer.

Conclusion

Streaming ASR is the timing backbone of a realtime avatar system. Partial results help you stay responsive, final results anchor turn boundaries, and careful endpointing keeps the interaction natural. The avatar itself should track the spoken audio, but the transcript timing determines when that audio starts, stops, or gets interrupted.

If you are building a voice agent, customer-support bot, game NPC, or embedded web experience, start by making the audio path truly streaming and by measuring latency at each handoff. Then connect the avatar layer cleanly, rather than treating it as an afterthought. For implementation details, see the docs and the relevant quickstarts in the linked repos.