Introduction

Low-latency speech-to-text for a talking avatar is mostly a systems problem: you need to get microphone audio into your backend quickly, run partial recognition often enough to feel responsive, and preserve ordering so the avatar can speak naturally without stepping on the user. If any one part is slow or lossy, the interaction feels “chatty” rather than conversational.

This post walks through a practical WebSocket-based STT pipeline for realtime avatar apps. By the end, you should be able to design an audio path that:

• captures small audio frames from the browser or client

• streams them incrementally to a backend STT service over WebSocket

• emits interim and final transcripts with low jitter

• feeds those transcripts into an avatar or voice agent without blocking the render path

The examples assume you are building around Protoface-style realtime avatars, but the pipeline patterns are broadly applicable.

What “low latency” actually means in this pipeline

For conversational UI, latency is not a single number. You usually care about three separate intervals:

1. Capture-to-send: how long it takes before recorded audio leaves the browser or client.

2. Send-to-interim transcript: how quickly your STT service can produce partial text.

3. Transcript-to-avatar reaction: how quickly your agent can decide whether to keep listening, barge in, or start speaking.

The trick is to optimize the whole path, not just the recognizer. In practice, the largest wins come from:

• sending small chunks, typically 20–100 ms of audio per frame

• keeping the transport persistent with WebSocket, rather than opening per-utterance requests

• handling partial results as first-class events, not as “nice to have” UI hints

• making finalization deterministic with VAD, explicit end-of-speech markers, or both

For an avatar, you also need the output side to stay synchronized. Transcripts often drive LLM turn-taking, and turn-taking drives whether the avatar remains idle, nods, or starts producing audio/video. That means the STT layer should expose a stable sequence of events, not just a blob of text at the end.

Capture audio in small frames and keep the stream continuous

Browser audio capture is where many implementations accidentally add 300–500 ms of delay. The main rules are straightforward:

• Prefer a continuous audio stream over stop/start recording cycles.

• Use a small frame size. 20 ms is a common default; 40–60 ms is a reasonable compromise if CPU overhead matters.

• Use a stable sample format and sample rate end-to-end, or resample once at the edge.

• Do not wait for full sentences before sending anything.

If you are using the Web Audio API in the browser, your job is to extract PCM frames and push them over a persistent socket. The client should not know anything about STT state beyond whether the connection is open and whether backpressure is building.

// Illustrative client-side shape: send raw PCM frames over WebSocket.

}

// Illustrative client-side shape: send raw PCM frames over WebSocket.

}

// Illustrative client-side shape: send raw PCM frames over WebSocket.

}

Two practical gotchas:

• Backpressure: if your socket buffers grow, drop or coalesce frames rather than letting latency drift. A realtime avatar is better off with a slightly degraded transcript than with stale audio.

• Jitter: if your input source is bursty, normalize frame size before sending. STT models behave better with evenly paced input.

Design the WebSocket protocol around events, not just bytes

A raw byte stream is fine for audio transport, but the backend and downstream agent need event boundaries. A clean WebSocket protocol usually includes:

• audio frames carrying encoded PCM or Opus payloads

• speech_start and speech_end markers, from client VAD or server-side VAD

• partial_transcript events for interim recognition

• final_transcript events when an utterance is committed

• error and close semantics that allow clean recovery

For latency-sensitive systems, the important design choice is where speech boundaries are decided. Client-side VAD can reduce wasted uplink and make the UI feel snappier, but server-side VAD is more consistent across devices and network conditions. Many production systems do both: the client hints early, the server confirms.

Sequence numbers help too. When the network is lossy or the client reconnects, a monotonically increasing frame index lets the server detect gaps without guessing. If you are feeding transcripts into an agent loop, sequence order matters more than perfect delivery; do not merge chunks opportunistically unless you can preserve utterance boundaries.

Turn transcripts into avatar behavior without blocking the media path

The STT service should not directly “control the avatar” in the sense of holding the rendering loop hostage. Instead, it should emit events that a separate orchestrator consumes. That orchestrator decides whether to:

• keep listening while interim text is changing

• trigger an LLM once a final transcript arrives

• cancel or interrupt avatar speech if the user barges in

• attach metadata like speaker identity, confidence, or timestamps

This separation keeps the media path deterministic. A good pattern is:

1. audio frames arrive over WebSocket

2. STT emits partial text as soon as it has enough evidence

3. final text commits an utterance boundary

4. the agent layer decides whether to answer, clarify, or stay silent

5. the avatar layer renders whatever the agent decides, independently of transcription

That means your STT layer should never assume a transcript is a complete turn until it is explicitly finalized. Partial transcripts are useful for responsiveness, but they are not stable enough to drive long-lived state.

# Illustrative Python backend consumer.

asyncio.run(handle_stream())

# Illustrative Python backend consumer.

asyncio.run(handle_stream())

# Illustrative Python backend consumer.

asyncio.run(handle_stream())

A small but important implementation detail: keep transcript assembly idempotent. Partial results often repeat previous words with small corrections. Your UI or agent should replace the current interim segment, not append every delta blindly.

Where Protoface fits: avatar sync and voice-agent integration

If your goal is specifically a talking avatar, the easiest place to connect the pipeline is at the agent boundary rather than inside the audio transport. Protoface exposes developer surfaces for sessions and avatars, and the LiveKit plugin is the most direct fit if you already have a voice agent running in LiveKit. In that setup, your STT pipeline feeds the agent, and the agent’s speech output is mirrored by the avatar in sync.

The plugin is useful because it keeps the avatar aligned with the voice agent instead of making you manually coordinate lip sync, session lifecycle, and media state. If you are already using LiveKit Agents, the integration path is small. For example, the plugin package can be added alongside your existing agent code, and the avatar follows the agent’s spoken output rather than requiring a separate rendering control loop. See the plugin repository and examples in the GitHub org and the integration notes in the documentation.

# Illustrative Python shape for a LiveKit agent with an avatar plugin.

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

# Illustrative Python shape for a LiveKit agent with an avatar plugin.

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

# Illustrative Python shape for a LiveKit agent with an avatar plugin.

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

If you prefer to manage sessions directly, the REST API gives you a standard backend integration point. A minimal call usually looks like this:

curl -X POST https://api.protoface.com/<relevant-endpoint> \
-d '{"name":"support-avatar"}'

curl -X POST https://api.protoface.com/<relevant-endpoint> \
-d '{"name":"support-avatar"}'

curl -X POST https://api.protoface.com/<relevant-endpoint> \
-d '{"name":"support-avatar"}'

Keep the API key server-side. The browser-facing embed option is the right choice when you want to avoid shipping credentials at all, but for an STT pipeline inside your own app, the backend should own session creation and any stateful orchestration.

Operational details that matter in production

A low-latency pipeline can still feel bad if operational concerns are ignored. A few things to check before shipping:

• Reconnect behavior: if the WebSocket drops mid-utterance, decide whether to replay buffered audio or force a fresh turn.

• Timeouts: silence detection should be explicit; do not rely on arbitrary socket idle close behavior.

• Quality tiers: if your platform bills by quality tier, route high-value sessions accordingly and monitor actual user-perceived latency, not just cloud cost.

• Observability: log frame timestamps, partial/final transcript times, and turn transitions so you can find where delay accumulates.

If you have an in-browser playground or a developer dashboard, use it to compare real microphone behavior across devices. The hard bugs are usually device-specific: mobile browsers resample aggressively, laptop mics drift, and some Bluetooth headsets introduce unpredictable buffering.

One final recommendation: keep the STT service, agent orchestration, and avatar rendering as loosely coupled as possible. The more each component can fail or reconnect independently, the easier it is to keep the overall conversation responsive.

Conclusion

The core recipe is simple: stream small audio frames continuously over WebSocket, emit partial transcripts early, finalize turns deterministically, and hand the results to an agent layer that stays separate from avatar rendering. That division of responsibility is what keeps a talking avatar feeling realtime instead of laggy.

If you are implementing this with Protoface, start with the integration surface that matches your stack: the LiveKit plugin for a voice-agent-centric architecture, the REST API or Python SDK for backend session control, or the iframe embed if you want a browser-only integration. The docs at docs.protoface.com are the right place to confirm exact fields, event names, and current examples.