Introduction

If you want a realtime avatar to speak as soon as your text-to-speech system starts producing audio, the main problem is not “how do I play a file?” It’s how to move synthesized speech across process boundaries with low latency, preserve chunk order, and keep the avatar playback aligned with the audio timeline. In practice, that means streaming TTS over a WebSocket, not waiting for a full MP3 or WAV payload to finish before playback begins.

By the end of this post, you should be able to design a TTS-to-avatar pipeline that:

• accepts partial TTS audio chunks over WebSocket,

• buffers just enough to avoid underruns without adding avoidable delay,

• keeps timestamps and sequencing sane, and

• hands the stream off to a realtime avatar player with synchronized lip movement.

I’ll use Protoface as the concrete example of the avatar side, but the streaming patterns here apply to any realtime voice agent stack.

Why stream TTS instead of sending complete audio files?

The key reason is latency. A conversation feels realtime only when the system can start speaking within a few hundred milliseconds of the user finishing an utterance. If you generate the entire speech audio first, then upload it, then decode it, you’ve already spent the time budget.

Streaming TTS changes the shape of the problem:

1. Your LLM produces text incrementally or in one shot.

2. Your TTS service begins emitting audio chunks as soon as it can synthesize the first phonemes.

3. Your client or media service decodes and buffers those chunks immediately.

4. The avatar playback layer consumes the same audio timeline so mouth motion tracks the sound.

That separation is important: the WebSocket is not “for video.” It is the transport for the audio stream, plus whatever metadata you need to keep that stream ordered and synchronized.

What actually needs to travel over the socket

Most TTS streaming protocols end up carrying some variation of the same message types:

• start: identifies the utterance, voice, sample rate, codec, and any session metadata.

• audio chunk: raw PCM, Opus frames, or encoded bytes, with a monotonically increasing sequence number.

• flush/end: signals that no more audio is coming for this utterance.

• error: reports synthesis failure or transport issues.

For realtime avatar playback, the details that matter most are:

• Codec: PCM is simplest but large; Opus is much smaller and common in realtime systems.

• Chunk size: smaller chunks reduce latency but increase overhead and jitter sensitivity.

• Sample rate: pick one rate end-to-end if you can. Transcoding in the middle adds delay.

• Ordering: sequence numbers are mandatory if you care about deterministic playback.

Binary WebSocket frames are usually the right choice for audio payloads. Keep control messages in JSON text frames or a compact binary envelope. Either way, define the contract explicitly; “send some bytes” is not enough once there is a user-facing avatar on the other end.

Designing the stream for low-latency playback

There are three buffers in play: the TTS engine’s internal synthesis buffer, the transport buffer in the socket stack, and the playback buffer in the avatar client. If any of them grows too large, you lose the realtime feel. If any of them is too small, you get underruns, stutter, or gaps between phonemes.

In practice, a good starting point is:

• Emit audio as soon as it is decodable, not when a paragraph is done.

• Keep chunks consistent in duration or byte size.

• Use backpressure so the sender slows down if the consumer falls behind.

• Bound the playback buffer to a small multiple of your chunk duration.

If you are streaming PCM, you can treat each chunk as a fixed number of frames and schedule playback precisely. If you are streaming compressed audio, the decoder may introduce variable delay, so you need a little more buffering and more careful timestamping. The engineering trade-off is straightforward: PCM is easier to reason about; Opus is more bandwidth-efficient and usually the better choice across the network.

A minimal WebSocket shape for streaming TTS

Here is a compact protocol sketch. The exact message fields are up to you, but the structure should look familiar:

{ "type": "start", "utterance_id": "u_123", "sample_rate": 48000, "codec": "opus" }
{ "type": "end", "utterance_id": "u_123" }

{ "type": "start", "utterance_id": "u_123", "sample_rate": 48000, "codec": "opus" }
{ "type": "end", "utterance_id": "u_123" }

{ "type": "start", "utterance_id": "u_123", "sample_rate": 48000, "codec": "opus" }
{ "type": "end", "utterance_id": "u_123" }

On the server side, you should validate that:

• chunks for a given utterance arrive in order,

• an utterance is only active once,

• the codec and sample rate stay fixed for the lifetime of the stream, and

• the client acknowledges or can tolerate dropped chunks if the connection degrades.

If you need to multiplex multiple utterances or speakers over one socket, include a session identifier and keep the playback state machine explicit. Otherwise, you’ll eventually mix audio from an interrupted response into the next one.

Server-side implementation pattern

On the backend, the best practice is to treat the TTS generator as a producer and the WebSocket as a bounded consumer. Don’t accumulate the whole utterance in memory unless you truly need reprocessing later.

A simple Python asyncio sketch:

import asyncio

await websocket.send(json.dumps({"type": "end"}))

import asyncio

await websocket.send(json.dumps({"type": "end"}))

import asyncio

await websocket.send(json.dumps({"type": "end"}))

That example is intentionally minimal. In production, you usually want a small queue between the TTS task and the socket writer so temporary synthesis bursts do not block the upstream agent. You also want structured logging around utterance IDs, chunk counts, and send latency. If the avatar sounds wrong, these are the first metrics that tell you whether the issue is generation, transport, or playback.

Client-side playback: buffer just enough, then start

On the receiving side, the hardest part is deciding when to begin playback. Start too early and you risk underruns; start too late and the avatar visibly lags the speech. A practical pattern is to wait until you have a small prebuffer, then play continuously while maintaining a narrow buffer window.

The state machine is usually:

1. Receive start and initialize decoder state.

2. Accumulate audio until you have enough to survive transient jitter.

3. Begin playback and keep decoding subsequent chunks in order.

4. On end, drain the remaining buffer, then release the utterance.

For avatar systems, playback and lip sync are coupled. The avatar renderer should be driven by the same audio clock used for speaker output, not by wall-clock arrival time of chunks. That distinction matters when the network jitters: audio should stay smooth, and the mouth should follow the audio timeline, not the packet timeline.

Common failure modes and how to avoid them

A few mistakes show up repeatedly:

• Variable chunk sizes: makes buffering and lip-sync prediction harder.

• Re-encoding midstream: adds latency and may shift timing.

• No sequence numbers: impossible to recover from out-of-order delivery.

• Overbuffering: the speech is correct but the interaction feels delayed.

• Ignoring backpressure: the sender outruns the receiver and memory grows until something drops.

Another subtle issue is cancellation. Users interrupt voice agents all the time. Your TTS stream should be abortable, and the avatar playback layer must be able to stop the current utterance cleanly and transition to the next one without finishing stale audio in the background.

How Protoface fits into this pipeline

The avatar side of the problem is handled by Protoface, which is designed to drop a synchronized talking face into realtime voice systems. If you already have a WebSocket-based TTS stream, the integration point is the agent or session layer that consumes that audio and drives the avatar playback.

In practice, developers usually wire this up through the LiveKit Agents plugin or the Python SDK, depending on where their voice stack already lives. The plugin path is especially useful when you are already running a LiveKit voice agent and just need a lip-synced face with minimal glue. See the examples in the plugin repository and the docs at docs.protoface.com for the exact session and media wiring.

For a quick API-driven session setup, the REST surface looks like any other authenticated control plane:

curl -X POST https://api.protoface.com/<session-or-avatar-endpoint> \
-d '{"name":"support-agent-session"}'

curl -X POST https://api.protoface.com/<session-or-avatar-endpoint> \
-d '{"name":"support-agent-session"}'

curl -X POST https://api.protoface.com/<session-or-avatar-endpoint> \
-d '{"name":"support-agent-session"}'

The exact endpoint and payload depend on what you are creating, so treat that as illustrative rather than copy-pasteable. The useful part is the shape: create or configure the avatar/session on the server, then connect your streaming TTS source to the realtime media session that renders the face.

Conclusion

Streaming TTS over WebSocket is less about “sending audio live” and more about maintaining a disciplined realtime contract: ordered chunks, fixed codec parameters, bounded buffering, and explicit cancellation. Once you have that, the avatar layer becomes straightforward because it can render against a stable audio clock instead of chasing late-arriving text or files.

If you’re implementing this today, start with a narrow protocol, make the playback buffer observable, and test interruption paths early. Then connect the stream to your avatar runtime and verify that audio start time, mouth motion, and stop/cancel behavior all line up under jitter.

For concrete integration details and current SDK examples, use the docs and quickstarts at docs.protoface.com and the relevant repositories on GitHub.