Introduction

If you are building a phone agent, the hard part is not just speech-to-text or text-to-speech in isolation. The hard part is keeping everything synchronized under real network and model latency: audio comes in as a stream, STT arrives incrementally, the agent decides what to say next, TTS starts producing audio before the full response is known, and the user expects the experience to feel continuous. Once you add an avatar, there is a fourth stream to keep aligned: lip motion must track the synthesized speech closely enough that the face looks intentional rather than “close enough.”

This post walks through the streaming mechanics that make that work in practice. By the end, you should understand how to wire a low-latency voice pipeline for phone agents, what to watch for when streaming STT and TTS, and where an avatar layer fits into the architecture without turning the system into a timing mess.

Start with the pipeline, not the vendor

A useful mental model is:

phone audio → telephony/media bridge → streaming STT → agent turn logic → streaming TTS → avatar lip-sync/video rendering

Each hop has different latency and buffering behavior. The main goal is to minimize time to first useful output while preserving turn boundaries.

For a phone agent, the inbound audio usually arrives as small chunks over WebSocket, RTP, or a provider SDK. You do not want to batch too aggressively just to simplify code, because every extra 200–500 ms of buffering is immediately noticeable in a live conversation. A practical streaming STT setup usually:

• keeps a short rolling buffer for voice activity detection or endpointing,

• forwards frames continuously to the recognizer,

• consumes partial transcripts as interim hypotheses, and

• only treats a segment as final when the STT provider signals end-of-utterance or you infer it from silence.

That last point matters because the agent should often start planning before the transcript is final. If your orchestration waits for final text every time, the response feels laggy. If it reacts too early to unstable partials, it may interrupt itself or answer the wrong intent.

Streaming STT: treat partials as hints, finals as commitments

Streaming STT is best handled as a state machine, not as a simple request/response API. You generally have three kinds of events:

• audio frames: raw PCM or encoded frames from the call leg,

• interim hypotheses: unstable partial text, and

• final transcripts: segment boundaries you can safely hand to the agent.

Two implementation details are worth getting right:

1. Endpointing. The speech recognizer may be good at turn detection, but telephony audio is noisy enough that you should expect false starts and false ends. In practice, many systems combine provider endpointing with a small application-level silence threshold so you can hold the turn open for a beat if the user is clearly still talking.

2. Transcript diffing. Interim transcripts often rewrite earlier words. Your agent layer should consume them as updates, not as immutable messages. If you log or trigger downstream actions on every partial, you will create duplicate work and spurious tool calls.

Streaming TTS: start audio early, but preserve coherence

TTS is where many “good enough” demos become real systems. The difference is whether your synthesizer can begin output before the entire response is finalized. For a conversational agent, you want the first audio packet to leave as soon as you have enough text to commit to a phrasing direction.

That introduces a trade-off:

• If you chunk text too small, the voice may sound choppy and the prosody can reset too often.

• If you chunk text too large, latency rises and the user hears dead air.

The sweet spot is usually sentence fragments or clause-sized segments, depending on the TTS engine. Good orchestration code keeps a small text buffer, flushes on punctuation or natural clause boundaries, and allows cancellation if the user interrupts.

For phone agents, interruption handling is not optional. If a caller starts talking over the assistant, you should stop TTS quickly, clear any queued avatar frames, and return to STT capture. If you do not, the agent feels sticky and the conversation becomes hard to interrupt, which is one of the fastest ways to make a voice assistant feel broken.

Lip-sync is a timing problem, not just a visual problem

Once you add an avatar, the audio pipeline and visual pipeline need to share the same notion of “what is being said right now.” The avatar does not need perfect phoneme-level alignment to be useful, but it does need stable, low-jitter sync with the speech stream.

There are a few practical rules:

• Drive animation from the speech timeline. Use the same TTS output that plays to the user as the source of truth for mouth motion.

• Prefer continuous playback over stop/start updates. Frequent resets create visible popping or “dead face” gaps.

• Handle cancellation cleanly. If speech is interrupted, the mouth state should decay or close quickly rather than finishing the old utterance.

• Expect jitter. Network and browser rendering jitter are normal; a small buffer is better than chasing perfect real-time alignment.

In other words, the avatar layer should be treated as another realtime consumer of the agent’s speech stream. If your architecture only thinks in terms of “generate text, then render video,” you will get something that works in clips but falls apart in live conversation.

What a minimal Go-facing integration looks like in practice

Even if your backend is not written in Go, the shape of the integration is the same: you need a session, a streaming conversation loop, and a way to attach the avatar layer to the agent’s audio output. For teams using LiveKit-based voice agents, that usually means inserting the avatar plugin into the agent pipeline so the spoken audio and talking face stay synchronized.

For example, the LiveKit plugin published as livekit-plugins-protoface is intended to be dropped into a voice agent so the agent emits a synchronized talking video face. The exact wiring depends on your agent stack, but the principle is straightforward: the agent continues to own STT, dialog, and TTS, while the avatar layer subscribes to the resulting speech stream and renders the corresponding lip-sync video.

Here is a small illustrative REST example for creating a realtime session. The exact JSON fields depend on the session shape in the docs, so treat this as a sketch of the workflow rather than copy-paste production code:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

If you prefer Python for orchestration or admin tasks, the SDK gives you the same basic flow from code. Again, keep the field names aligned with the docs:

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

If you are working in the LiveKit ecosystem, the plugin repo and examples are the quickest way to understand the intended integration points: GitHub repository, plus the Pipecat integration guide at docs.pipecat.ai if you are already building on Pipecat. The key idea is the same regardless of framework: keep the agent’s audio timeline authoritative and let the avatar subscribe to that timeline.

Operational gotchas: latency, interruption, and rate limits

A few issues show up repeatedly in production:

Latency accumulation. Each stage may only add 100–300 ms, but the sum becomes noticeable. Watch the budget across telephony ingress, STT partials, LLM reasoning, TTS startup, and video rendering. If one stage is slow, it is usually better to simplify the conversation flow than to keep adding buffers downstream.

Backpressure. If your TTS or avatar renderer cannot keep up, avoid letting queues grow unbounded. Old speech is worse than no speech. In a live call, stale audio or a delayed face is usually preferable only for a moment, not indefinitely.

Interruptibility. The user must be able to barge in. That means your system needs a clean cancel path through STT, dialog generation, TTS playback, and avatar rendering. Build cancellation as a first-class control path, not as an afterthought.

Security boundaries. Browser embeds are especially sensitive. If you surface an avatar on a web page, keep credentials out of the browser and enforce origin checks and session limits server-side.

Where Protoface fits

For teams that want to add the avatar layer without building the rendering and session plumbing themselves, Protoface provides the developer-facing realtime avatar API and the supporting surfaces around it: REST for session management, a Python SDK for programmatic control, and integrations for voice-agent stacks such as LiveKit. The practical benefit is not “more AI,” it is less glue code between your TTS stream and the talking face.

If you are wiring this into an existing stack, the docs at docs.protoface.com are the right place to verify the current API shape, auth model, and example payloads. Use the GitHub examples when you want to see an integration end to end rather than piecing it together from snippets.

Conclusion

Streaming STT, TTS, and avatar lip-sync all come down to the same engineering discipline: keep the audio timeline authoritative, consume partial information carefully, and make cancellation and endpointing explicit. If you design the system around continuous streams rather than discrete requests, you can keep latency low enough for phone agents while still producing a natural, synchronized face.

For your next step, build the smallest possible loop: inbound audio, streaming transcript, streaming response, streaming speech output, then attach the avatar layer and measure where the time actually goes. After that, tighten buffering and interruption handling before adding more features. The docs at docs.protoface.com and the linked quickstarts are the most direct way to adapt this pattern to your stack.