Introduction

If you are adding a voice agent, the hard part is usually not “making it talk.” It is making the audio feel continuous, low-latency, and operationally safe when the model is streaming partial responses, the TTS engine is chunking audio, and the avatar video has to stay aligned with what the user hears.

In practice, a production-ready TTS integration for an AI avatar has to solve three separate problems at once:

• turning text or agent output into stable audio chunks,

• keeping the avatar’s mouth motion synchronized with that audio, and

• doing all of it without leaking credentials or introducing brittle browser-side plumbing.

This post walks through the architecture I would use for that system, the failure modes to watch for, and how a platform like Protoface fits into a real deployment. By the end, you should have a clear mental model for wiring TTS into a realtime avatar pipeline that is suitable for production rather than just demos.

Start with the audio pipeline, not the avatar

For a voice or video agent, TTS is the timing source. Everything downstream depends on it. The avatar should not “guess” what the agent will say; it should react to the actual audio stream, or at least to a tightly controlled representation of it. That keeps lip sync stable and makes barge-in, interruption, and turn-taking behave predictably.

A useful mental model is:

1. The LLM or dialog manager emits text incrementally.

2. The TTS service converts that text into speech chunks.

3. An audio transport delivers those chunks over a realtime channel, typically WebRTC in browser-facing systems.

4. The avatar renderer consumes the same stream and drives mouth motion from audio energy, phonemes, visemes, or a model-specific alignment signal.

The key implementation detail is that “incrementally” matters. If you wait for an entire response before synthesizing audio, latency will be obvious. If you stream too aggressively with unstable partial text, you can create audible restarts, awkward prosody, or jitter in the mouth animation. In production, you want a buffering strategy that balances latency against stability.

Design for streaming TTS, not one-shot synthesis

Most real agents should synthesize in short windows. The exact window size depends on your TTS vendor and quality requirements, but the goal is the same: start audio early enough to feel conversational while preserving enough context for natural prosody.

In practice, that means your app should treat TTS as a stream of segments, not a single blob. A common pattern is:

• accumulate tokens until you hit punctuation or a pause boundary,

• synthesize the segment,

• queue the resulting audio to the playback pipeline,

• keep collecting the next segment while the current one plays.

This works better than naive token-by-token synthesis because TTS models need enough context to pronounce names, disambiguate sentence structure, and generate natural cadence. It also keeps your avatar motion aligned to actual prosody instead of a stream of disjointed syllables.

Handle barge-in, cancellation, and turn ownership explicitly

The moment you add a face to a voice agent, users expect interruption to feel natural. If they speak while the agent is talking, the system should stop cleanly, not keep animating a mouth that is no longer relevant.

That requires a turn controller in addition to TTS. At minimum, it should track three states:

• idle: nothing is playing, ready to respond,

• speaking: audio is queued or playing,

• interrupted: user speech has preempted the current turn, and any in-flight synthesis should be canceled.

Cancellation is important because TTS and avatar pipelines tend to have multiple buffered layers: model output, synthesized audio, network transport, and playback buffers. If you only stop the final playback layer, upstream work may continue wasting compute and may even race back into the session after the interruption. In a good implementation, cancellation propagates all the way back to the generation step.

For user experience, also make sure the avatar’s state changes immediately on interruption. Even if the backend still has a few milliseconds of buffered audio, the visual should stop “speaking” as soon as the turn is lost. That small detail makes the interaction feel much more responsive.

Keep lip sync tied to the audio clock

Avatar sync problems usually come from mixing clocks. If the audio player, the TTS generator, and the video renderer each advance independently, the avatar will drift. The fix is to choose one source of truth for playback timing. In most realtime systems, that is the audio clock on the receiving side.

There are two common approaches:

• Audio-driven visemes: the avatar reacts to amplitude and spectral features from the actual audio as it is played.

• Alignment-driven animation: the TTS service provides timing metadata, and the avatar renderer maps it to mouth shapes.

Audio-driven animation is simpler to integrate and often good enough for high-level realism, but it can be less precise for certain phonemes. Alignment-driven animation can look better if your TTS provider exposes stable timing information. The trade-off is that any mismatch between metadata and actual audio playback becomes visible very quickly.

Whichever method you use, treat network jitter and buffering as first-class concerns. If the avatar is rendered in a browser, WebRTC is usually the right transport because it is designed for realtime media and handles jitter, packet loss, and adaptive playout better than ad hoc websocket audio delivery. The avatar face should move based on what is actually being heard by the client, not based on when your server happened to generate it.

Production concerns: auth, isolation, and observability

A prototype often puts everything in one process. A production integration should separate responsibilities:

• the backend owns credentials and session orchestration,

• the client receives only ephemeral media/session material,

• the avatar runtime consumes realtime audio/video without needing long-lived secrets.

This is especially important for browser embeds. Never expose your API key to the client if the browser is not fully trusted. If a user can view source, the secret is no longer secret.

You also want enough observability to answer basic questions quickly:

• How long does it take from text emission to first audio packet?

• How often are turns canceled mid-synthesis?

• Are failures coming from TTS, transport, or avatar rendering?

• Is latency worse on specific networks or regions?

In practice, instrument the pipeline at each boundary: text commit, TTS request start, first audio byte, audio playout start, and session end. That gives you enough signal to distinguish a slow model from a slow transport.

Example: driving a voice agent with the LiveKit plugin

If your agent already uses LiveKit, the lowest-friction path is to add the avatar at the agent layer so the voice and video stay coupled inside the same session. The quickstart examples are useful if you want to see a complete flow, but the core pattern is simple: instantiate the plugin, attach it to the agent, and let the media pipeline handle the realtime session.

from livekit.agents import Agent, JobContext

await agent.run(ctx)

from livekit.agents import Agent, JobContext

await agent.run(ctx)

from livekit.agents import Agent, JobContext

await agent.run(ctx)

The important part is not the exact constructor shape; it is the architectural boundary. The agent produces speech, the plugin binds that speech to an avatar session, and the transport keeps audio and motion synchronized. That is much easier to reason about than a separate video subsystem trying to infer when the agent is speaking.

Example: server-side session creation with the REST API

For workflows where you need explicit control over avatar lifecycle, use the REST API from your backend. That keeps your credentials off the client and lets you create sessions with server-owned policy.

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

Exact fields and response shapes are documented in the API reference. The pattern you want is straightforward: create a session on the server, hand the browser only what it needs to join that session, and keep the secret material out of the frontend.

Example: Python SDK for backend orchestration

If you prefer a typed backend integration, the Python SDK is a good fit for provisioning avatars, creating sessions, or wiring internal tools around usage and lifecycle. The code below is illustrative; check the docs for the current method names and request models.

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

This style is particularly useful if your application already has a control plane in Python, for example a bot orchestrator or a job runner that provisions per-tenant sessions on demand.

How Protoface fits without overcomplicating the stack

What makes the docs worth reading is that the integration surface is narrow enough to fit into existing systems, but still covers the common production paths: a LiveKit plugin for voice agents, a REST API for server-managed sessions, and browser-friendly embeds when you do not want to ship your own realtime frontend.

For the TTS problem specifically, the useful part is that you do not have to invent a separate lip-sync channel. You can keep your agent’s audio pipeline intact and attach the avatar where the speech is already flowing. That reduces moving parts and makes cancellation, sync, and ownership easier to reason about. If you are evaluating the LiveKit path, the plugin repository is the right place to start because it shows the intended coupling between agent output and avatar playback.

Conclusion

A production-ready TTS integration for AI avatars is mostly an exercise in controlling timing and ownership. Stream speech in manageable chunks, drive the avatar from the actual audio clock, propagate cancellation all the way through synthesis, and keep secrets server-side. If you do those things well, the avatar feels responsive instead of “animated.”

If you are implementing this now, start with one integration surface and get the lifecycle right before you optimize quality. The documentation at docs.protoface.com is the place to verify API details, and the GitHub examples are a good way to see how the pieces are wired together in practice.