How Does an AI Avatar Work in Astro? Understanding Realtime Video, TTS, and STT Pipelines
Astro AI avatar pipeline guide: STT, TTS, turn-taking, lip sync, and client-side realtime video integration.
Introduction
Building an AI avatar is not just “generate some video.” In a realtime agent, you are coordinating at least three streams: user audio in, model reasoning and response text out, and synthesized speech plus lip-synced video out. If any of those stages stalls, the whole experience feels broken: you get awkward latency, talking over the user, mouth motion that does not match the audio, or a face that keeps moving after the agent has stopped speaking.
This post explains the pipeline end to end: how speech-to-text (STT), the agent, text-to-speech (TTS), and the avatar renderer fit together in a realtime system, what Astro needs to do to keep the UI responsive, and where the usual failure modes show up. By the end, you should be able to reason about the architecture, debug basic latency issues, and wire an avatar into a voice agent without treating the video layer as magic.
The realtime avatar pipeline: STT, agent, TTS, and video
At a high level, an AI avatar is a synchronization problem.
The user speaks into the microphone. That audio is streamed to an STT engine, which produces partial and final transcripts. The agent consumes those transcripts, decides when the user is done, and generates a reply. That reply is converted to speech by TTS. The avatar system then uses the audio waveform and timing metadata to drive lip sync, facial motion, and video frame generation.
The critical detail is that the avatar should not wait for the full reply if it can stream. The best systems begin synthesizing audio as soon as the first tokens of the response are available, then start rendering the face from that audio. That keeps perceived latency down. In practice, you want:
Low-latency STT for partial transcripts and turn detection.
Streaming text or streaming TTS so speech starts before the response is complete.
Audio-aligned facial animation so visemes and head motion stay synced to the spoken waveform.
Playback-state coordination so the UI knows when the avatar is speaking, listening, or idle.
The model itself usually does not “know” about video. The video layer is driven by the audio timing. That means your integration code has to preserve ordering and timestamps carefully. If the TTS stream is buffered too aggressively, the avatar will feel delayed even if the LLM is fast. If audio packets arrive out of order or with inconsistent durations, lip sync will drift.
Where Astro fits: keep the browser as a thin realtime client
In an Astro app, the easiest mistake is to treat the avatar like a static widget. It is not. It is a realtime media session. Astro should own the page shell, component composition, and any server-side secrets, but the actual audio/video session belongs in the browser with a clear lifecycle: connect, stream, render, disconnect.
For a browser-based agent, the frontend typically handles three responsibilities:
Capture mic audio using WebRTC or the browser media APIs.
Receive media from the avatar session as a video element or WebRTC track.
Track session state so the UI can display “listening,” “thinking,” or “speaking.”
In Astro, that usually means a client island/component rather than a server-rendered block. You do not want to serialize live media state through Astro server rendering. Keep the avatar component client-only, hydrate it when the page loads, and let the SDK or iframe manage the streaming internals.
The main implementation concern is auth. If your architecture uses API keys or session tokens, do not ship those to the browser unless the product explicitly supports a browser-safe embed flow. For custom agent flows, issue short-lived credentials server-side and pass only the minimum necessary session data to the client.
STT and turn-taking are more important than people expect
Most of the “this avatar feels bad” complaints are actually turn-taking problems, not video problems. Human conversation is full of interruptions, pauses, and backchannels. Realtime agents need explicit rules for when the user is done speaking and when the assistant should start replying.
That usually means combining:
VAD or endpointing to detect speech boundaries.
Partial STT to show progress and help the agent reason early.
Interruption handling so the user can barge in while the avatar is speaking.
Two practical gotchas:
Overlapping audio can confuse both the model and the user. If the assistant is speaking, decide whether user speech should cancel playback or be queued.
Premature endpointing makes the agent cut users off mid-thought. Tune silence thresholds carefully, especially with noisy microphones or speakers who pause naturally.
For avatar work, endpointing matters because every turn has a visible manifestation. A bad turn detector does not just produce awkward text; it produces a face that starts talking at the wrong time.
TTS and lip sync: the audio waveform is the source of truth
Realistic lip sync does not come from guessing mouth shapes from text alone. The cleanest pipeline uses the synthesized audio waveform, then derives mouth movement from phonemes, visemes, or timing information aligned to that waveform. Even when the avatar is generated from a fixed face model, the renderer needs timing that matches the exact spoken audio.
That has a few implications:
Do not rebuffer audio arbitrarily. If playback is delayed, the mouth can still be correct relative to the audio, but the user experiences latency.
Use the same source of timing everywhere. The audio output, mouth shapes, and speaking-state UI should all follow the same clock.
Handle sentence breaks and prosody. Short chunks can create unnatural motion if the TTS engine resets prosody too often.
For developers, the simplest mental model is: STT produces text, the agent produces reply text, TTS produces audio, and the avatar renderer uses that audio to animate the face. If you change the TTS vendor, you are not just changing voice quality; you are changing latency, chunking behavior, and the timing characteristics of the lip sync pipeline.
Protoface in practice: drop the avatar into the voice agent
This is where Protoface fits cleanly: it exposes the avatar layer as a realtime service rather than making you build the video synchronization stack yourself. For voice agents, the most direct integration is the LiveKit plugin, which lets an existing agent gain a synchronized talking face without rewriting the audio pipeline.
If you are already using LiveKit Agents, the integration is intentionally small: the agent keeps owning STT, LLM, and TTS, while the plugin handles the avatar session and video side. The shape is roughly this:
The details vary by agent framework and the exact plugin version, so use the docs for the concrete constructor and session options. The important part is the boundary: your agent code speaks audio/text as usual, and the avatar layer consumes that stream and returns synchronized video.
If you prefer to provision sessions directly, the REST API is the right surface. That is useful when your backend wants to create an avatar session, attach custom instructions, or manage lifecycle explicitly before handing a session token to the client. A minimal pattern looks like this:
Again, the exact request shape lives in the docs, but the workflow is the same: create a session server-side, keep secrets off the browser, and stream media over the session channel.
If you are integrating from Python rather than a voice-agent framework, the Python SDK is the natural place to manage avatars and sessions programmatically. It is the right fit for backends that need to create sessions on demand, inspect usage, or automate provisioning. The repository and examples are here: GitHub repo.
Trade-offs and gotchas you should plan for
There are a few engineering trade-offs that matter in production:
Latency vs. quality tier: higher-quality audio/video tends to cost more time and compute. Pick the tier that matches the UX target, not just the demo.
Streaming vs. batching: batching is simpler but feels slower. Streaming improves responsiveness but requires better state handling.
Browser embeds vs. custom integration: embeds are easiest to ship safely; custom integrations give you more control over agent behavior and app UX.
Secrets management: keep API keys on the server. If you need a browser-only experience, use the customer-managed iframe flow rather than exposing long-lived credentials.
For iframe embeds, the key benefit is operational simplicity: no backend, no API key in the browser, and built-in guardrails such as parent-origin allowlisting plus per-embed rate limiting. That is a good fit when you want an interactive avatar on a marketing site, support page, or lightweight product surface without building your own realtime media plumbing.
Conclusion
An AI avatar is really a realtime media system with a face attached. The quality of the experience depends on how well STT, turn detection, agent latency, TTS chunking, and audio-aligned rendering are coordinated. In Astro, keep the avatar client-side, preserve a clean session lifecycle, and treat the browser as a realtime renderer rather than a place to hide secrets.
If you want to implement this with less infrastructure work, start with the relevant surface for your architecture: the LiveKit plugin for voice agents, the REST API for server-managed sessions, or the iframe embed for a browser-safe deployment. The public docs at docs.protoface.com cover the concrete request shapes, and the GitHub examples are a good way to sanity-check your integration before wiring it into production.