How Do Realtime AI Avatars Work in Employee Onboarding? Streaming Video, STT, TTS, and Lip-Sync Explained
Explains realtime AI avatars for onboarding: STT, LLM, TTS, lip-sync, streaming video, latency, and LiveKit/Protoface integration.
Introduction
Realtime AI avatars are mostly a systems problem: you need to move text, audio, and video through a low-latency pipeline without making the conversation feel serialized. In an employee onboarding flow, that usually means an assistant that can answer questions, guide new hires through forms or policies, and present itself as a synchronized talking face instead of a plain voice box.
By the end of this post, you should have a clear mental model for how the pieces fit together: speech-to-text (STT), language generation, text-to-speech (TTS), and lip-synced video; where latency comes from; and how to wire a realtime avatar into a voice agent or web app without treating it like a magic black box. I’ll also show where Protoface fits in as a developer-facing avatar layer.
The core pipeline: STT, LLM, TTS, and video are separate streams
The first mistake people make is assuming “the avatar speaks” is one step. In practice, it’s a chain of services, and each one has its own latency and failure modes.
A typical onboarding interaction looks like this:
The user speaks into a browser or mobile mic.
Streaming STT converts audio frames into partial transcripts.
The agent/orchestrator decides when an utterance is complete, then sends text to the LLM.
The LLM returns a response, often as tokens or short chunks.
TTS turns those chunks into audio incrementally.
A video renderer or avatar service generates a talking face synchronized to the audio.
The client plays audio and video together, ideally with minimal drift.
The important detail is that these steps should be streaming, not batch. If you wait for a full transcript, then a full LLM response, then a full audio file, the conversation becomes visibly sluggish. In onboarding, that latency is particularly noticeable because the user is already in a state of attention: they are filling out documents, hearing policy explanations, and often interrupting with follow-up questions.
Where latency actually comes from
For a realtime avatar, the bottleneck is usually not “AI” in the abstract. It is one of these practical issues:
VAD timing: voice activity detection decides when the user has finished speaking. If it waits too long, the system feels slow; if it fires too early, it cuts people off.
STT partials: streaming transcripts can revise themselves, so downstream logic must tolerate unstable text.
LLM turn-taking: the model may need tool calls, policy checks, or retrieval before it can answer.
TTS startup time: good TTS engines usually stream audio, but there is still a first-audio delay.
Video frame generation: if the avatar is synthesized frame-by-frame, the rendering path must keep pace with audio playback.
Network jitter: WebRTC, WebSockets, or media relays all have buffering and retransmission trade-offs.
In employee onboarding, the experience degrades fast if audio and lip movement are not aligned. Users forgive slightly robotic prosody more easily than they forgive a face that keeps talking after the audio ends.
How lip-sync works in practice
“Lip-sync” is often used loosely, but there are a few different implementation styles.
The simplest approach is audio-driven mouth animation: the system listens to the generated audio and maps short-time energy, phonemes, or visemes to mouth shapes. This is fast and good enough for many interactive agents. More advanced systems use a learned video model to synthesize frames that match the speech signal and the avatar identity more naturally.
For developers, the key question is not which model is coolest; it is which component owns synchronization. You want one place in the stack responsible for coordinating:
speech segment boundaries,
frame timing,
audio playout position, and
client-side buffering.
If that coordination is split across the app, the TTS provider, and the renderer, drift becomes inevitable. The more robust pattern is to treat the avatar service as a media endpoint that consumes the spoken response and emits synchronized video, rather than trying to stitch together a separate webcam-like stream yourself.
Streaming video vs. “record then send”
Realtime onboarding avatars should stream, not pre-render. That sounds obvious, but it matters for architecture.
With streaming video, the client starts receiving frames before the response is complete. That means the user sees the avatar begin speaking almost immediately, which reduces perceived latency. It also means interruptions are possible: if a new user utterance arrives, the assistant can stop or pivot mid-response instead of waiting for a full prerecorded clip to finish.
With batch video, you may get higher quality per frame, but you lose interactivity. This is the right trade-off for a marketing explainer or a fixed training video, not for an onboarding assistant that needs to answer “Where do I upload my tax form?” and “Can I update my emergency contact?” in a live session.
WebRTC is usually the right transport for live media because it is designed for low-latency audio/video with jitter buffering, congestion control, and synchronized playout. You can technically move media over other transports, but then you end up rebuilding packet timing and playback logic that WebRTC already gives you.
What changes in employee onboarding specifically
Onboarding is a useful stress test because the assistant is not just chatting; it is guiding a process. That usually means the avatar needs to handle structured turns, not just free-form conversation.
Typical requirements include:
answering policy or benefits questions from a knowledge base,
walking the user through forms step by step,
escalating to a human when confidence is low,
staying consistent with company tone and policy, and
recovering gracefully when STT mishears names, dates, or acronyms.
Because the user is often multitasking across tabs and forms, visual presence matters. A synchronized face gives the assistant a conversational anchor. But the avatar is still just the presentation layer; the real reliability work is in your orchestration logic, prompts, retrieval, and fallback paths.
One useful design pattern is to keep the assistant’s response generation short and incremental. In onboarding, long monologues are usually worse than a sequence of concise prompts with checkpoints. That also helps the avatar stay aligned with speech and makes interruption handling much easier.
Example: a LiveKit voice agent with a Protoface avatar
If you are already using LiveKit for voice, the cleanest way to add a synchronized face is through the LiveKit Agents plugin. The plugin drops an avatar into the agent so your existing voice pipeline gains a talking video layer without you having to manually manage a separate media stack. The repo is here: quickstart examples are also useful if you want a full end-to-end reference.
The shape of this integration is the important part: the agent keeps doing the conversational work, while the avatar plugin handles the synchronized video presence. That separation keeps your core voice stack portable.
Example: creating a realtime session via the REST API
If you need programmatic control outside a voice-agent framework, the REST API is the other common entry point. Use it to create and manage avatars and realtime sessions from your backend, authenticated with an API key. The exact payload fields are documented in the docs, but the request pattern is straightforward:
That kind of API-driven session creation is useful when onboarding is embedded in your own product flow, for example after a user accepts an offer or completes account verification. Keep the API key on the server side; do not ship it to the browser.
Example: Python SDK for backend orchestration
When you want to manage avatars and sessions from Python, the SDK is the easiest path. It is especially handy if your onboarding backend already runs scheduled jobs, webhooks, or workflow code in Python.
Use the SDK when you want typed, reusable backend code. Use the REST API when you want a language-agnostic integration or you are wiring the avatar into an existing service mesh, queue worker, or internal admin tool.
How to think about the browser side
If you are embedding the avatar directly in a website, the critical security rule is simple: do not expose your backend credentials to the browser. For customer-managed iframe embeds, the parent page can host the avatar without handling an API key at all. That keeps the browser surface much smaller, which is especially valuable for onboarding flows that already touch PII, HR data, or sensitive identity information.
The practical upside is that you can add an interactive avatar to an internal portal or onboarding page with less frontend plumbing. The trade-off is that you should still think carefully about parent-origin allowlists, per-embed instructions, and session limits so the experience remains controlled.
One integration lesson: keep media and policy separate
For onboarding assistants, the media pipeline and the business rules should be loosely coupled. Let the avatar handle presence, speech, and visual timing. Let your agent layer handle what is allowed, what must be escalated, and when to ask a clarifying question.
That division buys you a few things:
you can swap TTS or STT providers without rewriting the avatar layer,
you can test policy changes without touching video rendering,
you can degrade gracefully if the avatar service is unavailable, and
you can reuse the same assistant across voice-only and video-enabled surfaces.
In other words, the avatar is a media component, not your product logic.
Conclusion
Realtime AI avatars work by streaming several systems together: STT for user input, an LLM or agent for reasoning, TTS for speech generation, and a video layer that keeps the face aligned with the audio. The engineering challenge is less about making any single model “smart enough” and more about keeping the pipeline low-latency, synchronized, and interruptible.
For employee onboarding, that architecture matters because the assistant needs to feel responsive while also handling structured workflows, policy constraints, and occasional escalation. If you are building this stack, start with the conversational pipeline, then add the avatar as a presentation layer once the turn-taking and latency budget are under control.
If you want implementation details, integration guides, and quickstarts, start with docs.protoface.com and the linked examples in the Protoface GitHub repositories.