Building a Multilingual Conversational Video Agent with Pipecat, WebRTC, and LiveKit
Build a multilingual conversational video agent with Pipecat, WebRTC, and LiveKit: sync ASR, LLM, TTS, and avatar lip-sync.
Introduction
Adding a video face to a voice agent sounds simple until you try to make it feel real. The hard parts are not just rendering an avatar; they are synchronizing audio, lip motion, and turn-taking across a low-latency transport, while preserving conversational state and handling multilingual speech without breaking the experience.
This post walks through the architecture of a multilingual conversational video agent built on Pipecat, WebRTC, and LiveKit. By the end, you should understand how to wire up audio in, audio out, and a synchronized talking face; where the latency budgets usually go; and how to avoid the common failure modes that make these systems feel robotic.
Start with the stream, not the avatar
The right mental model is: you are building a realtime duplex media pipeline first, and an avatar is just one participant in that pipeline. For a conversational agent, the important data flows are:
User audio from the browser or client device.
ASR to turn speech into text, usually incrementally.
LLM orchestration to decide what to say and when to yield the floor.
TTS to synthesize the agent response as audio.
Video avatar rendering that lip-syncs to the same response audio.
WebRTC is the transport that makes this usable in a browser. It gives you low-latency audio/video, jitter handling, NAT traversal, and a clean way to attach tracks. LiveKit is useful here because it gives you the session plumbing for realtime voice agents, while Pipecat gives you a programmable pipeline for chaining ASR, LLM, and TTS components.
The key constraint is synchronization. If the avatar starts animating before audio is ready, or if the audio and video are generated by different clocks, the result looks wrong immediately. In practice, the agent should emit a single conversational turn object or a tightly coordinated set of events so that speech, visemes, and gaze changes stay aligned.
Design for multilingual turn-taking explicitly
Multilingual support is not just “pick a model that supports many languages.” You need to preserve language identity across the turn so your downstream systems can choose the right STT, response language, and TTS voice. If you let the pipeline infer language too late, you tend to get awkward switching, bad punctuation, and incorrect pronunciation.
A good default is:
Detect or infer the user’s language from the first few seconds of audio or the first transcript chunk.
Lock the conversation policy to that language for the current session unless the user changes it explicitly.
Pass the language code through to the model layer and TTS layer.
Keep the avatar behavior language-agnostic; only the speech content and voice change.
That last point matters. The face should not know about translation. It should only know about timing: when speech begins, when it pauses, and when it ends. Viseme generation, jaw motion, and head motion all derive from the audio or phoneme timing. Language selection belongs higher up the stack.
Pipecat pipeline shape: audio in, text in the middle, audio out
Pipecat works well for this because it lets you define a realtime graph of services rather than a monolithic bot loop. The simplest shape is:
WebRTC input track into an audio frame stream
ASR service into text
LLM service to generate a response
TTS service to synthesize the reply
Video/avatar service to emit synchronized talking-face output
That architecture keeps your control plane separate from the media plane. You can restart the model layer without tearing down the WebRTC session, and you can swap TTS voices or language policies per session.
A minimal Pipecat-style skeleton looks like this:
That is intentionally incomplete; the exact service classes and wiring depend on your stack and the docs. The important part is that each stage is explicit and the session carries language and turn metadata forward.
WebRTC and LiveKit: keep latency budgets honest
Once you introduce video, your latency budget gets tight. If you want a natural-feeling agent, users generally tolerate a short pause before the response starts, but they do not tolerate inconsistent timing. The main sources of delay are:
Audio capture and network jitter from the client.
ASR partial-finalization delay.
LLM token generation time.
TTS first-audio chunk time.
Avatar render and stream publication delay.
There are two practical optimizations that matter more than micro-optimizing each service:
1. Start playback and animation on first usable audio. Don’t wait for the full response to be synthesized. Stream TTS if your provider supports it, and publish the avatar video as soon as the output audio stream is available.
2. Keep state on the server side of the conversation. The browser should not be responsible for assembling conversation context or orchestrating the model turn. It should just join the session, send mic audio, and render the received tracks. That reduces client complexity and makes recovery from reconnects much easier.
In a LiveKit-based setup, the agent is typically a room participant that consumes the user’s audio track and publishes its own audio and video tracks. The video track is not “camera video” in the usual sense; it is generated avatar output, but it fits the same transport. That means standard WebRTC concerns still apply: bandwidth adaptation, track replacement, and reconnect behavior should all be tested under real network conditions.
Useful failure modes to test before shipping
Multilingual voice agents fail in predictable ways, and it is worth testing them early:
Language switching mid-session: if the user starts in English and switches to Spanish, do you keep answering in the original language or adapt correctly?
Barge-in: can the user interrupt the agent while the avatar is speaking, and does the video stop promptly?
Short utterances: does the system respond sanely to “yes,” “no,” or a one-word correction?
Accent and code-switching: does ASR preserve enough fidelity for downstream reasoning?
Network jitter: does the avatar keep lip-sync if packet loss increases or the connection renegotiates?
The usual fix for these issues is not “better prompts.” It is cleaner session state, explicit interruption handling, and a clear contract between the media pipeline and the avatar renderer.
Where Protoface fits
This is the part where Protoface becomes useful: it gives you a developer-facing realtime avatar layer that can plug into a LiveKit voice agent so the agent gains a synchronized talking video face. If you are already using Pipecat for the conversational logic, the integration point is straightforward enough that you do not need to build your own lip-sync pipeline.
The LiveKit integration is documented in the Pipecat guide and the plugin repository: Pipecat integration guide and plugin repo. A typical flow is that your agent publishes the normal voice session, and the plugin adds the avatar video surface on top of that session. The details of session creation, avatar configuration, and exact fields are documented separately, so treat any code below as illustrative rather than copy-paste complete.
For backend-managed provisioning or session orchestration, the REST API at api.protoface.com is the place to create and manage avatars and realtime sessions. A minimal request pattern looks like this:
If you prefer to do this in application code, the Python SDK is available as well. The SDK is the right choice when you want to create sessions programmatically, manage avatars, or tie session lifecycle into your own backend.
If you are embedding on a website and do not want to expose any backend credentials, the customer-managed iframe path is the safer model. It keeps the API key out of the browser, supports parent-origin allowlisting, and lets you scope per-embed voice and instructions without turning your frontend into a trusted service.
Operational details that matter in practice
When you deploy this kind of agent, treat it like a realtime service, not a static widget. Monitor session duration, reconnect rate, STT failures, TTS first-chunk latency, and barge-in behavior. Those are the metrics that correlate with user experience.
Also pay attention to rate limits and abuse controls if the avatar is public-facing. Session duration caps and per-IP throttles are not just billing protections; they prevent a single client from pinning expensive realtime resources for too long. For internal apps, usage tiering and environment separation are equally important so you do not mix test traffic with production sessions.
Finally, keep your prompts and voice selection stable. Changing both at once makes debugging hard. If something sounds off, isolate the issue in one layer: ASR accuracy, response generation, TTS voice choice, or avatar timing.
Conclusion
A good conversational video agent is mostly an exercise in systems design. Get the media transport right, keep the conversation state explicit, preserve language through the pipeline, and make sure the avatar is synchronized to the same audio that drives the response.
If you are building this with Pipecat and LiveKit, the practical next step is to wire up a small end-to-end prototype, then stress it under interrupt, reconnect, and language-switch conditions. From there, use the docs at docs.protoface.com to fill in the exact integration details for your chosen surface, whether that is the LiveKit plugin, the REST API, or the Python SDK.