LiveKit Simulcast Explained: How to Keep Realtime Avatars Smooth on Slow Networks
How LiveKit simulcast keeps realtime avatar streams smooth on slow networks with adaptive layer switching and low-latency fallback.
Introduction
Realtime avatars fail in predictable ways on bad networks: the video freezes while audio keeps moving, lip sync drifts, or the browser starts dropping frames to keep up. If you are shipping a voice agent, support bot, or interactive web experience, the avatar is part of the product surface, not decoration. When the network gets constrained, you need to preserve the perceptual qualities that matter most: continuity, low latency, and stable motion.
This post explains how LiveKit simulcast helps with that, what it does and does not solve, and how to think about it when you are embedding a talking avatar into a realtime agent stack. By the end, you should be able to reason about layer selection, bandwidth adaptation, and the practical trade-offs for keeping an avatar smooth without over-consuming network budget.
What simulcast actually changes
In WebRTC, the sender does not have to publish a single monolithic video stream. With simulcast, the sender encodes multiple versions of the same video at different spatial resolutions and/or quality levels. A receiver can then choose the layer that fits current bandwidth, CPU, and viewport constraints. Instead of forcing the network to carry a full-resolution stream all the time, the system can step down gracefully when conditions degrade and step back up when they improve.
That sounds simple, but it matters a lot for avatars. A talking face usually has a small on-screen footprint relative to a full camera feed, so the “best” quality is often not the highest resolution available. What you want is a stream that looks stable at the size it is displayed, preserves mouth motion, and avoids visible recovery stalls when network conditions fluctuate.
Without simulcast, adaptation is often coarse. The sender may need to renegotiate, or the transport may simply choke and cause visible artifacts. With simulcast, the receiver can pick a lower layer immediately, which is exactly what you want when the user is on a congested Wi‑Fi network or a weak mobile connection.
Why avatars are especially sensitive to network adaptation
Realtime avatars have a different failure mode than generic video calls. The face is usually synthesized, lip-synced to audio, and tightly coupled to the agent’s response timing. That means the user notices not just packet loss, but any mismatch between the audio and the face. A few practical implications follow:
Latency matters more than resolution. A slightly softer image is usually acceptable; a delayed mouth shape is not.
Temporal stability matters more than peak quality. Frequent quality oscillation is distracting, even if the average bitrate is lower.
Small-frame detail is often wasted. If the avatar is rendered at 320–480px wide, sending a high-res layer may not improve perceived quality much.
So the goal is not “always use the highest quality.” The goal is “adapt fast enough that the avatar stays coherent.” Simulcast is one of the cleanest ways to do that because the receiver can make a local decision based on network conditions instead of waiting for the sender to reconfigure the pipeline.
How to think about layer selection in practice
Simulcast is only useful if the receiver can pick the right layer for the current situation. In a LiveKit-based stack, that usually means the client or agent participant subscribes to one of the available streams based on available bandwidth and intended display size. The important engineering mindset is this:
Encode for the display target. If the avatar is shown in a modest widget, do not design your pipeline around full-screen video.
Prefer a low-latency fallback path. When bandwidth drops, a fast downgrade is better than waiting for congestion control to recover the high layer.
Keep the audio path independent. Audio should not be held hostage by the video layer. The avatar can degrade; the agent’s speech should remain intelligible.
In practice, the smoothest avatar experiences use the lowest layer that still looks acceptable at the actual rendered size. If the user expands the widget, the receiver can move up a layer. If they shrink it or move to a weaker network, the receiver steps down. The point is to preserve motion continuity, not raw pixel count.
Bandwidth, bitrate, and the real bottleneck
Developers sometimes treat “video quality” as a visual problem, but on realtime systems it is mostly a transport problem. The network path determines what the user sees:
Upstream bandwidth limits what the sender can publish.
Downstream bandwidth limits what the receiver can sustain.
Jitter and loss affect whether the chosen layer remains stable.
CPU on the client can also matter, especially if the browser is decoding multiple layers or running in a constrained environment.
Simulcast reduces the chance that the entire experience collapses when one of these dimensions gets worse. But it is not magic. If the chosen lower layer is still too large for the available bandwidth, or if the client cannot decode it smoothly, you will still see issues. The practical win is that your adaptation choices are finer-grained and faster.
A useful rule of thumb: design the avatar’s default presentation so that the lowest simulcast layer is still acceptable. If the lowest layer looks broken, your fallback is not good enough.
Implementation details that tend to matter
There are a few details worth keeping in mind when you wire an avatar into a WebRTC application:
Keyframes affect recovery. When the receiver switches layers or recovers from loss, the next keyframe determines how quickly the image stabilizes.
Layer switching should be invisible. Sudden changes in sharpness are less distracting than freezes, but large swings in quality can still feel jarring.
Avatar motion patterns are not natural camera motion. Talking faces have dense mouth motion and limited scene complexity. That often means you can bias toward lower resolutions and let the encoder spend bits on expression and lip motion instead of background detail.
Browser viewport matters. If the avatar is rendered in a narrow sidebar, the highest layers are mostly wasted.
If you are debugging, inspect actual bitrate, selected layer, packet loss, and freeze events rather than judging by eye alone. The difference between “it looks a little soft” and “the receiver is stuck on a higher layer than the network can sustain” is often visible in the metrics before it is obvious on screen.
A practical LiveKit integration path
If you are using a LiveKit voice agent, the cleanest way to add a talking face is the LiveKit Agents plugin. The relevant plugin repository exposes a drop-in path for attaching a Protoface avatar to an agent so the video face stays synchronized with the agent’s speech.
At a high level, your agent still owns the voice conversation logic. The avatar is just another realtime participant that renders the visual output. That matters because simulcast is then part of the transport strategy for the avatar stream, not some separate browser-only hack. You get the same basic adaptation model you would expect from other WebRTC video sources, which is exactly what you want in a mixed voice/video agent.
An illustrative Python setup looks like this:
If you are creating sessions outside the agent runtime, the REST API is the better fit. For example, you might provision an avatar or session before a call starts, then hand the resulting session data to your app:
The exact endpoint names and fields are in the docs, but the architectural idea is the same: create the avatar/session server-side, then let the LiveKit transport adapt the stream to the network it is actually running on.
Gotchas when tuning for slow networks
There are a few traps that show up repeatedly:
Overprovisioning the visual layer. If the avatar is small on screen, high-res simulcast layers do not buy much.
Ignoring audio-video coupling. If the video adapts cleanly but the audio pipeline stalls, the experience still feels broken.
Assuming one network profile. Mobile, enterprise Wi‑Fi, and home broadband all fail differently. Simulcast is most valuable when the conditions are variable.
Using quality tiers as a substitute for transport design. A higher billed tier can improve the base render, but it does not remove the need for sensible bitrate adaptation.
Also, test with actual throttling. It is easy to overestimate how forgiving a desktop browser looks on a fast office connection. Try constrained bandwidth, packet loss, and switching between good and bad networks mid-session. That is where simulcast earns its keep.
How Protoface fits here
This is the kind of problem Protoface is built to sit inside of: a LiveKit agent that needs a synchronized talking face without forcing you to hand-roll avatar transport logic. In this setup, the LiveKit Agents plugin is the practical integration point. You keep your agent code focused on conversation and use the avatar layer to handle the visual side, while WebRTC adaptation takes care of the network variability underneath.
If you are building on top of LiveKit, that separation is useful. You do not want application code trying to emulate congestion control or invent its own fallback scheme for each client. You want the avatar stream to participate in the same realtime transport system as the rest of the call, with adaptation that is predictable under slow or unstable conditions.
Conclusion
Simulcast is not a cosmetic feature; it is a resilience mechanism. For realtime avatars, the goal is to preserve conversational continuity under changing network conditions by letting the receiver switch among multiple encoded layers without expensive renegotiation or visible stalls. That is especially important when the avatar is small, lip-synced, and tightly coupled to speech timing.
If you are integrating a LiveKit voice agent with a talking face, start by making sure the avatar can degrade gracefully on constrained networks, then test how it behaves when bandwidth and loss change mid-session. For implementation details, examples, and the current integration surfaces, check docs.protoface.com. If you want a concrete path, the plugin and quickstarts in the GitHub repos linked there are the fastest way to validate the end-to-end behavior in your own stack.