Unity Realtime Avatar Pipeline: WebRTC vs RTMP for Low-Latency Conversational Video
Compare WebRTC vs RTMP for Unity realtime avatars: latency, A/V sync, interruption handling, and voice-agent integration.
Introduction
If you are shipping a realtime avatar inside a voice agent, the transport choice matters more than the avatar model. The core problem is not “can I send video?”; it is “can I keep audio, video, and agent turn-taking aligned under jitter, packet loss, and variable network conditions without adding noticeable latency?”
For conversational avatars, the answer is usually WebRTC, not RTMP. WebRTC is designed for interactive media: low-latency, bi-directional, adaptive, and tolerant of unstable last-mile networks. RTMP was designed for ingest and broadcast. It still has a place, but it is the wrong default for a live conversational face that needs to react quickly to speech and interruption.
This post walks through the practical differences, how to think about end-to-end latency budgets, and where each protocol breaks down. By the end, you should be able to choose a transport for a realtime avatar pipeline with a clear view of the trade-offs, and understand how to integrate an avatar service like Protoface into a voice agent stack without introducing avoidable lag.
What “low latency” means in an avatar pipeline
In a conversational system, latency is not just “time until first frame.” You care about the whole turn:
Speech-to-avatar delay: how long after the user starts speaking the face begins reacting.
Turn-end responsiveness: how quickly the avatar stops or changes expression when the user interrupts.
Audio-video sync: whether lip motion tracks the actual spoken audio, not an earlier or later buffer.
Recovery behavior: whether the stream resumes cleanly after jitter, packet loss, or congestion.
For interactive avatars, the useful latency budget is usually measured in hundreds of milliseconds, not seconds. Once you drift into multi-second buffering, the face feels detached from the conversation. That is acceptable for a prerecorded webcast; it is unacceptable for a voice agent that needs to respond in real time.
A typical conversational path looks like this:
User audio is captured in the browser or in a voice-agent runtime.
STT, NLU/LLM, and TTS produce the agent response.
The avatar renderer synthesizes lip sync and facial motion from the speech stream.
Video is delivered to the client and presented in sync with audio.
The transport protocol sits at the last step, but it affects the whole pipeline because buffering and retransmission behavior determine how aggressively you can keep the avatar “live.”
WebRTC: built for interactive media
WebRTC is the right tool when the video is part of a conversation. It uses UDP-based media transport with congestion control, jitter buffering, NAT traversal, and built-in support for real-time audio/video synchronization. That combination matters because it optimizes for freshness over perfect delivery.
Practically, WebRTC gives you:
Low glass-to-glass latency: media can move with minimal buffering.
Adaptive behavior: when network quality drops, it can reduce bitrate or adjust pacing rather than stalling the stream.
Full duplex: sender and receiver can both contribute media, which fits voice agent architectures.
Native A/V sync: audio timestamps and video frames are tracked together.
For avatar systems, the important point is not “WebRTC is faster” in the abstract. It is that WebRTC tolerates the realities of conversational UX: interruptions, variable mobile networks, and rapid turn changes. If the user says “wait” halfway through a sentence, you want the avatar to stop or change state immediately. A protocol that prefers continuity over responsiveness will make the avatar feel slow, even if its average throughput is good.
There are a few implementation details to keep in mind:
Signaling is separate from media. You still need an app-layer channel for negotiation.
ICE/STUN/TURN can fail in the wild. Corporate networks, strict NATs, and mobile hotspots deserve testing.
Bitrate and resolution matter. A crisp but unstable 1080p stream is usually worse than a slightly softer 720p stream that stays interactive.
For voice-agent products, the usual choice is to let the agent stack handle turn detection and audio, then keep the avatar transport as close to real time as possible. The avatar should not become a secondary buffering layer on top of the agent.
RTMP: fine for ingest, poor for conversation
RTMP is widely known because it is simple and historically common for streaming ingest. But the architecture is optimized for pushing media toward a broadcaster, not for low-latency two-way interaction. In practice, RTMP usually implies persistent buffering, TCP head-of-line blocking, and higher end-to-end delay.
That makes RTMP a poor fit for conversational avatars for a few reasons:
Latency accumulates. TCP retransmission and buffering can add seconds under real network stress.
Interruption handling is sluggish. If the user interrupts, the stream can keep playing stale media until buffers drain.
A/V interaction is less natural. RTMP is not designed around fine-grained interactive turn-taking.
Client support is uneven. Browsers do not natively consume RTMP; you usually need a transcode or playback bridge.
RTMP still has valid use cases. If your goal is one-way distribution to a CDN or a legacy ingest endpoint, it is straightforward. If your goal is a talking face that should visibly react within a fraction of a second, it is the wrong transport primitive. You can make it “work,” but you will spend effort compensating for delay rather than building product behavior.
A useful rule of thumb: if the human on the other side can speak over the avatar, RTMP is usually the wrong choice. If the avatar is effectively a broadcast source, RTMP may be acceptable.
How to choose: decision criteria that actually matter
When developers compare WebRTC and RTMP, the debate often gets reduced to “latency vs simplicity.” That is too vague. A better checklist is:
Interaction model: Is the avatar responding to user speech, or just playing video?
Turn-taking: Do you need interruption, barge-in, or rapid state changes?
Delivery environment: Browser, mobile app, embedded web widget, or broadcast pipeline?
Network variability: Are users on home broadband, office NATs, or mobile connections?
Operational tolerance: Can you accept a few seconds of delay, or do you need “feels live” behavior?
If the answer to the first three is “yes, yes, browser/web UI,” then WebRTC is usually the default. If you are building a support agent, a sales assistant, or an NPC inside a live product experience, it is hard to justify RTMP because the user experience degrades in exactly the moments that matter most.
One subtle but important point: conversational systems rarely fail because the average latency is slightly too high. They fail because latency is inconsistent. A stream that is mostly fast but occasionally spikes becomes hard to talk over. WebRTC’s congestion control and adaptive buffering exist specifically to reduce that kind of user-visible jitter.
Implementation notes for real systems
Whatever transport you choose, the avatar should be treated as part of the realtime control loop, not as a media afterthought. In practice, that means:
Keep a tight coupling between the agent’s speech output and the avatar’s lip-sync timing.
Prefer short media buffers over large ones, unless you are intentionally prioritizing smoothness over responsiveness.
Test under packet loss and NAT traversal, not only on local Wi-Fi.
Instrument both media latency and turn latency; they fail differently.
If you are integrating with a voice-agent framework, look for a path that preserves synchronized audio/video rather than transcoding through an intermediate broadcast stack. The more hops you add, the more likely you are to accumulate delay or desynchronize motion from speech.
For teams working directly from Python, a simple REST flow to create a session is often enough to wire up a test harness. Exact fields depend on the API shape in the docs, but the pattern is straightforward:
If you are using Python, the SDK is the cleanest way to automate session creation and inspection:
Those examples are intentionally schematic; use the docs for the exact request/response schema and available fields.
Where Protoface fits in this decision
For developers already running a LiveKit-based voice agent, the most direct path is the LiveKit plugin. It lets you drop a synchronized talking face into the agent without building a separate video stack around it. That is the key integration point when your primary problem is conversational latency, not broadcast video.
The plugin is published as livekit-plugins-protoface on PyPI, and the examples in the relevant repository are the fastest way to see how the avatar stays aligned with the agent’s speech stream. If you are building on LiveKit, this avoids the common mistake of treating avatar video as an unrelated side channel. The avatar becomes part of the agent runtime, which is where it belongs.
If you are evaluating the integration pattern, the docs at docs.protoface.com are the right place to confirm the latest API shape, session lifecycle, and plugin usage. For Pipecat users specifically, there is also a dedicated integration guide in the Pipecat docs and package ecosystem, but the underlying principle is the same: keep media on a realtime transport and avoid inserting a streaming-broadcast protocol into an interactive path.
Conclusion
If the avatar is part of a conversation, use WebRTC. It is the protocol that matches the product requirement: low-latency, bi-directional, adaptive delivery with synchronized audio and video. RTMP remains useful for ingest and broadcast, but it is usually the wrong default for an interactive face that must react to speech, interruptions, and changing network conditions.
The practical test is simple: if a human would notice a half-second delay as “laggy,” you need an interactive media transport, not a streaming transport. Start with the lowest-latency path you can sustain, measure under bad networks, and keep the avatar tightly coupled to the agent’s turn-taking loop.
If you want to implement this quickly, start with the relevant quickstart or plugin example, then validate the session lifecycle and media settings in the docs. From there, you can iterate on the agent behavior without fighting your transport layer.