Introduction

If you want a talking avatar in a real estate agent experience, the hard part is not drawing a face. It’s keeping three streams in sync: the user’s audio, the agent’s generated response, and the video face that lip-syncs to that response with low enough latency to feel live.

In practice, that means you need a WebRTC pipeline that can move audio/video in real time, plus an agent architecture that can hand off generated speech to an avatar renderer without introducing obvious delay or desynchronization. By the end of this post, you should understand the shape of that system in Rust, the main integration points, and where Protoface fits when you want to add a realtime avatar without building the avatar stack yourself.

WebRTC is the transport, not the agent

For a talking avatar, WebRTC is usually the right transport because it is optimized for low-latency, interactive media. You are not streaming a pre-rendered video file. You are sending and receiving live tracks, negotiating codecs, managing ICE candidates, and keeping jitter under control. The user should hear and see the agent as a conversation, not as a clip playback.

A useful mental model is:

• Audio in: microphone audio from the user.

• Agent core: ASR, dialogue state, tool use, and response generation.

• Audio out: synthesized or agent-generated speech.

• Video out: a talking face that is synchronized to the outgoing speech.

Rust is a good fit for the media plumbing because you can keep the hot path predictable and avoid putting all your hope in a single asynchronous loop staying healthy under load. You still need to think carefully about backpressure and frame timing, but Rust makes it easier to keep the pipeline honest.

Design the media pipeline around synchronization

The key implementation detail is that the avatar should not be driven by arbitrary timers. It should be driven by the audio it is speaking. That usually means the audio pipeline becomes the source of truth for timing, and the video face follows it.

In a Rust service, the cleanest layout is something like:

1. Join a WebRTC session and subscribe to the user audio track.

2. Run ASR or forward audio to your agent stack.

3. Generate a spoken response, ideally as a stream rather than waiting for the entire utterance.

4. Send the outgoing speech to the avatar layer so it can lip-sync to the exact audio timeline.

5. Publish the avatar video track back into the WebRTC session.

The most common mistake is buffering too much. If you accumulate a full sentence before producing audio, your avatar will look “correct” but feel sluggish. For conversational UX, latency matters more than perfect batching. A short audio buffer is usually better than a large one, even if it makes implementation slightly more complex.

Rust sketch: a WebRTC agent process

This is intentionally schematic, because the exact Rust WebRTC crate and media APIs depend on your stack. The point is the shape of the code, not a drop-in implementation.

// Pseudocode-ish Rust: connect to a room, receive user audio, and publish an avatar video track.

}

// Pseudocode-ish Rust: connect to a room, receive user audio, and publish an avatar video track.

}

// Pseudocode-ish Rust: connect to a room, receive user audio, and publish an avatar video track.

}

A few practical notes about this pattern:

• Keep audio and video clocks aligned. If your video renderer expects a fixed frame rate, feed it from the same stream that produced the audio playback timestamps.

• Use streaming responses. If your agent stack can emit partial audio, the avatar can begin moving earlier.

• Separate signaling from media handling. Join/leave, auth, and room state should not live in the same code path as frame processing.

What to watch for in a real estate agent use case

Real estate conversations are deceptively demanding. Users ask compound questions, interrupt frequently, and often switch between high-level preference questions and concrete inventory lookups. Your avatar is not just a visual garnish; it becomes part of the perceived responsiveness of the whole agent.

Three gotchas matter here:

• Turn-taking: if the agent talks too long, it feels like a monologue. You want short acknowledgements, then concise answers, then a clean handoff for the next user turn.

• Stability under interruptions: if the user cuts in, you need to stop the outgoing audio and video quickly. Otherwise the face keeps talking after the user has already interjected.

• Latency budgeting: ASR, LLM, TTS, and avatar rendering all consume time. If any one layer adds an unnecessary full-buffer wait, the experience degrades noticeably.

In other words, the avatar layer should behave like a media endpoint, not a separate effect system. It needs to consume the same utterance that the user hears, at the same time they hear it.

How Protoface fits without rewriting the media stack

This is where Protoface is useful: it gives you a developer-facing realtime avatar layer that you can drop into an existing voice agent architecture instead of building your own lip-sync pipeline from scratch.

If you are already running a LiveKit-based agent, the most direct path is the LiveKit plugin. The plugin publishes a Protoface avatar into the agent so the voice turn gets a synchronized talking video face. That keeps the integration close to the media plane, which is where it belongs.

For example, the plugin surface is meant to be used from a voice agent process rather than from the browser. A minimal integration will look roughly like this:

# Illustrative only; check the repository and docs for exact parameters

# Illustrative only; check the repository and docs for exact parameters

# Illustrative only; check the repository and docs for exact parameters

If you want to create or manage avatars and sessions directly, the REST API is the better fit. That is useful for provisioning flows, back-office tools, or automation around a fleet of agent personas. A simple request pattern looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

Exact endpoints and fields live in the docs, but the important part is the security model: API keys stay on the server, and the media session is created server-side rather than exposed directly in the browser.

When to use the REST API, and when not to

For a Rust service, the REST API is mainly useful for orchestration: create a session, attach the avatar, and hand the session metadata to your WebRTC agent. It is not the media transport itself. You still need your media stack for the actual audio/video exchange.

If you are building a website that should host an interactive avatar without a backend, customer-managed iframe embeds are the cleanest option. They keep the API key out of the browser and let you set parent-origin allowlists, per-embed voice and custom instructions, and rate limits. That is a good fit for landing pages or lightweight lead-gen experiences, but it is a different product shape than a Rust-based voice agent service.

Practical debugging checklist

When this kind of system behaves badly, the symptoms are usually obvious: the mouth moves late, the voice arrives before the face, the avatar freezes during long answers, or the session reconnects but the media state does not recover cleanly.

Before blaming the avatar renderer, check these first:

• Clock drift: verify audio timestamps and video frame timestamps are derived from the same timeline.

• Buffer depth: measure how many milliseconds of audio you queue before rendering.

• Codec and packetization: mismatched audio packet sizes can create avoidable jitter.

• Backpressure: if the renderer falls behind, decide whether to drop frames or delay the stream.

• Reconnect behavior: confirm that the session can recover after a transient network loss without duplicating tracks.

It also helps to log latency at each stage: inbound audio, transcription, agent response start, avatar audio enqueue, first video frame, and publish time. Once you have those numbers, the bottleneck usually stops being mysterious.

Conclusion

If you are streaming a talking real estate agent avatar in Rust, the core problem is synchronization: the avatar must stay tightly coupled to the speech that drives it, and the whole system has to stay interactive under real network conditions. WebRTC handles the live media path, Rust gives you control over the service boundary and backpressure, and the avatar layer needs to follow the audio timeline rather than invent its own.

For most teams, the fastest way to get this working is to keep your Rust media pipeline focused on agent logic and WebRTC, then use a dedicated avatar surface for the visual layer. Start with the documentation, and if you want to see concrete integration patterns, the quickstarts in the GitHub org are a good next stop.