Introduction

If you are building a real estate assistant, the obvious baseline is a text chatbot: it can answer listing questions, qualify leads, and schedule showings. The problem is that real estate is still a trust-heavy, high-friction workflow. People want to ask follow-up questions naturally, hear a human-sounding response, and get a sense that the agent is present and responsive. A realtime avatar helps bridge that gap by putting a synchronized talking face in front of the voice agent.

This post shows how to build that experience in Rust, with the architecture you actually need in production: realtime audio in, low-latency model inference, streaming audio out, and a video face that stays aligned with the generated speech. By the end, you should be able to wire a Rust backend to a voice agent, attach an avatar session, and understand the trade-offs around latency, session management, and browser delivery.

What “realtime avatar” means in practice

There are three moving parts:

• Audio transport: capturing microphone audio from a client and shipping it to your backend or agent runtime with low latency.

• Agent loop: transcribing speech, deciding what to say, generating a response, and streaming audio back quickly enough that the conversation feels live.

• Avatar rendering: driving a video face from the same speech stream so lip movement and expression track the spoken output closely.

The important implementation detail is that the avatar should be treated as part of the session, not as a separate, loosely coupled video widget. If audio and avatar state drift, users notice immediately. The usual failure modes are obvious: delayed mouth movement, clipped first phonemes, or the avatar continuing to “speak” after audio has stopped.

In a Rust system, the cleanest shape is to keep your agent logic and session orchestration on the server, then use a browser client or a voice stack to handle transport. Rust is a good fit because it can manage concurrent streams predictably and is strong at building reliable session services.

Designing the session flow

A production-grade flow for a real estate assistant usually looks like this:

1. The client creates or joins a conversation session.

2. Your backend allocates an avatar session and stores the session identifiers.

3. The user speaks; your agent receives audio, transcribes it, and produces a response.

4. The response is streamed as audio, while the avatar consumes the same realtime output to keep lip sync aligned.

5. You persist lightweight session state such as property preferences, budget, location, and showing intent.

The trick is to keep session setup separate from conversation handling. That gives you a clean place to enforce tenant-level limits, rate limit by IP or session, and attach instructions like “prioritize suburban single-family homes under $900k.”

For real estate specifically, it helps to keep structured state instead of relying entirely on free-form conversation memory:

• budget range

• preferred neighborhoods

• bed/bath minimums

• must-have features such as parking, school district, or commute constraints

• lead status: browsing, qualified, ready to tour, or already working with an agent

That state can be used both for responses and for analytics later. The avatar is presentation; the state is the product.

Implementing the Rust backend

For the backend, keep the responsibilities narrow:

• authenticate the user or browser session

• create an avatar session on demand

• store the session ID alongside your own conversation record

• stream agent audio and state updates through your realtime layer

Protoface exposes a REST API for creating and managing avatars and realtime sessions. The exact fields are documented, but the shape is straightforward: send authenticated requests with your API key, receive a session object back, and use that session to drive the avatar.

Here is a minimal Rust-style sketch of the control flow. The HTTP client and payload fields are illustrative; use the docs for the exact request schema.

use reqwest::Client;

.json::<serde_json::value>()

}</serde_json::value>

use reqwest::Client;

.json::<serde_json::value>()

}</serde_json::value>

use reqwest::Client;

.json::<serde_json::value>()

}</serde_json::value>

What matters here is not the exact JSON keys, but the boundary: your backend owns the API key, creates sessions server-side, and hands the browser or agent runtime only the session-specific data it needs.

Streaming audio and keeping latency under control

Realtime voice systems live or die on latency. In practice, the user will tolerate small delays in the model’s reasoning, but they will not tolerate pauses between speaking and hearing the first audio frame. A few engineering details matter:

• Use streaming, not batch. Do not wait for the whole response before emitting audio.

• Propagate backpressure. If your downstream audio sink is congested, stop buffering indefinitely.

• Keep session state hot. Avoid rebuilding prompts and context from scratch on every turn.

• Watch phoneme onset. The first 100–200 ms of speech is where sync bugs show up most clearly.

Rust makes it practical to handle concurrent audio, transcript, and control streams with Tokio tasks or channels. A common pattern is:

tokio::spawn(async move {
});

tokio::spawn(async move {
});

tokio::spawn(async move {
});

For the avatar, the key is that its video updates should be driven by the same output stream as the voice. If the avatar is fed a separate or delayed signal, lip sync will drift. In other words, the avatar should follow the source of truth: the agent’s audio timeline.

Real estate-specific interaction design

Real estate assistants fail when they sound generic. The implementation should reflect the workflow users actually care about.

Useful behaviors include:

• asking for budget before suggesting homes

• confirming geography before discussing commute times

• distinguishing “browse listings” from “book a tour” intent

• handling property comparisons with structured criteria

• escalating to a human agent when the user is ready to tour or negotiate

Technically, this means your prompt and session state should be opinionated. Do not let the agent improvise endlessly. Keep a small set of fields, update them explicitly, and reflect them in responses. If the assistant says, “You said two bedrooms, under $750k, near downtown,” the user feels heard and the avatar becomes much more effective.

It also helps to treat the avatar as a UX layer, not a substitute for product logic. Search, filtering, and lead qualification should still be backed by your own services or data source. The avatar presents the result; it should not be responsible for inventing listing data.

Using Protoface where it fits

This is the point where the avatar plumbing gets annoying if you build it yourself: session management, lip-synced video delivery, and browser-safe embedding. Protoface is designed to cover that layer so you can focus on the agent and business logic.

For a Rust backend, the practical integration path is the REST API: create a session server-side, keep your API key private, and attach the returned session to the voice workflow. If you want a fully managed browser delivery path, customer-managed iframe embeds avoid exposing any API key in the browser and let you set per-embed instructions and voice configuration.

If you are already using Python for agent orchestration, the Python SDK is a convenient way to create and manage avatars and sessions programmatically. For a quick sanity check, the dashboard at app.protoface.com is useful for inspecting sessions, API keys, and usage while you are iterating.

For documentation and request schemas, use the official docs: https://docs.protoface.com.

Example: creating a session with curl

When you are wiring the backend, it is often easiest to validate the API before integrating it into Rust. A minimal request looks like this:

curl https://api.protoface.com/v1/sessions \
}'

curl https://api.protoface.com/v1/sessions \
}'

curl https://api.protoface.com/v1/sessions \
}'

The exact endpoint and payload fields may differ depending on the object you are creating, so treat this as a shape check, not copy-paste production code. The important part is the security model: authenticate server-side, create the session there, and never expose the key in a client bundle.

Common gotchas

• Overlong prompts: if you stuff the entire property catalog into the system prompt, latency and quality both degrade.

• Loose turn boundaries: if your agent does not know when the user has finished speaking, it will interrupt or over-talk.

• Sync drift: separate audio and video pipelines can fall out of alignment unless one controls the other.

• Browser exposure: do not send API keys to the frontend just to create avatar sessions.

• State leakage: reset or scope conversation state properly when a user starts a new lead.

Most of these are not avatar-specific; they are the same issues you hit in any realtime voice agent. The avatar just makes them visible faster.

Conclusion

The basic pattern is simple: keep the agent and session orchestration on the server, stream audio in and out with low latency, and drive the avatar from the same speech timeline so lip sync stays tight. In Rust, that usually means a small set of concurrent tasks, explicit session state, and a clean boundary between your product logic and your media layer.

If you want to implement this without building the avatar stack yourself, start with the REST API or the browser embed flow, then iterate on your agent behavior. The docs at docs.protoface.com cover the current request shapes and integration details, and the GitHub quickstarts linked from the main repo are useful when you want a working reference before you adapt it to your own voice stack.