Introduction

Real estate is a good fit for realtime avatars because the interaction pattern is already conversational: answer questions, qualify intent, explain inventory, and hand off when the user wants a human. The hard part is making that feel immediate and coherent instead of like a chat box with a video sticker on top.

In this post, I’ll walk through the architecture for an interactive real estate virtual agent in Rust using WebSocket streaming for the application layer and a realtime avatar for the visual layer. By the end, you should be able to wire up a Rust service that streams user turns, maintains session state, triggers avatar speech, and keeps the UI responsive enough for a property-finding flow.

What “interactive” means in practice

For this kind of agent, “interactive” is not just text generation. You typically need four pieces working together:

• Low-latency input ingestion so partial user messages, microphone transcripts, or UI events can be processed quickly.

• Session state so the agent remembers the user’s budget, location, property type, and whether they want buy/rent.

• Streaming responses so the avatar can start speaking before the full answer is complete.

• A media surface that renders talking video with lip-sync and timing aligned to the generated speech.

WebSocket streaming is a good fit for the app-side protocol because it gives you a persistent connection, bi-directional messages, and straightforward incremental updates. You can send user utterances as they arrive, stream back token chunks or control events, and avoid the overhead of polling.

Rust service architecture for a real estate agent

A practical Rust setup usually looks like this:

1. The browser opens a WebSocket connection to your Rust backend.

2. The frontend sends events such as user_message, mic_transcript_partial, or property_card_clicked.

3. The backend updates a per-session state machine and calls your LLM or routing layer.

4. As the model responds, the backend streams text chunks and speech directives back to the client.

5. The avatar layer consumes the synthesized speech stream and renders synchronized video.

The key design choice is to separate conversational state from transport. Your websocket should be a thin realtime envelope around a session object, not the place where business logic accumulates. That makes retries, reconnects, and multi-step flows much easier to reason about.

Modeling the session state

Real estate conversations have a small but important set of fields that should survive turns:

• intent: buy, rent, or browse

• location preferences: city, neighborhood, commute constraints

• budget range

• property type: condo, single-family, apartment, townhouse

• must-haves: parking, pet-friendly, school district, yard, etc.

A simple Rust struct is enough to start:

#[derive(Default, Clone)]
}

#[derive(Default, Clone)]
}

#[derive(Default, Clone)]
}

When a new message arrives, update the struct incrementally. Don’t try to re-derive everything from scratch on every turn. In practice, you’ll want a narrow extraction step that turns the latest utterance into structured updates, then a response step that uses the accumulated state.

WebSocket streaming in Rust

The exact framework is up to you; Axum, Actix, and Warp all work. The important part is a message protocol that supports incremental updates and control events. A minimal JSON shape might look like this:

{
}

{
}

{
}

Your server can then stream back partial agent output:

{
}

{
}

{
}

And later a completion event:

{
}

{
}

{
}

That protocol is intentionally boring. Boring is good here. It lets the frontend render subtitles, show typing state, and hand text to your speech pipeline without coupling itself to the LLM provider.

Code sketch: receiving and responding

Here is a simplified Rust sketch to show the shape of the loop. The message handling is illustrative; wire it into your chosen async runtime and JSON schema.

async fn handle_user_message(session: &mut LeadSession, text: &str) -> Vec<String> {

}

async fn handle_user_message(session: &mut LeadSession, text: &str) -> Vec<String> {

}

async fn handle_user_message(session: &mut LeadSession, text: &str) -> Vec<String> {

}

In production, the extraction step would be more robust than string matching, but the flow is the same: parse, update state, generate a response, stream it.

Speech, timing, and avatar synchronization

The visual layer only looks natural if you treat speech as a stream, not a single finalized blob. A common mistake is to wait for the full answer, synthesize the entire response, and then start the avatar. That adds avoidable latency and produces a noticeable “dead air” gap.

Instead, structure your agent so that speech can begin as soon as the first sentence is stable. A good operational pattern is:

1. Collect the user turn.

2. Generate a concise first sentence that confirms understanding.

3. Stream the rest of the answer or follow-up questions.

4. Keep the avatar speaking only while audio is actually available.

For real estate, this matters because the first response is often just the slot-filling acknowledgement: location, budget, property type, and a next question. If the agent can say that immediately, the interaction feels competent even before it has retrieved listings.

Where Protoface fits

This is where Protoface is useful: it gives the voice agent a synchronized talking video face without forcing you to build avatar rendering yourself. In a Rust-first architecture, the avatar is usually not part of your application server; it sits behind the API or the voice stack you already have.

If you are using a Voice/agent framework and want the avatar attached to that runtime, the LiveKit plugin is the cleanest path. For a direct integration, the REST API lets you create and manage avatars and realtime sessions from your backend. The docs at docs.protoface.com are the right place for the exact session and avatar fields.

A minimal API call pattern looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

That kind of session object is what your Rust backend can associate with a websocket session ID. Your app keeps conversation state; the avatar session handles the visual realtime surface. Keep those responsibilities separate.

How to think about the voice pipeline

For a real estate assistant, the voice pipeline usually has three steps: speech-to-text, orchestration, and text-to-speech. If you already have a voice agent framework, your Rust service may only need to manage the orchestration layer and the websocket protocol to the UI. If your agent is entirely custom, you still want the same boundaries.

Two practical constraints matter most:

• Backpressure: don’t keep emitting text chunks if the downstream speech or avatar layer cannot consume them fast enough.

• Interruptibility: allow the user to interrupt mid-answer. Real estate users do this constantly when they remember a neighborhood, school district, or price ceiling.

That means your websocket protocol should support cancellation. A simple stop or barge_in message from the client is often enough. When it arrives, stop synthesis, halt avatar speech, and return the agent to a listening state.

Operational gotchas

A few mistakes show up repeatedly in realtime avatar integrations:

• Overloading the websocket with every internal event. Only send what the browser needs.

• Ignoring reconnects. Mobile browsers and flaky networks will drop connections; session resumption matters.

• Letting prompts sprawl. Keep the system instructions tight so the agent stays focused on the property-finding job.

• Mixing secrets into client code. API keys stay server-side; never expose them in the browser.

For customer-facing web embeds, the security model should be different from your internal tools. If you need a browser-native avatar without backend work or exposed secrets, use the iframe-based embed path and keep the allowlist and rate limits on the server side.

Testing the interaction loop

Before you wire in production data or listings, test the loop with a controlled conversation script:

1. User asks for a 2-bedroom rental in a specific city.

2. Agent extracts budget and timeline.

3. Agent summarizes constraints back to the user.

4. User interrupts with a new neighborhood or price limit.

5. Agent updates state without losing the previous turn.

If the state updates are correct and the avatar starts speaking quickly, you are probably in good shape. If the response feels sluggish, measure the time to first meaningful token, time to first audio, and time to avatar start separately. Those are different bottlenecks.

Conclusion

A useful real estate virtual agent is mostly an exercise in disciplined realtime systems design: keep session state explicit, stream over WebSockets, support interruption, and keep the visual avatar synchronized with the speech pipeline. Rust is a solid choice for the backend because it forces you to be clear about state, ownership, and async boundaries.

If you want to add the avatar layer without building media plumbing yourself, start with the Protoface docs and pick the integration path that matches your stack. The quickstarts linked from the project repo are a good way to validate the flow end-to-end before you integrate against live property data.

Next step: read the docs, wire up a minimal websocket loop, and get a single conversational turn rendering cleanly before expanding into search, filtering, and handoff.