Rust Realtime Avatar Architecture for Real Estate: TTS, STT, and Streaming Video
Rust-based real-time avatar architecture for real estate voice agents: STT, streaming TTS, lip sync, and LiveKit integration.
Introduction
If you are building a real estate assistant, you are usually not building “just” a chat bot. You are building a voice agent that can answer listing questions, qualify leads, schedule tours, and hand off to a human when needed. The missing piece is often the face: a realtime avatar that makes the interaction feel like a live agent instead of an audio-only IVR.
This post walks through the architecture behind a practical realtime avatar stack for real estate: how text-to-speech (TTS), speech-to-text (STT), and streaming video fit together; where latency accumulates; what matters for lip sync; and how to think about the control plane for sessions and avatars. By the end, you should be able to reason about the full pipeline and wire an avatar into a voice agent or website without guessing at the mechanics.
Start from the media pipeline, not the product demo
The easiest way to get this wrong is to start with the avatar and treat everything else as “behind the scenes.” In practice, the hard part is coordinating three realtime systems:
STT turns live user audio into partial and final transcripts.
LLM / agent logic decides what to say and when to interrupt or continue.
TTS + video synthesis emits audio and a synchronized talking face, usually as a streaming media track.
For a real estate workflow, the agent often needs to handle back-and-forth turn-taking with low perceived latency. A user might ask, “Is the backyard fenced?” and then interrupt with “Actually, what’s the HOA fee?” The system should recognize the interruption, stop speaking if appropriate, and render a face that stays synchronized with the new audio stream.
Latency budget: where the seconds go
In a live conversation, latency is not one number; it is a chain of small delays. A realistic budget looks something like this:
User starts speaking.
Audio frames reach your STT service over WebRTC or a similar realtime transport.
STT emits partial transcripts; your agent can begin intent detection before the user finishes.
Your agent selects a response, possibly after tool calls or retrieval.
TTS starts producing audio as a stream, not a finished file.
The avatar renderer consumes that audio and video together so mouth movement tracks the phonemes closely enough to look natural.
The important point is that the avatar should not wait for the entire spoken response to be synthesized. If you buffer too aggressively, the response becomes visibly laggy even if the content is correct. For real estate, that is especially noticeable when the assistant is reading concise facts like price, square footage, or school district details.
Architecting the conversation loop
A useful mental model is to separate the conversation loop into control and media planes:
Control plane: session creation, avatar selection, configuration, auth, usage tracking, and any per-session instructions.
Media plane: audio input, transcript events, TTS output, and the streaming video face.
The control plane should be simple and reliable. The media plane should be optimized for jitter, interruptions, and partial results. Keep these concerns separate so you can swap STT or TTS vendors without rewriting session management.
Practical STT and TTS considerations
For STT, partial transcripts matter more than perfect final transcripts in a live avatar. Partial results let your agent:
detect when the user is still speaking,
avoid cutting in too early,
support barge-in when the user interrupts,
extract entities like address, budget, or move-in date as soon as they appear.
For TTS, the key requirement is streaming output. If the engine can emit audio incrementally, your avatar can start speaking before the full response is generated. That matters for “short answer first” behaviors in real estate, such as: “Yes, it has a two-car garage” followed by a longer elaboration.
One subtle gotcha: if you change speaking rate, voice, or punctuation handling, you affect not only the audio but also perceived lip sync. A model that inserts long pauses in awkward places can make even a good avatar look broken. Test with realistic agent prompts, not just isolated sentences.
Streaming video and lip sync
A realtime avatar is typically rendered as a continuously updated video stream, not as a sequence of independent generated clips. That distinction matters because the renderer needs to align mouth motion with audio timing, frame cadence, and any expression state the agent carries across turns.
At a minimum, the renderer needs:
audio timing so visemes can be aligned to phonemes or approximate speech segments,
frame pacing so the video stays smooth under network jitter,
state continuity so the face does not “reset” between sentences unless you want it to.
For developers, the practical consequence is that you want a transport and SDK that treat the avatar like part of the session, not like a one-off render job. In a voice agent, the avatar should follow the same turn-taking lifecycle as audio. When the agent is silent, the face should idle naturally; when the agent speaks, the face should track the current output stream.
Real estate-specific workflow: lead qualification and scheduling
A real estate assistant usually mixes structured and unstructured conversation. The structured parts are predictable: address, price range, financing status, move-in window, and tour availability. The unstructured parts are where the avatar helps: it keeps the interaction approachable while the system collects enough signal to route the lead correctly.
Design the conversation as a state machine with a few explicit checkpoints:
Greeting: identify the listing or neighborhood.
Qualification: ask about budget, timing, and purchase vs. rental intent.
Property facts: answer questions from your listing data source.
Scheduling: offer tour slots or hand off to an agent.
Fallback: when uncertain, ask a clarifying question instead of guessing.
The avatar is not just presentation. It can improve turn discipline. A visual speaking indicator helps users know when the assistant is done, which reduces accidental interruptions and improves the perceived quality of the whole agent.
Where Protoface fits in this architecture
In this stack, Protoface is the layer that turns a voice agent into a realtime talking face without forcing you to build and host the avatar pipeline yourself. For developers already using LiveKit-based voice agents, the simplest integration point is the LiveKit plugin: the quickstart examples and the plugin wiring show how the avatar becomes another realtime participant in the session.
A minimal LiveKit-side pattern looks like this:
If you are managing avatars and sessions from your backend, the REST API is the clean control-plane entry point. Authentication is via API key, and the docs cover the exact request shape. A representative curl call looks like this:
Use the Python SDK when you want to create or inspect sessions programmatically from your own app backend. The SDK keeps the operational side of the integration in one place instead of scattering API calls across route handlers.
If you are exploring the exact session and avatar fields, or you need the supported options for quality tiers and runtime configuration, the docs are the source of truth: docs.protoface.com.
Browser embeds, security boundaries, and why they matter
For consumer-facing real estate sites, iframe embeds are often the cleanest path when you do not want any backend logic in the browser. The important architectural point is that the browser should never see your API key. A customer-managed iframe keeps the sensitive session creation on the server side and constrains the embed with parent-origin allowlisting, per-embed instructions, and rate limits by IP and duration.
That matters operationally. Real estate sites can see bursty traffic from ad campaigns and listing syndication. If you expose the wrong surface in the browser, you invite key leakage and unbounded usage. An iframe boundary gives you a simple, auditable integration surface for interactive avatars on landing pages, listing detail pages, or lead capture flows.
Trade-offs and gotchas worth testing
A few implementation details tend to surprise teams the first time they ship a realtime avatar:
Interruptions are normal. Make sure barge-in stops or de-prioritizes the current TTS stream cleanly.
Partial transcripts are not optional. Waiting for final STT results makes the whole interaction feel slow.
Voice and avatar must share timing. If audio leads video by too much, lip sync looks broken even when each subsystem is “working.”
Per-session instructions matter. Real estate prompts should be scoped to the listing, region, and compliance constraints for that deployment.
Quality tier affects cost and latency trade-offs. If you do not need the highest fidelity for every workflow, you can save budget by choosing an appropriate tier.
Also remember that the best avatar experience is usually not the most expressive one. In a property search flow, credibility matters more than theatrics. A clean, stable, well-timed face beats a flashy one that stutters or drifts out of sync.
Conclusion
A realtime avatar for real estate is mostly a systems problem: keep the media path low-latency, keep the control plane simple, and treat lip sync as a first-class requirement instead of an afterthought. STT should provide partials quickly, TTS should stream, and the video layer should stay synchronized with the agent’s spoken output across interruptions and turn changes.
If you are implementing this today, start with your voice agent architecture, then add the avatar as a realtime participant rather than as a post-processing step. For implementation details, session semantics, and integration options, check the docs at docs.protoface.com and the relevant quickstart repository for your stack.