Introduction

If you are building a real estate open house assistant, the hard part is not generating a response. The hard part is making the assistant feel present: a face that speaks in sync with the audio, updates fast enough to stay natural, and can live inside a voice workflow without turning your app into a video pipeline project.

This post walks through the practical shape of that integration: how realtime avatar lip-sync works, how to wire it into a voice agent, what latency and synchronization constraints matter, and where the operational edges are. By the end, you should be able to stand up a realtime AI avatar that can greet visitors, answer property questions, and stay visually aligned with the agent’s speech.

What “realtime lip-sync” actually means

For an open house assistant, the avatar is usually not “generating video” in the traditional sense. In practice, you are coupling three streams:

• text or structured conversation state from your agent

• audio generated by a TTS engine or voice pipeline

• a video face stream that animates mouth movement and facial motion to match that audio

The key requirement is sync. The avatar renderer has to consume audio with low enough latency that mouth shapes track phonemes closely enough for humans to perceive alignment. If audio lags behind the face, the result feels broken even if each subsystem works independently.

For an open house use case, that usually translates to a few concrete design decisions:

• Keep the conversational path short: user speaks, agent responds, avatar reacts.

• Avoid long blocking steps in the response pipeline, especially if you need a quick greeting.

• Prefer a live transport that can stream audio and avatar frames continuously rather than “render then send” behavior.

• Treat the avatar as a presentation layer attached to the voice agent, not as the source of truth for dialogue.

Step 1: Start with the voice agent, then attach the face

The most reliable implementation pattern is to build the voice agent first and treat the avatar as a synchronized output surface. That keeps your conversation logic independent from visual rendering concerns.

In a LiveKit-based stack, this often means your agent already produces streaming audio. You then add a plugin that takes that audio stream and drives the avatar video in parallel. The practical benefit is that you do not need to redesign the agent architecture; you are just adding a second media sink.

A minimal integration sketch looks like this:

from livekit.agents import WorkerOptions, cli

from livekit.agents import WorkerOptions, cli

from livekit.agents import WorkerOptions, cli

The important detail is conceptual: the plugin should sit at the boundary where your agent emits speech, so the avatar can mirror the same timing. If you instead try to generate video from the final transcript after the fact, you lose the realtime behavior that makes the interaction feel responsive.

Step 2: Design for latency, not just correctness

With an open house assistant, users tolerate modest imperfections, but they notice delay immediately. A useful mental model is to budget for end-to-end responsiveness rather than only for model latency. The relevant pipeline is usually:

1. speech input is transcribed or interpreted

2. the agent decides on the response

3. audio is synthesized or streamed

4. the avatar stream is updated to track that audio

5. the browser receives and renders the media

Each step adds delay. The most common failure mode is letting the language model take too long before the first audible response. For a real estate assistant, it is better to return a brief acknowledgement quickly and follow with a fuller answer than to wait for a perfectly composed response.

Some practical guidelines:

• Use streaming TTS or incremental speech generation if your stack supports it.

• Keep persona prompts concise and stable; do not rebuild the whole conversation policy on every turn.

• Avoid unnecessary media transcoding between your agent and the avatar layer.

• Test on real browsers and consumer laptops, not just on a local dev box.

If your deployment target is a public open house page, network variance matters more than in a controlled demo. Plan for users on mobile connections and slower Wi-Fi. That means keeping assets small, minimizing round trips, and using a transport that is designed for realtime media rather than generic polling.

Step 3: Keep the conversation domain narrow and operationally safe

Real estate is a good use case for a focused agent because the domain is bounded. The assistant does not need open-ended general intelligence; it needs to answer property-specific questions consistently. That lets you reduce prompt drift and keep the voice experience predictable.

For this kind of assistant, I would usually split the logic into three layers:

• Greeting and triage: welcome visitors, ask whether they want a tour, pricing details, neighborhood info, or booking help.

• Property facts: answer from a curated data source instead of relying on free-form memory.

• Escalation: hand off to a human agent when the question is ambiguous, sensitive, or outside the approved scope.

This is less about AI sophistication and more about keeping the realtime system stable. If the assistant is allowed to ramble, you increase response length, latency, and the chance of inaccurate information. In a sales environment, short and grounded responses are usually better.

A few gotchas worth calling out:

• Do not let the avatar become the authority for facts; the source of truth should be your property data.

• Keep fallback responses short so the avatar can recover quickly from misunderstandings.

• Use explicit confirmation for anything operational, such as scheduling follow-up calls.

• Test interruptions. Visitors will interrupt the assistant while walking around the property.

Where Protoface fits

Protoface is useful here because it gives you the avatar layer without forcing you to build the lip-sync plumbing yourself. For a voice-agent stack, the LiveKit plugin is the cleanest path: you keep your agent logic in LiveKit and add a synchronized talking face through the openai realtime quickstart-style integration patterns and the plugin examples in the LiveKit ecosystem. If you are working in Python, the SDK and docs are the right place to confirm exact parameters and session lifecycle details.

A simple REST flow for provisioning an avatar or session is straightforward: authenticate with your API key, create the object, then start the realtime session from your application backend. The API is intentionally server-side; do not expose the key in the browser.

curl -X POST https://api.protoface.com/v1/sessions \

curl -X POST https://api.protoface.com/v1/sessions \

curl -X POST https://api.protoface.com/v1/sessions \

The exact request shape depends on the current API, so treat this as illustrative and verify fields in the docs. The important architectural point is that session creation and control live on the backend, while the browser only receives the media or embed needed for playback.

Operational notes for an open house deployment

If this is going to real visitors, not just internal demos, a few operational details matter.

First, monitor usage and quality tier selection deliberately. The visual quality you need for a lobby display is not always the same as what you need for a mobile embedded assistant. Second, keep an eye on session duration limits and resource usage; a kiosk-style deployment can run much longer than a normal support interaction. Third, make sure your failover path is graceful: if the avatar session drops, the voice agent should still be able to answer textually or route to a human.

If you choose to put the assistant directly on a website, Protoface also supports customer-managed iframe embeds. That can be the right fit when you want a fast deployment with no backend exposure in the browser, plus controls like parent-origin allowlisting and per-embed instructions. For a real estate landing page, that is often enough to get a production-safe demo up quickly, especially if you want the assistant embedded on a property page without touching the site’s own auth model.

Regardless of integration surface, the debugging pattern is the same: verify audio first, then lip-sync timing, then browser playback, then agent behavior. If the voice is clear but the avatar feels out of phase, the problem is usually transport timing or buffering, not the language model.

Conclusion

The core of a realtime AI avatar is simple: stream speech from your agent, keep the video face attached to the same timing source, and minimize anything that adds avoidable latency between a user question and the first visible response. For a real estate open house assistant, that gives you a practical, low-friction way to provide a human-like front end without building a custom video stack.

If you want to implement this with the pieces described here, start with the docs at docs.protoface.com, then pick the integration surface that matches your stack: LiveKit plugin for a voice agent, Python SDK for backend control, or an iframe embed for a browser-first deployment. Build a narrow property-specific conversation, test on real network conditions, and optimize for response latency before you worry about visual polish.