Introduction

If you want a real estate avatar that can speak, lip-sync, and react in real time, the hard part is not rendering a face. It is keeping three streams in sync: the agent’s audio, the avatar video, and the application state that drives the interaction. In practice, that means you need a low-latency transport, a clean server-side control plane, and a frontend that can render the avatar without turning your Nuxt app into a media pipeline.

This post shows a practical architecture for streaming a lip-synced avatar in a Nuxt frontend with a FastAPI backend and TypeScript client code. By the end, you should understand how to:

• create and manage an avatar/session from FastAPI,

• stream or receive the avatar media in the browser with minimal glue code,

• keep secrets server-side while still letting the frontend control the session, and

• avoid the common latency and synchronization mistakes that make avatars feel uncanny.

What “lip-synced streaming” actually means

For a conversational avatar, “lip-sync” is not a separate animation effect you add after the fact. It is a timing problem. The avatar’s visual mouth movements must be driven by the same speech signal the user hears, or by a tightly synchronized phoneme/timing model derived from that speech. If video and audio are generated independently, even small drift becomes obvious in a few seconds.

At a systems level, you usually have one of two patterns:

• Agent-driven media: your voice agent produces audio, and the avatar layer consumes that audio or its timing metadata to produce synchronized video.

• Session-driven media: your backend creates a realtime session for the avatar and the browser connects to it as a streaming client.

In both cases, the main constraint is latency. If your UI waits for a full “reply” before rendering anything, the avatar will feel laggy. A better design is to establish the session first, then stream incremental speech or media updates as they arrive.

Backend-first control with FastAPI

For production use, keep all avatar credentials and session creation on the server. The browser should never see your API key. FastAPI is a good fit because it gives you a small, explicit API surface for creating sessions, issuing short-lived client parameters, and returning only the minimum information the Nuxt app needs.

At a high level:

1. Your Nuxt app calls your FastAPI endpoint.

2. FastAPI authenticates the user and asks the avatar service to create a realtime session.

3. FastAPI returns a session descriptor or join payload to the browser.

4. The browser uses that payload to connect and render the avatar stream.

A typical FastAPI route looks like this:

from fastapi import FastAPI, Depends

return resp.json()

from fastapi import FastAPI, Depends

return resp.json()

from fastapi import FastAPI, Depends

return resp.json()

The exact request shape depends on the API docs, but the pattern is stable: your server authenticates with the management API, the browser gets a minimal session result, and the browser never handles long-lived secrets.

Nuxt integration: keep the browser thin

In Nuxt, avoid overcomplicating the frontend. Your page only needs to fetch the session payload, mount a video element or embedded player, and listen for lifecycle events. If the avatar is streamed through WebRTC, the browser should be doing what browsers are good at: decoding media and rendering it efficiently.

Here is a simple TypeScript composable that calls your FastAPI endpoint and then hands the response to a renderer component:

export async function createAvatarSession() {

}

export async function createAvatarSession() {

}

export async function createAvatarSession() {

}

Then in a Nuxt page or component:

<script setup lang="ts">

</template>

<script setup lang="ts">

</template>

<script setup lang="ts">

</template>

The important implementation detail is not the component syntax; it is that the frontend should not synthesize session credentials. If your app needs tenant isolation, issue scoped, short-lived session data from FastAPI and keep the policy checks on the server.

Streaming and sync trade-offs

When developers first wire this up, the most common failure is treating the avatar like a static video asset. That works for a demo and fails in conversation. A realtime avatar needs a transport that can adapt to network conditions and preserve A/V timing. WebRTC is usually the right abstraction because it gives you low latency, jitter buffering, and browser-native media handling.

There are a few practical trade-offs to keep in mind:

• Latency vs. quality: higher quality video costs more bandwidth and encoding time. For a real estate avatar, you generally want “good enough” facial detail with low conversational latency, not cinematic output.

• Server orchestration vs. direct client logic: the server should manage identities, sessions, and policy. The client should only connect and render.

• Audio-first responsiveness: if your agent can start speaking before the avatar fully settles, the experience still feels responsive. Users tolerate a brief visual warm-up much more than speech delay.

• State coherence: if your avatar is describing a listing, the spoken content, displayed property card, and any interaction events should be derived from the same app state transition.

For real estate specifically, that means you often want the avatar to sit beside a property card, with the spoken response generated from the same listing data that powers filters, maps, and scheduling actions. If those drift, the experience feels broken even if the media layer is technically correct.

Using Protoface for the avatar layer

This is where Protoface fits naturally: it provides the avatar/session layer so you do not have to build the media plumbing yourself. You can create and manage avatars and realtime sessions through the REST API, or integrate an avatar directly into a voice agent with the LiveKit plugin when you already have an agent pipeline in place.

If you want to create sessions from Python, the SDK is the cleanest starting point. The code below is intentionally illustrative; check the docs for the exact resource names and fields:

from protoface import ProtofaceClient

print(session)

from protoface import ProtofaceClient

print(session)

from protoface import ProtofaceClient

print(session)

If your backend already orchestrates a LiveKit voice agent, the LiveKit plugin is often the shortest path to a synchronized face. The plugin drops a Protoface avatar into the agent flow so speech and video stay aligned without custom media synchronization code on your side. See the plugin repository and examples here: https://github.com/protoface-ai/protoface-quickstart-openai-realtime is one quickstart option, and the Python/agent-facing package is documented in the ecosystem; the docs page is the authoritative reference for current integration details.

In practice, the workflow is:

1. create an avatar/session on the backend,

2. return only the session details required by the browser or agent,

3. connect the frontend or voice agent to the realtime stream,

4. let the avatar service handle the synchronized talking face.

If you want to inspect the raw management API directly during development, a cURL request is often the fastest way to confirm auth and payload shape:

curl https://api.protoface.com/avatars/sessions \
}'

curl https://api.protoface.com/avatars/sessions \
}'

curl https://api.protoface.com/avatars/sessions \
}'

That is usually enough to prove the backend path before wiring the Nuxt component.

Gotchas that matter in production

A few issues show up repeatedly in real deployments:

• Do not expose API keys in Nuxt runtime config sent to the client. Keep management API calls behind FastAPI.

• Do not assume a session is reusable indefinitely. Realtime sessions should be treated as ephemeral.

• Handle reconnects explicitly. Mobile networks, tab suspends, and proxy timeouts will happen.

• Rate-limit the session endpoint. A public-facing avatar endpoint is an abuse target if you do not gate it.

• Measure time-to-first-audio and time-to-first-frame. Those are the metrics users feel.

If you are embedding the avatar in a site that serves many tenants, also make sure your session creation endpoint is tied to your own auth layer. The avatar service should not become your authorization system.

Conclusion

A lip-synced realtime avatar is mostly an integration problem: create the session on the server, keep the browser thin, and preserve audio/video timing across the stream. With Nuxt on the frontend and FastAPI in the middle, you can keep the control plane explicit and the media path low-latency.

Use the docs to confirm the current request shapes and SDK methods, then start with one narrow path: a single avatar, one session endpoint, and a basic browser renderer. Once that works, the rest is mostly product work.

For implementation details and current examples, start at https://docs.protoface.com and, if you want a reference integration, browse the related repos from the GitHub org: https://github.com/protoface-ai.