Adding a Talking SDR Avatar to an Astro Site with FastAPI, WebSocket, and TTS Streaming
Build a talking SDR avatar in Astro with FastAPI, WebSocket control, and streaming TTS for real-time lip sync.
Introduction
If you already have a voice agent, the next obvious step is giving it a face. Not a prerecorded talking-head clip, but a realtime avatar that tracks speech, stays synchronized with the generated audio, and can sit inside an existing web app without turning your frontend into a science project.
In this post, we’ll build the shape of that system for an Astro site: a browser UI, a small FastAPI backend, a WebSocket connection for low-latency control, and streamed TTS audio that drives the avatar’s speech timing. By the end, you should understand the moving parts well enough to wire a talking avatar into a production app and avoid the usual timing and state-management traps.
What actually has to happen in realtime
A talking avatar is only useful if the video and audio line up closely enough that the user stops noticing the machinery underneath. That means you need three things working together:
Audio generation: your TTS or voice agent produces speech incrementally, not as a fully buffered blob if you care about responsiveness.
Session coordination: the backend keeps track of which browser tab owns which avatar/session, plus any auth or per-session settings.
Video rendering: the avatar service uses the speech stream to generate lip-synced video frames with minimal lag.
For a web app, the main architectural question is where control lives. If you let the browser talk directly to your avatar backend, you risk exposing credentials and complicating authorization. If you keep the browser thin and let FastAPI broker the session, you can maintain a clean trust boundary and still get low latency via WebSocket.
Project shape: Astro frontend, FastAPI backend, WebSocket control plane
Astro is a good fit for the UI layer because the page shell can stay static while a small client component handles the live experience. The backend does the sensitive work: creating sessions, handing out short-lived session data, and relaying state changes between the browser and your avatar service.
A practical split looks like this:
The Astro page loads a client component with a “Connect” button and an avatar container.
The component opens a WebSocket to your FastAPI server.
FastAPI creates or resumes a realtime avatar session, then streams state updates back to the browser.
Your TTS pipeline emits chunks of audio; the avatar session consumes them and renders synchronized video.
The key point is that WebSocket is for your application control plane, not necessarily for the raw media path. The media path often involves a dedicated realtime transport under the hood, while your WebSocket just carries commands, session status, and metadata.
FastAPI WebSocket endpoint for session control
Below is a minimal example of the backend shape. It accepts a browser connection, waits for a “start” command, and returns a session payload that the client can use to attach the avatar view. The exact fields you send to the avatar service depend on the docs, but the pattern is the important part.
That example is intentionally small. In a real app you’ll want explicit state transitions: idle → connecting → ready → speaking → ended. If you skip that, frontend bugs become hard to distinguish from backend latency.
Streaming TTS without blocking the user experience
If you wait for full text generation, then synthesize the whole paragraph, then start playback, your avatar will always feel behind the conversation. Streaming helps because the first audio chunk can start rendering while later chunks are still in flight.
Conceptually, the pipeline is:
User input arrives.
Your agent generates text incrementally or emits a response plan.
TTS produces audio chunks as soon as enough text is available.
The avatar session consumes audio chunks and keeps lip sync aligned.
The most common bug here is buffering too much on the server or in the browser. A small amount of jitter buffer is fine; a large one defeats the point of realtime. Keep chunk sizes consistent, propagate backpressure, and make sure your frontend doesn’t “optimistically” play an outdated segment after the user has already interrupted.
A simple client loop in the Astro component
In the browser, you usually want the client component to do three jobs: connect, show connection state, and attach the avatar view when the server says it is ready. If your avatar is rendered in an iframe, this can be very small. If you are embedding a media element or canvas, the client needs more lifecycle handling.
In practice, I would also add heartbeat messages and reconnect logic. WebSocket failures on consumer networks are normal, and a one-shot connection strategy tends to fail exactly when the user is mid-conversation.
Why session ownership and auth boundaries matter
Once you introduce realtime avatars, you have a new class of resource that needs lifecycle management. A session can be live, idle, interrupted, or expired. If the browser can create sessions directly with a long-lived API key, you have an avoidable security problem. If the backend owns session creation, you can validate the user, enforce limits, and tie session lifecycle to your app’s own auth model.
That also makes quota and billing easier. You can record when a session starts, how long it stays active, which quality tier it uses, and whether the user is still entitled to continue. Those are all server-side concerns, not browser concerns.
Where Protoface fits in this pattern
This is the part where a dedicated avatar API saves time. Protoface gives you the avatar/session layer so you don’t have to build the lip-sync and realtime video machinery yourself. For a web app like this, the most natural integration is the REST API plus the Python SDK: FastAPI owns the backend trust boundary, creates sessions with your server-side API key, and hands the browser only what it needs to attach the avatar experience.
A very small server-side call looks like this in spirit:
And if you prefer to inspect the HTTP layer directly, the same shape works over the REST API:
Use the public docs for the exact request and response fields, because those are the parts that tend to evolve. The important operational pattern is stable: keep the secret on the server, create a session there, then let the browser join the experience with a minimal surface area.
Gotchas you will hit if you skip the details
1. Lip sync is not just “audio + video.” If the audio stream is chunked poorly or arrives late, the avatar can look mechanically off even when the voice sounds fine. Test with realistic network conditions, not localhost only.
2. Interruptions need explicit handling. If the user speaks over the agent, you should stop the current TTS stream and invalidate any queued video/audio chunks for that turn.
3. Browser lifecycle matters. Tabs sleep, mobile radios drop, and users navigate away. Clean up sessions when the client disconnects.
4. Keep secrets out of the client. The browser should never see a long-lived API key. If you need iframe-style isolation for a public page, use a customer-managed embed model instead of leaking credentials into frontend code.
Conclusion
The core pattern is straightforward: let FastAPI own trust and session orchestration, use WebSocket for low-latency control, stream TTS instead of batching it, and hand the browser only the minimum state required to render the avatar. That gives you a realistic path to adding a talking SDR, support rep, or interactive guide without entangling the frontend with your backend credentials.
If you want the implementation details, start with the docs at docs.protoface.com and then pick the integration surface that matches your stack. For Astro plus a Python backend, the REST API and Python SDK are the cleanest place to begin. If you already have a voice agent, the LiveKit plugin is the shortest route to a synchronized talking face; the relevant examples are in the linked GitHub repos.