Introduction

If you want a classroom tutor that can answer questions live, explain concepts with a face, and feel less like a chatbot glued onto a page, the core problem is not “generate a video.” It’s coordinating three real-time systems: speech, inference, and video delivery. The hard part is keeping them synchronized enough that the avatar’s mouth movement tracks the generated audio while the conversation remains low-latency and interruptible.

This post shows the practical shape of that system using FastAPI on the backend and TypeScript in the browser. By the end, you should understand how to:

• create and manage an avatar/session from a server-side API,

• connect a browser client to a realtime avatar stream,

• handle the common latency and lifecycle issues that matter in a tutoring workflow, and

• decide where an iframe embed is simpler than rolling your own transport.

We’ll keep the example grounded in a classroom tutor: a student asks a question, your app forwards it to an agent, and the avatar speaks the answer with lip-synced video.

Architectural shape: keep the control plane server-side

The important design decision is to keep session creation and credential handling on the backend. The browser should never see your long-lived API key. In a typical setup, FastAPI acts as the control plane:

1. the frontend asks your backend to start a tutoring session,

2. the backend creates or configures the avatar/session through a server API,

3. the browser receives only short-lived session details or an embed URL, and

4. the client connects to the realtime media path.

That separation matters because realtime avatars usually combine WebRTC-style streaming, speech generation, and stateful session metadata. The media path is latency-sensitive; the control path should be authenticated, observable, and easy to revoke.

For the backend, use FastAPI for a small number of explicit endpoints. For the frontend, TypeScript should handle the lifecycle of the session token or iframe URL and then attach the avatar to the UI. Resist the temptation to create sessions directly from the browser with a permanent key.

FastAPI: create a tutor session from your backend

Below is a minimal pattern for starting a tutoring session from FastAPI. The exact request fields depend on the avatar/session schema in the docs, so treat this as illustrative rather than copy-paste complete.

from fastapi import FastAPI, Header, HTTPException

return resp.json()

from fastapi import FastAPI, Header, HTTPException

return resp.json()

from fastapi import FastAPI, Header, HTTPException

return resp.json()

A few implementation notes matter more than the code itself:

• Timeouts: treat session creation as an external dependency and fail fast.

• Idempotency: if your app can double-submit, make session creation idempotent on your side.

• Per-user scoping: tie the returned session to your authenticated classroom user, not just to a tab.

• Instructioning: keep tutor behavior in the session configuration, not buried in frontend state.

In a real classroom app, you’ll often create a new session per exercise or per student interaction window. That gives you better isolation for usage tracking and makes reconnect logic less surprising.

TypeScript: consume the session and mount the avatar

On the client, the job is to request your backend endpoint, then use the returned session data to connect the avatar. You can model this as a simple lifecycle: create session, attach UI, stream audio/video, tear down on navigation.

type TutorSession = {

}

type TutorSession = {

}

type TutorSession = {

}

If your integration uses a direct media connection rather than an iframe, the shape is similar: fetch a short-lived join credential from your backend, then open the session from the browser using the vendor’s client library or WebRTC primitives. The important rule is the same: the browser gets only session-scoped access.

For a tutor avatar, the UI usually needs three states:

• connecting: session created, media not ready yet,

• listening: student is speaking or typing, agent is processing,

• speaking: avatar is rendering audio-synced speech and video.

Keep these states explicit. It makes interruption behavior and retry handling much easier.

Why synchronization is the real problem

When people say “talking avatar,” they often focus on generated video frames. In practice, the visual stream is only convincing if it stays aligned with the audio. The avatar’s mouth shape is usually driven by the spoken audio track or by phoneme/viseme timing derived from it. If your pipeline introduces buffering or jitter, the result looks off immediately, even if the underlying model is good.

That means you should design for:

• low end-to-end latency: minimize round trips between user speech, LLM response, TTS generation, and playback;

• interruptibility: if the student starts talking, the tutor should stop speaking quickly;

• state handoff: preserve conversation context across turns without reinitializing the avatar every time;

• network tolerance: expect transient reconnects, especially on school Wi-Fi.

For a classroom setting, interruptibility is not optional. Students ask follow-up questions mid-answer all the time. A good implementation lets the current utterance be canceled, updates the agent state, and resumes with the new turn without leaving the avatar “stuck” in a speaking animation.

Also be careful about browser autoplay policies. If the avatar speaks immediately after page load, you may need a user gesture before audio can start. In a tutor app, the obvious gesture is “Start lesson.”

When an iframe is the right answer

If your goal is “put an interactive avatar on a page quickly” rather than “own every transport detail,” an iframe embed is the cleanest route. It keeps the API key out of the browser entirely, lets you set per-embed instructions, and avoids shipping media/session complexity in your own frontend.

That is especially useful for internal classroom tools, LMS pages, or a prototype where you want to focus on UX and prompt design first. The browser just loads a hosted interactive surface; your backend can still decide when and where to create sessions and can enforce your own access rules.

For the server-side control plane and docs, see Protoface. The browser embed is the part that tends to save the most time when you do not need a custom media stack. If you prefer to build directly into a LiveKit-based voice agent, the OpenAI Realtime quickstart is a good reference for the surrounding voice-agent plumbing, and the Python package has examples for programmatic session management in the Python SDK repo.

Operational details that matter in production

A tutor avatar seems simple until you run it under real load. A few practical issues show up quickly:

• Quota and quality tier: track usage per classroom or tenant so you can predict cost and degradation behavior.

• Session cleanup: terminate inactive sessions; don’t leave them alive across tab closes.

• Observability: log session IDs, reconnects, turn latency, and failures at the boundary between your app and the avatar service.

• Prompt drift: keep tutor instructions versioned so behavior changes are intentional.

• Content safety: educational apps usually need guardrails for age-appropriate responses and policy-constrained subjects.

If you use a LiveKit-based voice agent, the same operational ideas still apply. The avatar is a visual participant in a larger agent pipeline, not a separate toy UI. Your monitoring should show whether the bottleneck is TTS, agent reasoning, media setup, or client connectivity. If you are already using LiveKit Agents, the plugin surface is useful because it gives the agent a synced talking face without redesigning the rest of the stack; see the plugin repository and package metadata for the integration entry point.

That “agent plus face” model is the right mental model for tutoring: the avatar is not the intelligence, it is the presentation layer over the intelligence.

Conclusion

A classroom tutor avatar is mostly a systems integration problem: keep credentials server-side, create short-lived sessions from FastAPI, mount the realtime experience in TypeScript, and design explicitly for latency, interruption, and cleanup. Once those pieces are in place, the face becomes a straightforward extension of your voice agent rather than a separate product.

If you want to go deeper, start with the public docs and then pick the integration surface that matches your app: REST API for backend control, Python SDK for orchestration, LiveKit plugin for agent-driven voice, or iframe embeds when you want the simplest browser story. The quickest next step is usually to prototype session creation, then wire a single classroom turn end-to-end and measure the actual latency before adding more UI.

For reference material and quickstarts, start at docs.protoface.com and the linked examples from the project README.