Introduction

If you want a language tutor that feels present, you need more than text-to-speech. The agent has to listen, think, speak, and show a synchronized face without introducing extra lag or desynchronization. That means dealing with the usual realtime problems: WebRTC media timing, buffering, turn-taking, partial transcriptions, and keeping the video avatar aligned with the agent’s audio output.

This post shows how to build that pipeline in Rust, with the avatar rendered as a realtime stream rather than as a pre-recorded clip. By the end, you should have a clear mental model for how the pieces fit together, how to structure async Rust around a voice agent, and where a developer platform like Protoface fits when you want to add the avatar layer without building the full media stack yourself.

Start from the media model, not the UI

The common mistake is to think of the avatar as a frontend concern. In practice, it belongs in the media path. Your tutor agent typically has three concurrent loops:

• ingest: microphone audio or a remote caller’s audio arrives as a stream,

• reason: ASR, prompt orchestration, and LLM generation run incrementally,

• egress: synthesized speech and lip-synced video are emitted as realtime media.

The hard part is that these loops are coupled by timing. If the model emits a response faster than the avatar stream can be produced, you need backpressure. If the caller interrupts, you need to cancel the active turn cleanly. If the avatar’s video is derived from the same speech signal that drives TTS, the two outputs must share a timeline or they will drift.

In a Rust implementation, the simplest reliable shape is an async pipeline with bounded channels:

mic audio --> ASR --> turn manager --> LLM --> TTS --> avatar/video stream

mic audio --> ASR --> turn manager --> LLM --> TTS --> avatar/video stream

mic audio --> ASR --> turn manager --> LLM --> TTS --> avatar/video stream

Rust is a good fit here because you can model each stage as an async task, keep state explicit, and use bounded queues to make overload visible instead of silently accumulating latency.

Model the tutor as a turn-taking state machine

A realtime language tutor is not just “chat with voice.” It has to decide when the student is speaking, when the agent should respond, and when to yield. That is a turn-taking problem, and you want it codified as state, not spread across callbacks.

A practical structure is:

• Listening: accept inbound audio and accumulate partial transcripts.

• Thinking: assemble the user turn, apply instructions, and generate a response.

• Speaking: stream TTS audio and the avatar face together.

• Interrupted: stop synthesis, clear queued video, and return to listening.

The important detail is cancellation. In async Rust, every stage should be cancellable via a shared token or channel. When barge-in is detected, you do not want the LLM to keep generating a stale completion while the user is already talking again.

Conceptually, you can wire this with Tokio tasks and a small shared context:

struct SessionCtx {
}

struct SessionCtx {
}

struct SessionCtx {
}

That is not a complete implementation, but it captures the pattern: keep session state explicit, send incremental events between tasks, and let each stage exit promptly when a new user turn starts.

Keep audio and avatar timing coupled

The avatar only looks convincing if its motion is synchronized to the spoken audio. For a language tutor, this matters more than photorealism: learners notice timing errors immediately when mouth shapes lag audio by even a few hundred milliseconds.

There are two common approaches:

1. Generate audio first, then derive video timing from it. This is easier when the avatar service accepts the speech audio or phoneme/alignment data and renders the face from that source of truth.

2. Generate video and audio independently, then align them. This is more flexible but much harder to keep stable under network jitter.

For most developers, the first option is the correct one. Treat the speech stream as authoritative and let the avatar pipeline follow it. That means your media graph should preserve ordering and timestamping end to end. If you are relaying through WebRTC, do not strip timing metadata at the edges. If you are buffering outbound audio chunks, keep them small enough to avoid noticeable lip-sync delay.

In Rust, that usually means:

• chunking outgoing audio into steady, low-latency frames,

• keeping a bounded queue between TTS and the avatar renderer,

• propagating timestamps or sequence numbers,

• dropping stale output on interruption instead of trying to “catch up.”

For a tutor, a slightly lower-fidelity face is better than a beautiful but late one. Users forgive texture; they do not forgive desynchronization.

Rust async shape: one task per concern, bounded everywhere

Async Rust works well here if you keep the structure boring. Resist the urge to build a giant callback object that owns everything. Instead, separate concerns into tasks and use channels for handoff:

tokio::spawn(async move {

});

tokio::spawn(async move {

});

tokio::spawn(async move {

});

Use bounded channels rather than unbounded ones. If the model or network path slows down, bounded queues force you to make a decision: backpressure, drop, or cancel. For realtime voice, silent buffering is usually the worst outcome because it turns into latency that no one explicitly owns.

For long-running sessions, also be deliberate about resource cleanup:

• close outbound media tracks when the session ends,

• cancel synthesis tasks on disconnect,

• avoid holding large transcript or audio buffers in session state longer than necessary,

• log turn boundaries and interruption events so you can debug timing issues later.

Where Protoface fits: add the avatar layer without building a face renderer

This is the part where a developer platform is useful. If your Rust voice agent already handles the conversation logic and transport, you do not need to invent the avatar runtime from scratch. Protoface provides the realtime avatar layer that can be attached to a voice agent, so the agent gets a synchronized talking face instead of just audio.

For LiveKit-based agents, the relevant surface is the plugin in the LiveKit ecosystem. The idea is straightforward: keep your existing voice agent architecture, add the Protoface plugin, and let it manage the avatar session and synchronized video output. The plugin and examples are documented in the repository, and the main docs cover the API and session model: docs.protoface.com and the plugin repo at github.com/protoface-ai/protoface-plugin-pipecat.

If you prefer to orchestrate sessions directly, the REST API is also available. A minimal session creation request looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

The exact fields depend on the session and avatar shape in the docs, but the pattern is the same: create an avatar/session, attach it to your media workflow, and stream the result to the client. If you are working in Python first, the SDK makes it easy to wire up programmatic session management before you port the orchestration to Rust.

from protoface import Client

)

from protoface import Client

)

from protoface import Client

)

The key engineering point is that the avatar is not a separate product surface in your app; it is part of the same realtime turn lifecycle as audio, transcripts, and model state.

Practical gotchas when moving from prototype to production

Once you have a demo working, the failures are usually in the edges:

• Interrupt handling: make sure barge-in cancels TTS and video generation immediately, not after the current phrase.

• Network jitter: WebRTC helps, but your own buffering policy still matters. Keep queues small and observable.

• Session isolation: do not share mutable state across users unless you really mean to; every tutor session should be independent.

• Prompt drift: if the tutor behavior depends on instructions, keep those instructions attached to the session, not scattered across application code.

• Operational visibility: log turn start/end, interruption events, and media errors so you can correlate “bad lip sync” reports with actual transport issues.

Also remember that avatars introduce a cost/quality trade-off. If your use case is a quick in-browser proof of concept, lower quality may be fine. If you are shipping a customer-facing tutor, quality tier becomes part of the product decision because it affects latency, realism, and cost.

Conclusion

Streaming a realtime language tutor avatar in Rust is mostly about discipline: explicit turn state, bounded async pipelines, cancellation on interruption, and a media path that keeps audio and video synchronized. If you get those pieces right, the avatar feels responsive instead of bolted on.

If you want to skip building the avatar runtime yourself, Protoface gives you the API and integrations to attach a synchronized face to an existing voice agent. Start with the docs, wire up a small session, and validate your timing under real network conditions before you optimize for polish: docs.protoface.com.