How Do Realtime Fitness and Wellness Avatars Handle Interruptions, Turn-Taking, and Live User Feedback?
Developer guide to realtime fitness avatars: interruption handling, turn-taking state machines, live feedback, and synchronized speech/video.
Introduction
Realtime fitness and wellness avatars look simple on the surface: a face speaks, the lips match the audio, and the user feels like they are in a live session. The hard part is not rendering the face. It is handling the messy parts of conversation: interruptions, turn-taking, overlap, and the stream of micro-feedback users give while exercising, breathing, stretching, or recovering. If you get those behaviors wrong, the avatar feels robotic even when the animation is perfect.
At a systems level, you are coordinating three things at once: a live audio stream, a speech or dialogue engine, and a video face that must stay synchronized with the current speaker. By the end of this post, you should have a clear mental model for how to design those interactions, what state your agent needs to track, and where the failure modes usually are.
Why turn-taking matters more in fitness and wellness
Fitness and wellness sessions are not ordinary chat. Users interrupt to say “slow down,” “hold that pose,” “too much resistance,” or “I need a break” while the agent is mid-sentence. They also give nonverbal feedback: delayed responses, short utterances, heavy breathing, or silence that actually means “I am still here, do not start the next thing yet.”
In practice, that means the assistant needs a conversational policy, not just speech-to-text and text-to-speech. The policy needs to answer questions like:
When should the agent stop speaking immediately?
When should the agent ignore a brief user noise burst and continue?
How long should the agent wait before taking the floor again?
What feedback should update the current exercise plan versus merely adjust tone?
If you think of the avatar as the visible endpoint of a voice agent, the animation layer is downstream of those decisions. The face should not “keep talking” if the agent has already yielded. Likewise, if the agent is reacting to a correction, the avatar should visibly acknowledge that shift quickly enough to preserve conversational trust.
Interruptions: detect them, classify them, then choose a policy
An interruption is not a single event. It is usually a sequence: user audio starts, voice activity detection or streaming ASR notices it, and your dialog manager decides whether the user intends to take the floor. A robust implementation usually separates three concerns:
Detection: some user audio is present.
Classification: is this a true interruption, an acknowledgment, background noise, or a continuation?
Policy: do we stop, pause, barge in, or ignore?
For live fitness coaching, the default should usually favor fast barge-in on meaningful speech. If the user says “stop” or “wait,” the system should cut off the avatar promptly. If the user makes a short grunt, coughs, or says “yeah” while the agent is speaking, the assistant may want to keep control of the turn.
One practical implementation pattern is to maintain a speaking state machine with at least four states:
Idle: no active turn.
Agent speaking: avatar is producing audio/video.
User speaking: agent has yielded or is yielding.
Hold: short debounce window after user audio to avoid ping-pong.
The state transition is driven by events from your audio pipeline and dialog layer. This is a better model than “play TTS until it ends,” because you need to support cancellation and mid-utterance revision.
Turn-taking: make it explicit in your agent logic
Turn-taking is not only about interruption. It is also about when the agent is allowed to speak in the first place. A wellness agent often has a rhythm: instruction, demonstration, user execution, check-in, correction, next instruction. If you do not enforce that rhythm, the assistant can talk over the user’s effort, which is exactly when users want quiet.
A good turn-taking policy usually includes:
Floor ownership: who currently has the right to speak?
Yield conditions: what events release the floor?
Resume conditions: when can the agent continue after the user?
Silence thresholds: how long to wait before prompting again?
For example, during a guided breathing exercise, you may want the avatar to speak only at the start and end of each cycle, not on every breath. During strength training, the agent may need to wait for a rep count or user confirmation before continuing. These are product choices, but they need to be encoded as deterministic behavior, not left to chance in a generative response.
A useful trick is to separate content generation from delivery timing. The LLM can decide what to say next, but the runtime decides when to say it based on turn state. That lets you interrupt or defer output without regenerating the entire response every time the user speaks.
Live user feedback: treat it as control input, not just text
Wellness users provide feedback that is often concise, implicit, and time-sensitive. “Harder.” “Lower intensity.” “Not that side.” “I’m dizzy.” These are not just conversational turns; they are control signals that should update the session state immediately.
In a realtime avatar system, this means your agent should parse feedback into categories such as:
Safety-related: stop, pain, dizziness, breathlessness.
Intensity-related: harder, easier, faster, slower.
Instructional correction: wrong side, misspoke, repeat last step.
Social acknowledgement: okay, got it, continue.
Safety-related input should preempt everything else. A wellness session is not a place to “finish the sentence.” The system should immediately stop the avatar’s current speech, acknowledge the user, and transition to a safe state. Intensity changes may only adjust the next instruction or the current rep tempo. A simple acknowledgement may only affect timing, not session logic.
Also, not all feedback comes as explicit commands. Hesitation, silence after a difficult movement, or shorter answers can be meaningful. If your app has access to live audio timing, use it. Do not require the user to issue structured commands when the behavior can be inferred from conversational context and cadence.
Synchronization: the face is part of the turn system
With a realtime avatar, the visual channel is not decorative. It participates in turn-taking. The face should signal whether the agent is actively speaking, listening, processing, or yielding. If the user interrupts, the avatar should stop mouth motion quickly and transition to a listening expression or neutral state. If there is a brief processing delay, a subtle “thinking” pause can reduce the appearance of lag.
This becomes important in streaming systems because audio and video are produced by different pipelines. A common bug is to cancel the text-to-speech stream but leave the avatar animation running for another beat. Another is to resume the user’s turn without clearing the avatar’s speaking state, which makes the next response feel out of sync even if the audio is correct.
From an implementation perspective, treat animation state as a consumer of the same turn events that drive speech. Do not let it infer state independently from raw audio alone. The event model should be authoritative.
What this looks like in a LiveKit agent
If you are using LiveKit Agents, the cleanest integration path is to add the avatar as a plugin so the voice agent keeps its own conversation logic and the avatar simply tracks the speaking stream. The plugin does not solve interruption policy for you; it makes the visual side follow whatever your agent decides.
The exact method names and configuration fields depend on the integration you are using, but the important part is the pattern: user input updates the turn state first, and the avatar follows that state. If you want a ready-made reference implementation, the plugin repo is the right place to look: https://github.com/protoface-ai/protoface-plugin-pipecat.
When you need a lower-level session workflow
For custom apps, the REST API is useful when you want explicit control over avatar and session lifecycle. That is a better fit if your backend already owns the realtime session orchestration, you need to create sessions on demand, or you want to attach session metadata such as the workout plan or user preferences.
The useful design point here is not the specific endpoint shape; it is that session state belongs on the server, where you can enforce turn rules, safety escalation, and rate limits without relying on the browser. The Python SDK is a reasonable fit if you want to automate this from your application backend. See the documentation at https://docs.protoface.com for the exact request and response fields.
Common gotchas
Over-eager barge-in: short filler sounds can cancel the agent too aggressively. Add a short debounce and classify interruptions.
Under-eager cancellation: if “stop” takes too long to preempt speech, users lose trust fast.
State drift: the dialogue manager thinks the agent yielded, but the avatar still shows speaking.
Ignoring silence as feedback: in fitness and wellness, silence often means the user is doing the thing, not abandoning the session.
Mixing control and content: safety and pace adjustments should update session state directly, not be buried in a generated paragraph.
If you are testing this kind of system, do not only check happy-path conversations. Simulate mid-utterance interruptions, repeated “wait” commands, long pauses, background TV noise, and short replies while the agent is speaking. Those are the cases that expose whether your turn model is truly realtime.
Conclusion
Realtime fitness and wellness avatars work when the interaction model is explicit: detect interruptions, classify them correctly, and drive both speech and avatar animation from the same turn state. The avatar should be a synchronized participant in the conversation, not a passive video layer glued onto a voice bot.
If you are building this kind of experience, start with a simple state machine, add interruption handling before you add more personality, and keep safety-related feedback on the fast path. Then wire the avatar to the agent’s turn events so the visual channel stays in lockstep with the conversation.
For implementation details, examples, and integration guides, the docs are the best next stop: https://docs.protoface.com. If you want to experiment quickly, the quickstarts linked from the repository are a good way to validate your turn-taking behavior in a real session.