Introduction

Push-to-talk sounds simple until you implement it in a realtime avatar system: the user should only be able to interrupt while they’re actively holding a key, speaking should start quickly, the agent should stop cleanly, and audio/video state must stay consistent across the transport and the model pipeline. If you get the turn-taking wrong, you end up with people talking over the avatar, clipped audio, late barge-in, or a UI that feels unresponsive.

This post walks through the mechanics of push-to-talk and interruptible turn-taking for a video avatar tied to a realtime voice agent. By the end, you should be able to reason about the turn state machine, wire up the client-side controls, and decide where interruption should be enforced in your stack. I’ll also show how Protoface fits into a LiveKit-based agent flow without requiring you to invent a separate avatar pipeline.

What push-to-talk really means in a realtime avatar app

In a typical voice-agent setup, there are three streams of state you care about:

• User input: microphone audio and UI intent such as “key down” / “key up”.

• Agent output: synthesized speech, which may be streaming in chunks rather than as one finished response.

• Avatar rendering: lip-sync, facial animation, and any visual state derived from the current speaker.

Push-to-talk is not just “mute the mic until the user presses a key.” In practice, it means the application should:

1. Only forward user audio to the agent while the push-to-talk gate is open.

2. Optionally pre-arm the capture path so the first syllable is not dropped.

3. Interrupt agent speech when the user takes the floor.

4. Keep the avatar’s speaking state aligned with the agent’s true audio state, not just UI guesses.

That last point matters. If your avatar continues lip-syncing after the agent has been interrupted, users will immediately notice the mismatch.

Model turn-taking as a small state machine

The cleanest implementation is a simple state machine on the client, with the server or agent runtime enforcing the actual interruption.

At minimum, track these states:

• idle: no one is speaking.

• listening: user can start transmitting audio.

• user_speaking: user audio is flowing to the agent.

• agent_speaking: the agent is producing audio and the avatar is rendering speech.

• interrupting: user began speaking while the agent was talking; agent output should stop.

Transitions are driven by a small set of events:

• keydown / keyup for push-to-talk.

• VAD or explicit “speech started” / “speech ended” events from the input side, if you use them.

• Agent audio start/stop events from the transport or SDK.

• An explicit cancel/stop signal to cut off generation or playback.

A common mistake is to let the browser be the only authority on interruption. The UI can decide when to request interruption, but the agent runtime should be the final source of truth for stopping synthesis and playback. That avoids race conditions when network latency or buffering is involved.

Client-side push-to-talk: capture, gate, and forward

For a browser client, the usual pattern is to keep the microphone permission warm, attach media to your realtime session, and gate the actual transmission of audio frames based on the push-to-talk key. The exact wire protocol depends on your voice stack, but the control flow is the same.

There are two practical implementation details that are easy to miss:

• Warm mic access: request media access before the user presses the hotkey so the first utterance is not delayed by permission prompts.

• Pre-roll buffering: hold a short buffer locally and flush it when the key is pressed, so the beginning of the first word survives human reaction time.

Here is a simplified browser-side sketch:

let pttDown = false;

}

let pttDown = false;

}

let pttDown = false;

}

This is intentionally generic. Your transport may be WebRTC, WebSocket, or an agent SDK abstraction. The important part is that the local gate controls what is sent, not just whether the UI lights up.

Interruptible turn-taking: where to stop the agent

Interrupting an agent should be explicit. Don’t rely on “the user started talking, so the model will eventually notice.” In realtime systems, eventually is too slow.

There are two layers you may need to stop:

1. Playback layer: stop rendering the agent’s current audio immediately.

2. Generation layer: cancel or truncate the in-flight agent response so the model does not keep speaking into a dead channel.

If you only stop playback, the model can keep generating text/audio in the background. That wastes tokens and can produce a delayed continuation when playback resumes. If you only stop generation but leave playback running, the user hears the rest of the buffered response. You generally want both.

For voice agents, a robust interruption flow looks like this:

1. User presses push-to-talk while the agent is speaking.

2. Client sends a barge-in signal to the session.

3. Agent runtime stops current synthesis/playback and marks the current turn as interrupted.

4. Client begins forwarding user audio.

5. When the user releases the key, the agent processes the user turn and responds.

Note that interruption should usually be idempotent. If your UI sends two barge-in signals because of event duplication, the second one should be harmless.

Synchronizing the avatar with the speech state

A video avatar is only convincing when the mouth motion, viseme timing, and “speaking” state are synchronized with the actual audio stream. If your agent has stopped but the avatar still animates for another second, it breaks immersion more than a slightly delayed response does.

The practical rule is simple: the avatar should follow the same speaking lifecycle as the agent audio pipeline. That means:

• Start speaking animation when audio actually starts flowing, not when the language model begins thinking.

• Stop animation when the audio stream is cancelled or naturally ends.

• When interrupted, transition quickly to an idle or listening pose rather than waiting for a full utterance boundary.

In turn-taking systems, “who has the floor” is a first-class concept. The avatar is just a visual representation of that concept. If you treat it as an independent animation effect, you’ll get out-of-sync behavior under load or packet loss.

Protoface with a LiveKit agent

If your agent already runs on LiveKit, the cleanest place to add a synchronized face is the LiveKit Agents plugin. The plugin publishes a Protoface avatar into the agent pipeline so the avatar follows the same speaking events as the voice agent. That keeps barge-in and speaking-state transitions local to the agent runtime rather than scattered across browser code.

A typical integration is conceptually small:

from livekit.agents import JobContext

ctx.add_output(session)

from livekit.agents import JobContext

ctx.add_output(session)

from livekit.agents import JobContext

ctx.add_output(session)

For the actual API shape, use the plugin examples in the repo and the public docs. The point here is not the exact constructor arguments; it’s the architecture: one runtime owns both speech interruption and avatar state, so the two cannot drift apart.

If you want to inspect the implementation path or adapt it to your own voice stack, the relevant repo is the LiveKit plugin examples on GitHub: https://github.com/protoface-ai/protoface-quickstart-videosdk. For the broader API surface and session management details, the documentation is at https://docs.protoface.com.

Server-side control and session management

In some apps, the browser only streams audio, and your backend decides when to create a session, which avatar to use, and when to terminate the turn. That is a good fit for the REST API or Python SDK if you want explicit session control, auditability, or a custom orchestration layer.

A minimal REST request pattern looks like this:

curl -X POST https://api.protoface.com/realtime-sessions \
}'

curl -X POST https://api.protoface.com/realtime-sessions \
}'

curl -X POST https://api.protoface.com/realtime-sessions \
}'

And if you prefer Python, the SDK gives you a programmatic path for creating and managing avatars or sessions:

from protoface import Client

)

from protoface import Client

)

from protoface import Client

)

For push-to-talk specifically, backend control is useful when you need to debounce interruptions, enforce per-session policy, or log every turn transition. It is also the right place to implement guardrails like maximum turn length or explicit session timeout.

The trade-off is latency: if the backend has to round-trip before the agent stops speaking, barge-in will feel sluggish. In practice, you want the client to send the interruption signal immediately, and the backend to enforce and record it.

Common gotchas and how to avoid them

• Dropping the first phoneme: add a short pre-roll buffer so the first audio frame is not lost before the key press is detected.

• Double-speaking after release: make sure keyup closes the gate before the agent starts its next response.

• Non-idempotent cancel: interruption should be safe to send more than once.

• Avatar lag: tie visual speaking state to the actual audio lifecycle, not to model inference start.

• UI-only turn-taking: the browser should request interruption; the agent runtime should enforce it.

One more practical point: if you support both push-to-talk and open-mic modes, keep them as two modes of the same state machine rather than two separate code paths. That prevents “works in demo mode, breaks in production mode” bugs.

Conclusion

Push-to-talk and interruptible turn-taking are mostly systems problems, not UI problems. You need a clear turn state machine, a fast interruption signal, and a single source of truth for when the agent and avatar are speaking. Once those pieces are aligned, the user experience becomes predictable: the user can take the floor instantly, the agent stops cleanly, and the avatar stays synchronized with the real audio state.

If you’re building this on top of a LiveKit agent or a custom realtime stack, start with the docs and examples, then test with intentional barge-in, rapid key presses, and network latency. The public documentation is at docs.protoface.com, and the quickstart repos are linked from the main project resources on GitHub.