Introduction

If you are wiring a voice agent to a realtime avatar, you eventually run into the same problem: the speech recognizer does not produce text in one clean chunk. You get partial transcripts while the user is still talking, then a final ASR result when the utterance ends or the engine decides it has enough confidence. If you treat those two events the same way, your agent will either repeat itself, react too early, or show visibly unstable behavior in the UI.

This post is about handling that boundary correctly. By the end, you should be able to build a simple transcript pipeline that:

• renders partial transcripts without committing to them as final state,

• uses final ASR results as the source of truth for downstream logic,

• avoids duplicate agent actions when partials are revised, and

• keeps an avatar’s mouth movement and speech timing aligned with the actual turn-taking model.

For the avatar side, I’ll use Protoface as the reference point, but the patterns here apply to most realtime voice stacks.

Partial vs. final transcripts: what they actually mean

ASR systems usually emit two kinds of text events:

• Partial transcripts are low-latency, incremental hypotheses. They are useful because they arrive quickly, but they are not stable. A later partial may rewrite earlier words, and many systems will retract or replace segments.

• Final transcripts are the stabilized output for an utterance or segment. They are what you should treat as committed text for state updates, logging, conversation memory, analytics, and downstream agent logic.

The important detail is that partials are not merely “early finals.” They are speculative. If you write them directly into your conversation history or trigger tool calls from them, you will get inconsistent behavior.

In practice, a robust client keeps two layers of state:

1. Ephemeral live text for what the user is currently saying.

2. Committed transcript entries for finalized utterances.

That split is the foundation for everything else in this article.

Designing the transcript state machine

The cleanest way to reason about this is as a small state machine per speaker turn. A single user turn can move through these phases:

• idle — no active utterance

• listening — partials are arriving and may change

• finalized — the ASR engine has committed a transcript

At minimum, track these fields for the current turn:

• utterance_id or another stable segment identifier

• partial_text

• final_text

• is_final

• timestamps for first partial and final result

A simple rule set works well:

• On each partial event, replace the current live text for that utterance.

• Do not append partials to conversation history.

• On the final event, replace the live text with the final text, mark the turn committed, and only then feed the transcript into memory, routing, or tool selection.

This prevents a common failure mode: the agent sees “I want to cance…” as a real user command, starts canceling something, and then has to undo the action when the final transcript becomes “I want to cancel my plan.”

Practical handling patterns

1) Render partials as a preview, not as durable state

The UI should show partials, because users expect low-latency feedback. But the display layer and the conversation engine should not share the same source of truth.

A good pattern is:

• the UI subscribes to a live transcript buffer for the active turn,

• the application logic only consumes finalized turns, and

• the active buffer gets cleared or replaced when finalization happens.

That lets you show a smooth typing-like experience without polluting the actual dialogue state.

2) Deduplicate by utterance identity, not by text

Many ASR engines emit multiple partials for the same utterance, and the text can change subtly each time. If you dedupe by text content, you will either drop legitimate updates or treat revisions as new turns. Use a stable ID from the recognition stream if you have one; otherwise, derive one from the session turn and timing.

For example, if the engine provides an event_id or segment_id, key your buffer by that ID. When the final event arrives for that ID, promote the buffered text to committed state. If the next user utterance begins, start a new buffer.

3) Delay expensive work until finalization

Anything costly or externally visible should usually wait for the final result:

• tool calls

• retrieval / RAG lookups

• database writes

• conversation memory updates

• analytics or audit logs

You can still do lightweight speculative work on partials if latency matters, but treat that as an optimization, not a contract. A common compromise is to prefetch likely context from partials, then confirm the action only after finalization.

4) Be explicit about barge-in and overlap

In a realtime avatar or voice-agent system, the user may interrupt the agent while it is speaking. That means you can have overlapping audio and overlapping transcripts. Don’t assume a single linear transcript stream.

When a new user utterance begins during agent speech:

• stop or fade out the agent response if your product supports barge-in,

• associate incoming ASR events with the new user turn, and

• separate the agent’s own generated text from the user’s recognized text.

This matters for lip sync too: the avatar should reflect the current speaker, not the last committed transcript. Final ASR is for commitment; realtime audio state is for animation timing.

Implementation sketch in Python

Here is a minimal example of the transcript handling logic you want in an app or service layer. The exact event names and fields depend on your ASR provider, but the shape is the same.

class TranscriptBuffer:

return {"type": "final", "utterance_id": utterance_id, "text": text}

class TranscriptBuffer:

return {"type": "final", "utterance_id": utterance_id, "text": text}

class TranscriptBuffer:

return {"type": "final", "utterance_id": utterance_id, "text": text}

That example is intentionally boring. Boring is good here. The important thing is that the final event is the only thing that mutates durable state.

How this fits with Protoface

When you add a realtime avatar through Protoface, the avatar is only one part of the pipeline. Your voice agent still needs a clean handoff between live ASR updates and committed user turns, because the avatar animation, response generation, and turn-taking logic all depend on the same underlying conversation state.

If you are using the LiveKit integration, the plugin exposes the avatar as part of the agent runtime, which makes it easier to keep speech, transcript state, and facial motion synchronized. The implementation detail that matters for transcript handling is still the same: render partials immediately if you want responsiveness, but trigger downstream logic only on final ASR results. The plugin repository includes the integration surface and examples; if you are building on that stack, start with the GitHub organization and the docs.

If you prefer to wire sessions yourself over the REST API, the same principle applies. Create or manage the avatar session server-side, feed the voice pipeline its live events, and keep your transcript commit logic independent from the rendering layer.

curl -X POST https://api.protoface.com/<session-endpoint> \
}'

curl -X POST https://api.protoface.com/<session-endpoint> \
}'

curl -X POST https://api.protoface.com/<session-endpoint> \
}'

The exact endpoint and payload fields are in the API docs, but the architectural point is the same: session creation is separate from the ASR lifecycle. Don’t couple “I have a session” with “I have a final user utterance.”

Common gotchas

• Replaying partials into memory. This creates duplicated or contradictory context. Only final transcripts should enter long-term conversation state.

• Assuming final means no more edits. Some systems emit a final result per segment, not per entire conversation turn. Confirm your provider’s segmentation model.

• Using transcript text as the only key. Revisions make text unstable. Prefer IDs.

• Ignoring silence thresholds. End-of-utterance detection is often based on pause duration, not just ASR confidence.

• Letting avatar playback outrun recognition. If your agent speaks before the user turn is finalized, you may interrupt the user or misread the intent.

A simple operational rule

If you want one rule to enforce in code review, make it this:

Partials are for UX; finals are for logic.

That one distinction keeps your system predictable. It also makes debugging much easier, because you can inspect the live buffer separately from the committed transcript history and immediately see whether a bug came from ASR instability or from your own state handling.

Conclusion

Handling partial transcripts well is mostly about discipline: keep a live preview for responsiveness, treat final ASR results as the only committed source of truth, and separate the transcript lifecycle from avatar rendering and agent logic. Once you do that, the rest of the realtime stack becomes much easier to reason about.

If you are integrating a realtime avatar into a voice agent or web experience, start from the public documentation at docs.protoface.com, then wire your ASR pipeline so it distinguishes partial and final events explicitly. The quickstarts in the linked repositories are useful when you want to see the full end-to-end shape, but the core rule stays the same no matter which surface you use.