How to Validate STT, TTS, and LLM Handoffs in an Interview Practice Voice Agent
Validate STT, LLM, and TTS handoffs in a realtime interview voice agent with turn traces, latency metrics, and barge-in checks.
Introduction
When you build an interview practice voice agent, the hard part is usually not generating answers. It’s validating the handoffs between speech-to-text (STT), the LLM, and text-to-speech (TTS) under real-time conditions.
Those boundaries are where most bugs hide:
STT transcripts arrive late, partial, or with punctuation that changes the prompt meaning.
The LLM starts responding before the user has finished speaking, then has to be interrupted cleanly.
TTS begins speaking too early, too late, or continues after a barge-in.
The avatar lags behind audio, which makes the system feel broken even when the text is correct.
By the end of this post, you should be able to design a practical validation loop for an interview practice agent: test each boundary independently, verify the end-to-end turn sequence, and instrument the system so you can tell whether failures come from recognition, reasoning, synthesis, or transport.
Model the voice agent as a state machine, not a chat loop
The first mistake is treating a voice agent like a regular chat bot with audio attached. A realtime interview practice agent is a state machine with overlapping streams:
Audio input: microphone frames moving to STT.
Transcript stream: partial and final STT results.
LLM turn state: prompt assembly, generation, tool calls, response truncation.
Audio output: synthesized speech chunks from TTS.
Avatar rendering: lip-synced video driven by the same output timing.
For validation, define explicit checkpoints for each transition. A useful minimum is:
User speech starts.
STT emits partial transcripts.
STT emits final transcript.
LLM receives the final user turn.
LLM emits the first token or first chunk of response.
TTS starts synthesis.
First audio packet is playable.
Avatar frame stream begins matching the audio cadence.
If you cannot observe these events, you cannot debug the handoff. You’re just guessing from the final answer text.
Validate STT before you validate the agent
In interview practice, STT quality matters less than transcript stability. A technically mediocre recognizer that produces consistent final transcripts is often easier to build on than a “better” recognizer that keeps revising itself in ways your prompt logic cannot absorb.
What to test
Partial-to-final churn: how often a partial transcript changes before finalization.
Endpointing latency: how long after the user stops speaking the final transcript appears.
False end-of-turns: whether short pauses trigger premature finalization.
Punctuation sensitivity: whether the final transcript inserts commas or periods that materially change intent parsing.
Domain terms: names, company acronyms, course titles, and common interview jargon.
For interview practice, create a short test set that includes:
Self-introductions with names and organizations.
Behavioral answers with long pauses.
Technical answers with acronyms and code terms.
Interruptions like “actually, let me rephrase that.”
You’re not looking for perfect transcription. You’re looking for transcript behavior your turn-taking logic can safely consume.
How to validate it
Log every STT update with timestamps and a turn identifier. At minimum, capture:
arrival time
partial vs. final flag
raw transcript text
speaker activity / endpoint signal if available
Then compute these simple metrics:
End-to-final delay = final transcript time - last speech frame time
Revision count = number of partial edits before final
Turn boundary accuracy = whether the agent waited for the correct stop point
For evaluation, compare the final transcript against a human reference and also inspect the partials. If the LLM only sees the final transcript, a slightly wrong partial is not necessarily a bug. If you stream partials into the model, it becomes a bug immediately.
Validate the LLM handoff as a prompt boundary problem
Once STT is stable, the next failure class is the handoff from final transcript to LLM input. The common issue is prompt contamination: partial STT, tool metadata, or prior assistant text leaking into the current turn.
What the LLM should receive
A clean turn should include only the finalized user text plus the minimal conversation context required for the interview scenario. Avoid feeding the model raw audio events or unstable partial text unless you have a very specific reason to do so.
For interview practice, the prompt should usually enforce three things:
Role consistency: the agent is the interviewer or coach, not a generic assistant.
Turn discipline: one user answer should map to one evaluation step or one follow-up question.
Output shape: answers should be concise enough to fit a spoken response without trailing clauses that TTS might elongate awkwardly.
What to test
Single-turn replay: feed a fixed transcript and verify deterministic behavior at the prompt boundary.
Multi-turn carryover: ensure prior context influences the model the way you expect, and no more.
Interrupt handling: if the user cuts off the agent, does the current response get discarded or truncated cleanly?
Retry behavior: if the LLM call fails, does the system repeat the same prompt or accidentally mutate the turn?
Useful instrumentation here includes the exact prompt payload, model response, token timing if available, and a correlation ID that follows the turn through the rest of the pipeline.
Example: feed a finalized transcript into a Python SDK session
Exact request and response fields depend on the SDK version, but the basic shape is the same: create a session, then attach your agent logic to the transcript events.
The important part is not the SDK call itself; it’s that your turn boundary is explicit. If the SDK or plugin gives you an event stream, use it to separate partial and final transcripts rather than inferring from silence alone.
Validate TTS as a streaming boundary, not a file conversion
TTS failures are often timing failures. A system can generate the right words and still feel wrong if synthesis starts too late, clips the first phoneme, or continues after the user barges in.
For a voice interview agent, test three properties:
Time to first audio: how long after the LLM finishes before audio actually plays.
Chunk continuity: whether streaming TTS emits smooth audio segments instead of stalling between clauses.
Cancellation: whether in-flight synthesis stops immediately on interruption.
Streaming TTS is preferable for realtime interaction because you do not want to wait for the entire utterance to finish before playback begins. But streaming also means you must handle partial output carefully. If the model revises a sentence mid-generation, your TTS layer needs a clear policy: buffer, flush, or cancel and restart.
Common failure modes
Double-speaking: the agent starts a new turn while the previous TTS stream is still draining.
Audio drift: the avatar mouth motion lags behind the synthesized audio by a noticeable margin.
Barge-in race conditions: the user interrupts, but the output pipeline keeps playing for another second.
Prosody mismatch: the model generates a short sentence, but the voice model stretches it unnaturally and changes perceived meaning.
For validation, record the first audio timestamp and the cancel timestamp on each turn. If you can’t stop TTS fast enough, your turn-taking logic will feel broken even if the dialogue is otherwise correct.
Validate the full handoff with a turn trace
The easiest way to debug end-to-end behavior is to produce a single timeline for each turn. A useful trace looks like this:
This makes regressions obvious. If the STT is fast but the LLM request lags, you have a prompt or orchestration problem. If the LLM is quick but playback starts late, the issue is TTS or transport. If audio starts on time but the avatar lags, the video sync layer is the culprit.
Also track turn abandonment. If a user starts speaking again before the previous answer finishes, your system should record whether it interrupted, queued, or ignored the new input. There is no universal right answer, but there must be a consistent one.
Where Protoface fits
This is exactly the kind of pipeline the Protoface LiveKit plugin is meant to sit inside: your existing voice agent keeps owning STT, LLM, and TTS, while the avatar stays synchronized to the live audio stream. If you’re already validating turn timing in a LiveKit-based agent, the plugin lets you observe the video face as part of the same interaction instead of bolting it on after the fact.
For a concrete starting point, the LiveKit plugin examples in the GitHub organization are useful when you want to see how the avatar session attaches to the agent runtime. If you prefer direct control over sessions and avatar lifecycle, the documentation is the right place to check the exact request shapes and event behavior.
Conclusion
To validate STT, TTS, and LLM handoffs in an interview practice voice agent, treat the system like a realtime pipeline with observable boundaries. Verify transcript stability before prompt design, verify prompt cleanliness before response generation, and verify TTS cancellation and startup latency before blaming the avatar.
If you build a turn trace and test against a small set of realistic interview prompts, most “mysterious” bugs become ordinary timing or state bugs. That’s the point: once the system is instrumented, you can fix the actual boundary that failed instead of tuning prompts blindly.
For implementation details, examples, and integration specifics, start with docs.protoface.com and then choose the surface that matches your stack: LiveKit plugin, REST API, Python SDK, or iframe embed.