Reducing Latency in a Svelte Realtime Interview Avatar with STT, TTS, and WebSocket Streaming
Svelte realtime interview avatar latency tuning with streaming STT, TTS, WebSocket control, and end-to-end timing analysis
Introduction
Latency is the difference between a “live” avatar and something that feels vaguely synchronous. In a realtime interview flow, users expect a short delay between spoken input, model thinking, and the avatar’s response, but not so much delay that the conversation stops feeling interactive. If you are combining speech-to-text (STT), a language model, text-to-speech (TTS), and a video face, the hard part is not any single component; it is the end-to-end pipeline.
This post breaks down where latency comes from in a Svelte-based interview avatar, how to reduce it without turning the architecture into spaghetti, and how to think about streaming boundaries so the face, voice, and transcript stay aligned. By the end, you should be able to identify the main latency contributors in your own stack and make pragmatic changes that improve perceived responsiveness.
What actually causes the lag
For a realtime avatar, the user experience is bounded by the slowest stage in the chain:
Mic capture → transport → STT → agent reasoning → TTS → avatar synthesis → render
Each stage has different latency characteristics:
Mic capture: buffering and browser audio processing can add tens of milliseconds before anything leaves the client.
Transport: WebSocket or WebRTC framing, network RTT, and queueing can dominate on poor connections.
STT: streaming STT reduces end-of-utterance delay, but you still pay for partial hypothesis stabilization.
Agent reasoning: LLM-first-token latency is often the largest variable. Prompt size and tool calls matter more than people expect.
TTS: good streaming TTS can start audio quickly, but sentence boundaries and prosody decisions can delay the first chunk.
Avatar rendering: lip-sync needs audio and timing metadata; if the face waits for a full clip, you lose the benefit of streaming.
The main optimization principle is simple: stream every stage that can be streamed, and avoid coupling the entire response to the slowest stage. In practice, that means your UI should react to partial STT, your backend should start generating the answer before the user fully stops speaking, and your avatar should be able to render incrementally rather than after the full TTS payload arrives.
Designing the client for fast turn-taking in Svelte
Svelte is a good fit for this kind of UI because state changes are explicit and lightweight, but it is easy to accidentally introduce avoidable delay in the browser. A few implementation details matter more than the framework choice itself.
Keep audio capture and UI state separate
Don’t tie microphone capture to rendering state. Capture should run continuously once the call is active, while UI state should only reflect coarse events such as listening, thinking, and speaking. If you reinitialize audio contexts, media tracks, or WebSocket connections on every state transition, you will create stutter that looks like network latency but is really client churn.
For a Svelte component, a minimal structure looks like this:
The important part is not the exact code, but the separation of concerns: audio transport stays alive while UI state moves independently.
Prefer incremental updates over end-of-utterance batching
A common anti-pattern is waiting for a final STT result before doing anything. That guarantees a visible pause even when the user’s intent is obvious halfway through the utterance. If your STT service emits partial hypotheses, feed them into your UI and, when appropriate, into downstream logic that can tolerate revision.
There are two ways to use partials safely:
Display-only partials: show them to the user, but do not trigger model turns until finalization.
Speculative execution: start response generation on partial text, then cancel or revise if the hypothesis changes materially.
Speculation reduces perceived delay, but it complicates cancellation, transcript correction, and tool calls. For interview-style avatars, display-only partials are often the safer first step.
Stream the model output, not just the input
Once the user finishes speaking, your biggest latency win usually comes from streaming the assistant response. If your LLM can emit tokens incrementally, start TTS as soon as you have a stable phrase boundary instead of waiting for the full answer. This works best when you split the response into chunks that are prosodically natural, such as clauses or short sentences.
A practical heuristic is:
Accumulate tokens until you hit punctuation or a short pause threshold.
Flush that chunk into TTS.
Keep generating the next chunk while audio for the current chunk is already playing.
This is how you hide model latency behind audio playback. The avatar does not need the entire answer up front; it needs enough audio to begin synchronized speaking.
Use WebSocket streaming carefully
WebSocket is a reasonable transport for realtime avatar control and streaming text/audio metadata, especially if you already have an app-server boundary in your architecture. It is simple, widely supported, and easy to debug. But WebSocket is not magic: if you shove large binary audio blobs through a single serialized channel without backpressure, you can create head-of-line blocking.
For low latency, keep these rules in mind:
Separate control messages from payload flow. Text events, state transitions, and cancellation messages should not wait behind audio chunks.
Chunk audio sensibly. Very small chunks increase overhead; very large chunks increase startup delay.
Honor cancellation. If the user interrupts, stop downstream TTS and avatar playback quickly. Otherwise the system will “talk over” the user even if the UI says it stopped.
Measure round-trip latency. Client timestamps without server timestamps are not enough to isolate where the delay lives.
In practice, a realtime avatar app benefits from a control plane that can say “start,” “continue,” “interrupt,” and “reset” as first-class events. That is a better fit than a single request/response API when you care about turn-taking.
Where Protoface fits
Protoface is most useful where the avatar itself is the latency-sensitive part of the stack. If you are already running a Svelte frontend and a streaming voice backend, you can use the REST API or the Python SDK to create and manage realtime sessions, then drive the avatar with your own STT/TTS pipeline and control events. The key advantage is that you do not need to build the lip-sync layer yourself; the avatar session is designed to stay synchronized with the audio it is given.
For developers building a voice agent backend in Python, the SDK gives you a straightforward way to create a session and attach it to your existing flow. The exact fields depend on the API shape in the docs, but the pattern is roughly:
If you are integrating into a LiveKit voice agent, the plugin path can be even cleaner because the avatar becomes part of the agent graph rather than a separate service you have to coordinate manually. See the plugin repo for the current integration pattern and examples: https://github.com/protoface-ai/protoface-plugin-pipecat. For the broader API surface, use the docs at https://docs.protoface.com.
If you prefer to create sessions from the client or server with raw HTTP, the REST API follows a normal Bearer-token flow:
Keep the API key on the server. If you are embedding an avatar in a website, use the customer-managed iframe approach so the browser never sees your key.
Latency trade-offs that are worth making
There is always tension between responsiveness, correctness, and implementation complexity. A few trade-offs are usually worth it:
Accept slightly imperfect partial transcripts if they let you begin rendering feedback sooner.
Use shorter TTS chunks if your current TTS engine supports streaming well, even if prosody is marginally less natural.
Prefer explicit cancellation over “let it finish” behavior when the user interrupts; users care more about turn-taking than finishing a sentence.
Keep the avatar session persistent across turns instead of recreating it every time; session setup is pure overhead.
What you should not do is optimize one layer in isolation. A faster STT service does not help much if your frontend waits for full responses before updating the UI. A low-latency TTS engine does not matter if your avatar renderer only starts once the audio file is complete.
Debugging the path end to end
If the experience still feels slow, instrument the pipeline with timestamps at each boundary:
Mic capture start
First partial STT event
Final STT event
LLM first token
First TTS audio chunk
Avatar speaking start
Playback completion
Once you can see those spans, the bottleneck is usually obvious. In interview products, the worst offender is often not network transport but a hidden synchronous step: waiting for full utterance finalization, serializing the response too aggressively, or rebuilding UI/audio state between turns.
Conclusion
Reducing latency in a realtime interview avatar is mostly about architecture discipline: stream early, cancel aggressively, keep session state warm, and avoid forcing the system to wait for whole utterances or whole responses when partial data is already useful. In a Svelte app, the client should stay responsive while audio and conversation state move independently. On the backend, your STT, model, and TTS layers should be designed to overlap rather than serialize.
If you are adding a synchronized talking face to an existing voice stack, start with the docs at https://docs.protoface.com and the relevant integration repo for your runtime. From there, measure the pipeline, remove unnecessary buffering, and only then tune the quality tier. That sequence usually gets you most of the perceived latency improvement without making the system harder to maintain.