Introduction

If you are building a voice avatar app in TypeScript, the hard part is not getting audio to play. It is keeping three realtime streams aligned: the user’s speech, the agent’s generated response, and the avatar’s visual state. Live captions, text-to-speech (TTS), and speech-to-text (STT) all need to agree on timing, partial results, and interruption behavior, or the experience quickly feels laggy and unstable.

This post walks through a practical architecture for a TypeScript voice avatar app that supports live captions, streaming STT, and streaming TTS. By the end, you should have a clear mental model for how to wire the pieces together, where the tricky edge cases are, and how to think about integrating a synchronized avatar without turning your app into a timing mess.

Model the conversation as a realtime pipeline

The cleanest way to build this is to treat the app as a bidirectional stream pipeline rather than a sequence of request/response calls.

At a minimum you have four stages:

• Capture: microphone audio from the user.

• STT: streaming transcription, usually with partial and final segments.

• LLM / agent logic: generate a response incrementally, not only at the end.

• TTS: synthesize audio from the agent’s output as chunks arrive.

The avatar sits on top of the audio pipeline. It should be driven by the same audio that reaches the user, not by a separate “mouth animation” timer. Otherwise lip sync drifts, especially when network jitter or interruption enters the picture.

That means your control plane should understand a few separate states:

• Listening: microphone active, STT partials arriving.

• Thinking: transcript finalized, agent generating text.

• Speaking: TTS audio streaming, avatar lip-syncing.

• Interrupted: user barges in, cancel the current response and reset playback.

In practice, most bugs come from treating these as purely UI states. They are also transport states. If you do not cancel pending TTS when STT detects a barge-in, you end up speaking over the user. If you do not surface partial transcript updates, your captions will lag behind the actual conversation.

Implement live captions with streaming STT

Live captions are usually just the visible side of streaming STT, but the implementation details matter. You want partial hypotheses for responsiveness, and final transcript segments for correctness.

A useful pattern is:

1. Stream audio frames from the browser to your backend over WebRTC, WebSocket, or a voice-agent transport.

2. Forward those frames to a streaming STT provider.

3. Render partial captions immediately, but style them as tentative.

4. Replace or commit the line when the STT service emits a final segment.

Two practical rules help a lot:

• Keep partial and final text separate. Do not mutate final transcript history with every new hypothesis.

• Track timestamps. Captions should align to utterances, not just to whatever text happens to arrive last.

Here is a minimal TypeScript shape for the caption state:

type CaptionSegment = {

}

type CaptionSegment = {

}

type CaptionSegment = {

}

If your STT provider supports word-level timestamps, use them. They are useful for highlighting current words during playback and for debugging latency. If it does not, segment-level timing is still better than nothing.

The main gotcha is endpointing. If you finalize too aggressively, you will chop up the user’s utterance. If you wait too long, the UI feels stale. Most voice apps tune this by listening for both silence duration and contextual completion signals from the STT service.

Stream TTS instead of waiting for a full sentence

For a voice avatar, TTS should be incremental. Waiting for a full assistant answer before synthesizing audio adds unnecessary latency and makes the avatar feel delayed. Streaming TTS lets you start speaking as soon as you have enough text to form a stable chunk.

The basic loop looks like this:

1. Agent emits text deltas or sentence fragments.

2. You buffer text until the chunk is stable enough to synthesize.

3. You send the chunk to TTS.

4. You play the returned audio into the same realtime session that drives the avatar.

Chunking is the part that usually needs tuning. If chunks are too small, the speech sounds choppy. If they are too large, first-audio latency gets worse. A good starting point is sentence boundaries plus a fallback timer when the agent is generating long, unpunctuated output.

You also need a cancellation story. When the user interrupts, stop generation, discard pending TTS chunks, and flush any queued audio. Otherwise the avatar keeps talking with stale context.

For UI, treat captions and speech as related but not identical. Captions can show the full final text even while TTS is still catching up. The visual avatar should follow the audio stream, not the caption stream.

Wire the avatar to the same realtime transport

In a TypeScript app, the avatar should subscribe to the same session lifecycle as STT and TTS. That means you create one session object that owns microphone capture, transcript events, synthesized audio, and playback state. The avatar is then just another consumer of the audio stream.

At the transport level, this typically means one of two things:

• WebRTC, when you want low-latency media transport and browser-native audio/video handling.

• Server-mediated streaming, when your app already has a voice-agent backend and you want explicit control over the pipeline.

Either way, the invariant is the same: one authoritative timeline for speech. Do not let captions, audio playback, and avatar animation each invent their own clock.

A few things to watch for:

• Barge-in: user speech should immediately pause agent speech and reset the avatar’s speaking state.

• Backpressure: if the browser cannot keep up with audio frames or caption updates, drop stale partials first.

• Reconnection: on reconnect, restore session state carefully; do not replay already-spoken audio unless you explicitly want that behavior.

• Latency budget: STT, agent generation, and TTS each contribute delay. Measure them separately.

A simple debugging trick is to log timestamps at each transition: microphone frame received, partial transcript emitted, final transcript committed, first TTS chunk ready, first audio frame played, and avatar speak state entered. If any of these drift apart, you have a clear place to look.

How Protoface fits in this architecture

This is where a developer-facing avatar layer helps. With Protoface, you do not have to build the lip-sync and session orchestration logic yourself. For a TypeScript voice app, the most direct fit is usually an iframe embed when you want a browser-native avatar without exposing credentials, or the REST API when your backend is creating and managing sessions.

If you already have STT and TTS in place, Protoface can sit on top of that media flow: your app keeps ownership of captions and conversation logic, while the avatar presentation is handled as part of the realtime session. The key benefit is that the avatar stays synchronized to the actual audio stream instead of a separate animation clock.

For the API side, session creation is a standard authenticated request from your backend:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

The exact request fields depend on the session model you choose, so use the docs for the current schema. If you prefer Python for orchestration, there is also a Python SDK for creating avatars and sessions programmatically, which is useful for backend workflows and automation.

If your voice agent runs in LiveKit, the plugin path is even simpler: drop the Protoface avatar into the agent pipeline and let the plugin handle synchronized video output. That keeps your application logic focused on STT, turn-taking, and response quality instead of media plumbing. The integration details and examples are in the relevant plugin repo.

Practical implementation notes for TypeScript apps

There are a few non-obvious choices that make these systems more stable:

• Use a single event bus for transcript, audio, and avatar state. Redux, Zustand, RxJS, or a small typed event emitter all work; the important part is determinism.

• Normalize all timestamps to a common clock on the server or session layer.

• Separate transport errors from conversation errors. A failed TTS request is not the same as an invalid user message.

• Prefer streaming APIs over polling. Polling makes partial captions and interruption handling worse.

Also be careful with browser audio playback. Autoplay policies, device selection, and echo cancellation can all affect perceived latency. If your app captures microphone input and plays synthesized speech in the same tab, test on real devices, not just desktop Chrome with local audio.

If you need a reference implementation or want to compare integration patterns, the quickstarts linked from the project README are a reasonable starting point, and the main documentation lives at the docs site.

Conclusion

Building live captions, STT, and TTS into a TypeScript voice avatar app is mostly an exercise in keeping one realtime conversation timeline consistent across multiple subsystems. Capture audio once, stream transcription incrementally, synthesize speech in chunks, and make the avatar follow the audio, not an independent animation loop.

If you are adding an interactive avatar to a voice agent or web experience, start by nailing the conversation state machine and interruption behavior. Then integrate the avatar layer on top of that stable pipeline.

For implementation details, current API shapes, and examples, check docs.protoface.com. If you want to plug an avatar into a LiveKit agent or compare integration options, the relevant GitHub repos linked from the docs and quickstarts are the fastest path.