Introduction

If you are wiring a realtime avatar into an Angular app backed by FastAPI, the hard part is usually not “drawing a face.” It is keeping three independent streams aligned under latency and failure: ASR/STT turns user audio into text, the LLM turns text into intent, and TTS turns the response back into audio that must stay synchronized with avatar motion. When this goes wrong, you do not get a clean exception; you get clipped speech, delayed lips, silent playback, or an avatar that keeps animating after the audio path has already died.

This post is about debugging that pipeline pragmatically. By the end, you should be able to trace failures from browser capture all the way through FastAPI, identify where timing and buffering go off the rails, and make the integration resilient enough to survive real network conditions. I will also show where Protoface fits if you want to offload the avatar/session layer instead of building every streaming edge yourself.

First, model the pipeline as separate clocks

Most avatar bugs come from assuming “audio streaming” is one thing. It is not. In a typical browser-to-backend voice flow, you have at least four clocks:

• Capture clock: the browser samples microphone audio, usually 16-bit PCM after resampling from device-native rate.

• Transport clock: WebSocket/WebRTC packets arrive in bursts, with jitter and occasional reordering or drop.

• Inference clock: STT and TTS each have their own buffering and chunking behavior.

• Playback clock: the avatar’s motion must track the audio that the user actually hears, not the text that was just produced.

If the system is “working” but the avatar lip-sync is off, the first question is not “is TTS broken?” It is “which clock drifted?”

In practice, I like to log five timestamps per turn:

• mic chunk captured

• chunk received by FastAPI

• STT partial/final emitted

• TTS audio started

• first audio frame played back

Once you have those, most bugs become obvious. If capture-to-receive is high, the browser or transport is the issue. If receive-to-STT-final is high, the STT vendor or chunk size is the bottleneck. If TTS-start-to-playback is high, your player or buffering strategy is wrong. If everything is fast except the avatar motion, your sync layer is dropping timestamps or using the wrong audio reference.

Debugging Angular capture and playback

Angular itself is usually not the cause, but it is where a lot of hidden browser behavior leaks in. A few practical failure modes show up repeatedly:

• Sample rate mismatch: your audio context is 48 kHz, but the backend expects 16 kHz PCM. If resampling is wrong, STT quality drops sharply.

• Chunk sizes too small: sending 10–20 ms frames over a WebSocket can create overhead and jitter amplification. 40–100 ms is often a more stable starting point for debugging.

• AudioContext lifecycle issues: Safari and Chrome both require user gestures to start audio. If you initialize too early, playback may be silently suspended.

• Autoplay restrictions: the TTS stream can be fine while the browser refuses to play it because the tab never received a user-initiated audio unlock.

• State changes in Angular zone: frequent audio callbacks can trigger unnecessary change detection and introduce UI jank. Keep raw streaming logic outside Angular’s zone when possible.

When debugging the browser, inspect actual PCM boundaries. Do not trust “it sounds okay” as evidence that the backend is receiving what you think. A 24-bit or float stream accidentally treated as 16-bit signed PCM can produce noise that some STT engines still partially decode, which makes the bug look like a model problem instead of an encoding problem.

A minimal capture path should make its assumptions explicit:

// Angular service sketch: capture mic, resample, send PCM16 frames
});

// Angular service sketch: capture mic, resample, send PCM16 frames
});

// Angular service sketch: capture mic, resample, send PCM16 frames
});

The exact implementation details are not the point; the key is that you control sample rate, frame length, and encoding explicitly.

FastAPI: separate transport, STT, and TTS responsibilities

On the backend, the most common mistake is to collapse the whole session into one coroutine and hope backpressure sorts itself out. It rarely does. A cleaner pattern in FastAPI is:

1. accept audio frames over WebSocket or HTTP upload

2. buffer frames in a small in-memory queue

3. feed an STT worker incrementally

4. emit partial transcripts as events

5. trigger TTS only on stable utterance boundaries or model turn boundaries

That separation matters because STT and TTS often have very different failure modes. STT is sensitive to input continuity and silence trimming; TTS is sensitive to request size, text normalization, and whether you send full sentences or partial fragments. If you start TTS on every partial transcript, you will get repeated starts, truncated speech, and unnecessary avatar motion restarts.

For debugging, prefer an event log over ad hoc prints. For each session, record a structured trace with a correlation ID:

• session opened

• audio frame count and byte size

• STT partial text

• STT final text

• TTS request text

• TTS first audio byte time

• stream completion or error

This makes it much easier to answer questions like: did the model hear the user, or did the browser stop sending audio after tab focus changed?

A short FastAPI endpoint sketch:

from fastapi import FastAPI, WebSocket

await ws.close()

from fastapi import FastAPI, WebSocket

await ws.close()

from fastapi import FastAPI, WebSocket

await ws.close()

For real systems, add a timeout on silence, a max buffered duration, and a clear state machine. A voice agent that never finalizes an utterance will keep accumulating context and look “laggy” even if every individual API call succeeds.

How audio pipeline failures show up in practice

Once the basic transport works, the failures become more subtle:

• Clipped speech: your TTS stream starts before the browser has unlocked audio, or you are dropping the first frames while waiting for a buffer threshold.

• Robotically delayed lip sync: the avatar renderer is using transcript timing instead of audio timestamps.

• Double-speaking: an earlier partial response was never canceled when a newer turn started.

• Half-second pauses between words: TTS chunks are too small, or you are waiting for a full sentence before rendering any audio.

• STT hallucination on silence: your front-end gain control or noise suppression is overprocessing low-level audio.

A useful rule: if speech quality drops only under real network conditions, inspect buffering and cancellation first. If it drops even on localhost, inspect format conversion and chunking first.

For networked avatars, cancellation is important. If the user interrupts the agent mid-response, your stack should invalidate the in-flight TTS stream and stop avatar motion from the old turn. Otherwise you get a visual lag where the mouth keeps talking after the intent has already changed. That is a pipeline coordination bug, not a rendering bug.

Where Protoface helps without hiding the real problem

If you want the avatar/session layer to behave like infrastructure instead of a custom side project, the Protoface REST API and Python SDK give you a cleaner boundary: create an avatar, open a realtime session, and let the platform handle the synchronized talking face while your app keeps ownership of STT, TTS, and turn logic. That is useful when the bug surface is already large enough without also managing avatar timing and video delivery.

For example, you can provision sessions from your backend and keep secrets server-side:

import os

print(resp.json())

import os

print(resp.json())

import os

print(resp.json())

If you are using a LiveKit voice agent, the livekit-plugins-protoface plugin is the more relevant surface. It lets you drop a synchronized avatar into the agent path so the avatar stays aligned with the voice workflow instead of forcing you to hand-roll the video side. That is especially helpful when debugging because it reduces the number of places where audio/video clocks can diverge. The examples in the repo are the fastest way to see the expected event flow; if you need the broader integration references, start with the quickstart repository or the docs.

Even when you use the platform, the same debugging discipline still applies: trace capture, transport, inference, and playback separately. The main difference is that you are no longer responsible for stitching avatar sync logic into the critical path.

Conclusion

Debugging realtime avatar streaming is mostly about making invisible timing visible. If Angular capture, FastAPI transport, STT, and TTS are treated as one black box, you will chase symptoms. If you instrument each stage and keep sample rates, buffering, and cancellation explicit, the failures become diagnosable.

Start with structured timestamps, verify your audio format end-to-end, and use a state machine for turn transitions. Then decide which parts you actually want to own. If you need the avatar/session layer to be production-grade without building it yourself, check the public docs at docs.protoface.com and the relevant examples in the GitHub quickstarts.