How to Connect STT, TTS, and Avatar Streaming in an Angular Receptionist App
Angular receptionist app architecture for streaming STT, TTS, and lip-synced avatar video with realtime turn-taking and barge-in support
Introduction
Building a receptionist app sounds simple until you wire together the three realtime paths that make it feel natural: speech-to-text (STT) for the caller’s input, a large language model or dialog layer for turn-taking, and text-to-speech (TTS) plus video streaming for the avatar’s response. The hard part is not any single subsystem; it’s keeping them synchronized so the user sees a face that starts talking when audio starts, stops when audio stops, and doesn’t drift out of sync when the conversation becomes interruptible.
This post walks through a practical architecture for an Angular receptionist app that captures microphone audio, sends it to a voice agent, receives streamed audio back, and renders a lip-synced avatar video surface. By the end, you should have a clear mental model for the data flow, the browser constraints, and the integration points you need to make the app feel realtime instead of stitched together.
System architecture: separate capture, agent, and render responsibilities
For a receptionist flow, treat the browser as an audio/video endpoint, not as the agent itself. The browser should:
capture mic input and play back remote audio,
maintain a WebRTC or similar realtime media connection,
render the avatar video element or iframe,
handle UI state such as listening, speaking, reconnecting, and errors.
The agent backend should:
perform STT on the caller stream,
manage turn-taking and interruption logic,
generate text responses,
produce TTS audio, and
stream avatar video in sync with that audio.
That separation matters because the avatar is not “doing the speech recognition” and the browser is not “running the bot.” You want a media pipeline where audio and video are generated from the same conversational state, so the mouth movement and audible output are aligned at the transport level rather than patched together after the fact.
Turn-taking and synchronization: what actually needs to stay in lockstep
In a receptionist app, the key failure modes are subtle:
Latency mismatch: STT finishes early, but TTS starts late, which makes the avatar look unresponsive.
A/V drift: video is rendered from one clock and audio from another, so lip sync degrades over time.
Interrupt handling: the user talks over the agent, but the agent keeps speaking because the browser never signaled barge-in.
State desync: UI says “listening” while the media pipeline is still playing the previous response.
The right pattern is to make one entity the source of truth for conversational state, then stream media from that state. In practice, that means your voice agent decides when to start, pause, cancel, or resume speaking, and your avatar video follows the same lifecycle. If the user interrupts, you stop both the TTS audio and the corresponding avatar stream immediately. Don’t try to “catch up” the video locally in Angular; the backend should emit media that is already synchronized.
Angular client: capture mic input and manage the session lifecycle
Angular’s job is mostly orchestration. You acquire mic access, establish the session, and attach the remote media streams to the UI. A minimal component usually has three states: idle, connected/listening, and speaking.
Here is a stripped-down shape of the client logic:
Two practical notes:
Use
playsinlineand explicit user gestures for playback. Browsers are still strict about autoplay for audio/video.Keep your call controls deterministic. If the session reconnects, decide whether to preserve the conversation state or restart it. “Half-connected” states are where most demo bugs live.
If your backend is producing both audio and video, prefer attaching the tracks as they arrive instead of polling for readiness. That reduces UI complexity and avoids race conditions where the avatar element exists but the media stream has not negotiated yet.
STT and TTS integration: keep the media boundary clean
The simplest mental model is: mic audio goes in, transcripts come back, a response is generated, then TTS audio comes out. The browser should not care which vendor performs STT or TTS as long as the transport contract is consistent.
For a low-latency receptionist experience, a few constraints matter:
Streaming STT beats batch transcription. Interim results let the agent start reasoning before the user fully finishes speaking.
Streaming TTS beats “speak the whole sentence after generation.” The user sees and hears the avatar sooner.
Interruptible playback is not optional. Receptionists need barge-in behavior so users can correct themselves or ask a follow-up mid-response.
One conversation clock. If your text, audio, and video are each buffered independently, the UI will feel laggy even if each component is fast in isolation.
When implementing this in Angular, avoid trying to synthesize media locally from text fragments. Keep the frontend thin: it should subscribe to remote streams and surface call state. If you need transcript display, render the partial STT results as ephemeral UI state, but don’t let them drive the avatar animation directly.
How Protoface fits: add the avatar where the voice agent already lives
This is the point where Protoface is useful. If you already have a voice agent that handles STT, dialog, and TTS, you can add a synchronized talking face without rebuilding the media pipeline yourself. The most direct integration is the LiveKit Agents plugin from the plugin repository and the corresponding Pipecat integration guide, which are aimed at developers who already have a realtime voice stack and want to drop in a lip-synced avatar.
In that model, your agent continues to own conversation logic, while the avatar stream is coupled to the agent’s output audio. The important property is synchronization: the face moves with the same speech that the user hears, rather than being animated by a separate timer on the client.
A minimal Python-side setup often looks like this in shape, though exact constructor arguments and fields should come from the docs:
If you are using the hosted API directly, the REST surface is also straightforward: create or manage an avatar, then create a realtime session tied to that avatar. Authentication uses your API key server-side, and the browser never needs to see it. A skeletal request looks like this:
Use the REST API or Python SDK when you want explicit lifecycle control from your own backend. Use the agent plugin when the avatar should be a part of the live conversation engine itself. In both cases, the design goal is the same: keep the avatar and speech stream bound to the same session so the browser only has to render one coherent realtime surface.
Practical Angular implementation details and gotchas
A few implementation details make a real difference in production:
Debounce UI state transitions. Realtime systems can emit short-lived connect/disconnect events during renegotiation. Don’t flicker the UI on every transient state.
Prefer a single media element per track type. Replacing
srcObjectis simpler than managing many detached elements.Handle page visibility and mobile audio routing. On iOS in particular, audio playback behavior can be unforgiving if the call starts without a user gesture.
Log timestamps at each boundary. Capture when mic audio was sent, when STT finalized, when TTS started, and when the video track attached. This is the fastest way to debug perceived lag.
Separate rendering concerns from transport concerns. Angular should render state; the transport layer should own retries, media negotiation, and cleanup.
If you need a hard rule of thumb: anything involving realtime media should be idempotent or restartable. Network churn happens. The app should be able to reattach to a session without forcing the user to refresh the page and start over.
Conclusion
Connecting STT, TTS, and avatar streaming in an Angular receptionist app is mostly an exercise in clean boundaries. Let the browser capture and render media, let the agent own conversational state and turn-taking, and keep the avatar stream synchronized with the speech stream rather than treating video as a separate animation problem.
If you want to implement this without assembling every piece yourself, start with the relevant quickstart in the GitHub examples, then read the public documentation at docs.protoface.com. The fastest path is usually to get one end-to-end call working first, then harden reconnects, interruption handling, and UI state after the media pipeline is stable.