Introduction

Adding a “talking face” to a realtime voice agent sounds simple until you try to make the pieces line up: speech-to-text needs to arrive quickly enough to drive turn-taking, text-to-speech needs to start streaming before the full response is known, and the avatar has to lip-sync on the same timeline as audio playback. In practice, this is a streaming systems problem, not just a UI problem.

This post walks through the architecture I’d use in Swift for a banking assistant: microphone capture, streaming STT, agent response generation, TTS, and avatar video playback with lip-sync. By the end, you should have a concrete model for how these systems fit together, where latency comes from, and what to watch out for when you put a realtime avatar in front of users.

Start with the latency budget, not the UI

For conversational systems, the user experience is mostly determined by timing. A good target is:

• partial STT results within a few hundred milliseconds

• first TTS audio chunk shortly after the agent starts responding

• avatar video frames updated in lockstep with audio, not on a separate timer

If the avatar starts moving before audio is ready, the mismatch is obvious. If audio starts but the face lags behind, users read it as jank. So the design rule is: one source of truth for the agent’s speaking timeline, then drive both audio and mouth shapes from that timeline.

In a banking assistant, this matters more than in many other apps because the user often asks short, time-sensitive questions: “What’s my balance?”, “Did that transfer go through?”, “Freeze my card.” The assistant needs to feel responsive without looking sloppy, because the interface is effectively part of the trust boundary.

How the realtime stack fits together

A practical architecture in Swift looks like this:

1. Capture microphone audio in short frames.

2. Stream frames to STT and receive partial transcripts.

3. Feed the transcript into your agent logic or LLM.

4. Generate a streamed TTS response as text arrives.

5. Send the synthesized audio to the avatar layer so video lip-sync follows the same playout clock.

6. Render the avatar video in a native view, WebRTC view, or embedded surface.

The key engineering point is that STT and TTS are both streaming systems. Do not wait for a full user utterance if your STT provider can emit partials; partials let you detect endpointing sooner, reduce turn latency, and sometimes interrupt or barge in correctly. Likewise, do not wait for the full assistant message if your TTS engine can stream audio incrementally.

Swift audio capture and streaming STT

On Apple platforms, AVAudioEngine is usually the simplest way to capture low-latency microphone audio. You want small buffers, consistent sample rate conversion, and a transport that can keep up with realtime delivery. For banking use cases, I would strongly prefer an always-on push model over polling.

A minimal capture setup looks like this:

import AVFoundation

try engine.start()

import AVFoundation

try engine.start()

import AVFoundation

try engine.start()

The exact STT transport depends on your provider, but the mechanics are always the same: keep buffers small, normalize format once, and preserve order. If you introduce extra resampling or dispatch hops, the delay adds up quickly.

Two gotchas show up often:

• Endpointing: If your STT model is too eager to finalize, users will feel like the assistant keeps interrupting them. If it is too slow, the agent feels sluggish. Tune end-of-speech detection for conversational speech, not dictation.

• Noise and echo: If the assistant’s own voice leaks back into the mic, you can get feedback loops or false transcriptions. Use echo cancellation where possible and keep microphone and speaker routing disciplined.

TTS and mouth movement need the same clock

Once you have the assistant’s text, the next mistake is to treat TTS and lip-sync as separate problems. They aren’t. A good avatar system consumes audio time, not just text. The timing of phonemes, visemes, and frame delivery needs to be aligned to the actual audio that will reach the user.

There are two common patterns:

• Audio-first: stream TTS audio to the client, then drive the avatar’s mouth shapes from that audio timeline.

• Audio-plus-markers: stream audio plus timing metadata from the TTS or avatar layer, and render video against those markers.

In both cases, what matters is consistent playout. If your client buffers 500 ms of audio, the avatar should move as if that buffer exists; otherwise the lips will look early. If your app supports interruption, you need a clean way to stop both audio and video at the same boundary.

From an implementation perspective, this means your Swift client should treat TTS playback as a session with explicit state:

• idle — no agent audio

• speaking — audio chunks arriving and being queued

• interrupted — stop playback and reset mouth state

• recovering — resume with a new turn

That state machine is boring, but it prevents a lot of visual glitching.

Banking-specific details: trust, privacy, and turn boundaries

A banking assistant is not just a demo bot with a nicer face. Users expect clear turn boundaries, predictable responses, and no ambiguity about what the system is doing. A few practical rules help:

• Keep the assistant’s spoken responses short unless the user explicitly asks for detail.

• Make confirmations explicit for irreversible actions.

• Do not let the avatar animate while the system is silently waiting on backend authorization.

• Design for barge-in: if the user starts talking, stop the current response cleanly.

If you are exposing account data, the avatar layer should never become your auth boundary. It is presentation. The actual banking workflow still needs your normal backend checks, scoped session handling, and audit logging.

This is also why embedded or browser-hosted avatar surfaces need careful origin and session controls. You do not want the visual layer to become an accidental shortcut around your security model.

Where Protoface fits without changing your agent architecture

This is the part where a dedicated avatar layer saves time. Protoface gives you a realtime avatar surface that can sit underneath your agent logic without forcing you to build the lip-sync stack yourself. For Swift teams, the important detail is that you can keep your existing STT/TTS/agent pipeline and hand off the avatar video concern to a platform designed for synchronized talking faces.

If your assistant already runs in a voice-agent stack, the LiveKit path is often the cleanest way to add the face. The quickstart examples are useful if you want to see how a realtime voice agent hands off audio and turn state to an avatar. If you are already using Pipecat, the Pipecat integration guide shows the same idea from that ecosystem: the avatar is just another streaming service in the pipeline.

For direct control, the REST API at api.protoface.com is the right surface for creating avatars and managing realtime sessions. A minimal request shape looks like this:

curl -X POST https://api.protoface.com/<endpoint> \
}'

curl -X POST https://api.protoface.com/<endpoint> \
}'

curl -X POST https://api.protoface.com/<endpoint> \
}'

The exact path and fields depend on the session type, so use the docs for the canonical schema. The important part is the workflow: create or open a session, stream the agent’s output through it, and let the avatar layer manage the synchronized talking video.

If you prefer Python for orchestration, the SDK gives you a programmatic way to create sessions or manage avatars from backend services. The pattern is straightforward:

from protoface_sdk import Client

)

from protoface_sdk import Client

)

from protoface_sdk import Client

)

That is often enough to keep Swift focused on media capture and playback while your server owns identity, session creation, and business logic.

Implementation notes for Swift clients

If you are building the client side in Swift, the pragmatic approach is to separate concerns into three layers:

• Media layer: microphone capture, speaker playback, and video rendering.

• Session layer: websocket/WebRTC/SDK connection management and reconnect logic.

• Conversation layer: transcript state, turn detection, and UI events.

Keep the media layer deterministic. Do not let UI rendering or network retries directly manipulate audio buffers. Instead, pass state changes through the session layer and let the media layer render whatever playout state is current.

For avatars specifically, the most common failure mode is drift: audio and mouth motion start aligned, then slowly diverge because one path buffers differently than the other. The fix is usually to centralize timestamps and make all playback use the same monotonic clock. If your stack crosses between native Swift and a browser-based surface, be especially careful about which side owns buffering and which side owns animation timing.

Conclusion

Realtime banking assistants are hard because they combine speech recognition, response generation, audio synthesis, and video synchronization into a single latency-sensitive system. The safest way to build one is to treat the avatar as part of the media pipeline, not as decoration, and to keep timing explicit from microphone frame to final frame of lip movement.

If you are implementing this in Swift, start with a clean STT/TTS streaming loop, add interruption and endpointing early, then plug in an avatar surface once the timing model is stable. For the Protoface side, the public docs at docs.protoface.com are the right place to confirm API schemas, session behavior, and integration details. From there, you can choose the surface that matches your stack: REST for backend control, SDKs for orchestration, or a voice-agent plugin if you are already on LiveKit.