Reducing Avatar Start Latency in iOS Fintech Apps with Swift and Streaming TTS
Cut iOS avatar start latency with Swift: prewarm AVAudioSession, stream TTS, move work off main thread, and sync turn state.
Introduction
When an iOS fintech app adds a talking avatar to a voice flow, the first thing users notice is not the lip sync quality or model choice. It is the time between “tap” and “the face is alive.” That start latency is made up of several different delays: audio capture, speech synthesis startup, network setup, avatar session creation, video decoding, and the app’s own UI work. If any one of those stages blocks on the critical path, the experience feels sluggish even when the backend is “fast enough” in aggregate.
This post is about reducing that startup latency in a practical way. By the end, you should be able to reason about where the delay is coming from, restructure the client flow so audio and video can start sooner, and integrate a streaming TTS pipeline with a realtime avatar surface without making the iOS app feel heavy.
Where avatar start latency actually comes from
In a realtime avatar app, “startup” is not a single event. It is usually a chain:
The user taps to start a conversation.
The app activates audio session permissions and configures playback/recording.
The agent receives text or a turn boundary.
The TTS service produces the first audio chunk.
The avatar session starts or resumes and binds to the audio stream.
The client receives the first video frames and decodes them.
For iOS apps, the most common mistake is treating all of that as one blocking operation. You do not want to wait for full synthesis before showing anything. You want to overlap work:
prepare the audio session before the user needs it,
open the realtime transport early,
start streaming the first TTS chunk as soon as you have enough text,
render the avatar container immediately, even if the face is still connecting.
The central idea is to shorten the critical path, not necessarily the total work. If the UI is waiting on audio synthesis, and audio synthesis is waiting on session creation, and session creation is waiting on a network round-trip, you will lose a lot of perceived responsiveness even if each step is individually reasonable.
Use streaming TTS, not full-buffer TTS
The biggest practical win is to stop treating speech as an all-or-nothing asset. Full-buffer TTS waits until the entire utterance is synthesized before playback starts. Streaming TTS returns audio incrementally, so playback can begin after the first chunk. That matters because users perceive the system as responsive when they hear the first phoneme quickly, not when the final byte of audio arrives.
There are a few implementation details that matter on iOS:
Chunk size affects startup: smaller chunks reduce time to first audio, but too-small chunks increase overhead and can cause choppy buffering.
Jitter buffering still matters: even with streaming, you need a small buffer to absorb network variation before driving the player.
Do not block UI on synthesis completion: render the avatar UI and play a “connecting” state while audio is in flight.
Keep text boundaries natural: if your upstream agent emits text token-by-token, you often get better results by streaming sentence fragments or phrase-level chunks rather than raw subword tokens.
From a systems perspective, the ideal path looks like this:
The exact API shape will vary by provider, but the pattern is the same: begin streaming audio as soon as you have enough text to speak, and avoid waiting for the “entire response” before making the app feel alive.
Make the iOS client do less work on the main thread
A lot of avatar startup latency is self-inflicted on the client. iOS apps often spend too long on the main thread during a conversation start: preparing views, decoding assets, configuring AVAudioSession, or doing synchronous network calls. In a fintech app, that is especially easy to do because teams tend to be careful with permissions, compliance UI, and state management. The result is a clean architecture that still starts slowly.
There are three client-side rules that help consistently:
1. Prewarm the audio pipeline. Configure AVAudioSession before the user hits the action that starts the conversation. If policy allows it, keep the session ready in the background while the user is on the prior screen. Do not do session setup, permission prompting, and network connection setup all in one button handler.
2. Render immediately, then connect. Your avatar container should appear right away, with a skeletal or placeholder state. Do not wait for the first decoded video frame to allocate the view hierarchy. The user should see that the app accepted the action within a few hundred milliseconds, even if the face takes longer to animate.
3. Move network and decoding off the main actor. Open WebRTC or websocket connections, parse session metadata, and buffer streaming audio on background tasks. Keep the main thread for view state only.
A useful profiling pattern is to instrument each stage separately:
tap to UI visible,
UI visible to session request sent,
session request sent to first audio chunk,
first audio chunk to first video frame,
first video frame to stable playback.
That breakdown makes it obvious whether your bottleneck is client work, TTS startup, avatar session creation, or network transport.
Coordinate the agent, not just the UI
Streaming TTS only helps if the agent itself is producing text in a stream-friendly way. In voice systems, the agent often controls when a turn starts and when enough content exists to speak. If your backend waits for a long-form completion before emitting the first sentence, the client cannot hide that delay.
The useful pattern is incremental turn construction:
generate the first clause early,
stream it to TTS immediately,
continue generating the rest of the answer in parallel,
only finalize when the thought is complete.
This works best when the avatar and the audio are driven from the same realtime turn state. Otherwise, you can get desynchronization: the face starts speaking while the agent is still planning, or the TTS stream advances faster than the avatar can animate. The fix is not more buffering; it is tighter orchestration of turn boundaries and audio timing.
If you are using a provider-agnostic agent stack, this is the layer where you decide whether the avatar should start on partial text or wait for a punctuation boundary. For most conversational UI, a short, coherent phrase is the right compromise. It reduces perceived latency without making the speech sound prematurely cut off.
How Protoface fits into this flow
This is where Protoface is useful: it gives you a developer-facing realtime avatar surface that can be attached to a voice agent and driven as part of the same conversation flow. If you are already using LiveKit Agents, the plugin integration is the most direct path when you want the agent to gain a synchronized talking video face with minimal glue code. The plugin and docs show the exact setup; the point here is the architecture, not a specific API shape.
A lightweight integration looks like this in principle: start the voice pipeline, start the avatar session, and feed the streaming speech into the avatar-enabled agent path. A sketch with the Python SDK would look like:
For teams that prefer lower-level control, the REST API can create and manage avatars and realtime sessions directly. A simple authenticated request pattern is:
Again, the exact fields are documented in the docs, but the architectural value is that you can separate session creation from the rest of the app flow and start the avatar as soon as the agent has enough text to speak.
Trade-offs and gotchas
A few implementation details are easy to miss:
Do not over-buffer: buffering too much audio increases startup latency and can make the avatar feel delayed even when playback is smooth.
Watch for network handshakes: if you create the avatar session only after the user taps, you are adding a round-trip to the critical path. Pre-create or prewarm when your UX allows it.
Handle interruption cleanly: phone calls, app backgrounding, and audio route changes can force you to restart the audio pipeline. Make sure the avatar can reconnect without a full teardown.
Keep voice and video in sync: if the TTS stream pauses or restarts, the avatar timing should follow the same turn state rather than free-running.
For fintech apps specifically, there is also a compliance angle: if you are using customer or account data to drive the conversation, make sure your session lifecycle matches your privacy and retention model. Realtime avatars should not change your data handling assumptions; they should fit into them.
Conclusion
Reducing avatar start latency is mostly about treating the startup path as a pipeline, not a single request. Prewarm the audio session, render the UI immediately, stream TTS instead of waiting for full synthesis, and keep the agent and avatar turn state aligned. If you do those things, the avatar will feel much faster even before you optimize the backend further.
If you want to wire this up in a real app, start with the docs, then pick the integration surface that matches your stack: the LiveKit plugin for an agent-centric voice flow, or the REST API/Python SDK for explicit session control. The implementation details are in docs.protoface.com, and the quickstarts linked from the project README are a good way to validate the end-to-end path before you fold it into your iOS app.