Introduction

If you already have a voice agent on iOS, the next obvious upgrade is to give it a face. Not a static avatar and not a separate video call stream bolted on afterward, but a synchronized talking face that tracks the agent’s speech in real time. That changes the interaction model: users can read intent from facial motion, the agent feels less like a black box, and your app can present a more polished conversational surface without building an entire media stack from scratch.

This post shows the moving parts you actually need to wire together in a SwiftUI app: microphone capture, a realtime transport, speech synthesis timing, and a video surface that stays in sync with the audio. By the end, you should understand how to embed a voice-enabled fintech avatar into an iOS app, what WebRTC is doing for you, and where the usual integration bugs show up.

The architecture: voice, video, and timing need to be coupled

The key thing to internalize is that a “talking avatar” is not just a video player. In a voice agent, the model or agent layer produces text or token streams, the TTS layer turns that into audio, and the avatar layer needs to animate mouth movement against that audio in a way that feels coherent. If the avatar lags behind the audio by even a few hundred milliseconds, the illusion breaks quickly.

In practice, WebRTC is a good fit because it gives you low-latency media transport and jitter handling for live audio/video. On iOS, your app is typically responsible for:

• capturing user audio and sending it to the agent session;

• receiving the agent’s audio back;

• rendering the avatar video track;

• keeping the UI responsive when network conditions change.

SwiftUI is fine for the app shell, but the actual media rendering usually lives in an interop layer that can host a video view from WebRTC or a related SDK. The important part is not the exact view class; it’s making sure the video surface can be updated independently of your declarative UI state so it doesn’t stutter during layout changes.

SwiftUI integration: keep the UI thin and the media pipeline explicit

A clean way to structure the app is to separate concerns into three layers:

1. Session orchestration — create or join a realtime avatar session, retrieve connection data, and manage lifecycle.

2. Media transport — establish the WebRTC connection, publish local mic audio, and subscribe to remote audio/video.

3. Presentation — bind connection status and transcript state into SwiftUI, while the actual video render surface stays in a dedicated UIKit or WebRTC-backed view.

For fintech workflows, this separation matters because you usually want a strongly controlled interaction boundary. Think of a KYC flow, card dispute triage, or account support: the avatar is the conversational layer, but the app still owns navigation, auth, and any sensitive state transitions.

A minimal SwiftUI skeleton looks like this:

import SwiftUI

}

import SwiftUI

}

import SwiftUI

}

The details inside AvatarCallViewModel depend on your WebRTC stack and how you obtain session credentials. The practical advice is to keep connection setup off the main thread, treat the media layer as asynchronous, and model state transitions explicitly: idle, connecting, connected, reconnecting, and failed.

Two implementation details are worth calling out:

• Audio session handling: configure AVAudioSession early and deliberately. In a fintech app, you don’t want speaker/mic behavior to change unexpectedly when a user gets a call or backgrounding occurs.

• Lifecycle discipline: release the peer connection cleanly when the screen disappears. WebRTC objects retain resources longer than SwiftUI views do, so ownership should be explicit.

WebRTC and lip sync: what actually has to line up

For a conversational avatar to feel credible, there are three clocks to keep close: the agent’s speech generation, the audio playback clock, and the avatar animation clock. If the system is designed well, the avatar animation is driven by the same realtime media session that carries the audio, not by a separate polling loop or UI timer.

That has a few practical consequences:

• Prefer stream-based updates over batched ones. You want the avatar to start moving as soon as audio is available, not after a full response completes.

• Avoid artificial buffering. Extra buffering can smooth jitter, but too much of it makes the face feel detached from the voice.

• Design for interruption. Users will cut the agent off mid-sentence. Your app should support barge-in without leaving the avatar in a half-open mouth state.

If you are already using a voice agent framework, check whether it exposes a media track abstraction or a plugin interface. That is usually the cleanest place to attach the avatar layer because it lets the existing agent continue handling ASR, dialogue policy, and TTS while the avatar consumes the same realtime audio stream.

Session setup with the REST API

In most apps, you should create sessions server-side so your API key never ships in the mobile client. The mobile app then receives a short-lived token or session payload that is safe to use on-device. The exact request/response fields are documented, but the pattern is familiar: authenticated backend creates an avatar session, frontend joins it, and the media layer connects using the issued session details.

An illustrative server call looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

The response shape is intentionally not shown here because you should rely on the current docs for the exact fields. The operational point is that your backend owns the secret, creates the session, and hands the app only what it needs to connect. That is the right model for fintech, where mobile clients should never be trusted with long-lived credentials.

If you need to inspect or manage usage, the developer dashboard is useful for session visibility and API key management, but it should not change the architecture of your app. Keep production traffic on your backend and treat the dashboard as an operational tool.

Where Protoface fits in this flow

Protoface is the layer that supplies the avatar session and the realtime talking face so you do not have to build lip-sync animation or video session orchestration yourself. For an iOS app, the main value is that your existing voice agent can keep doing the conversational work while the avatar is attached as a synchronized media surface. The public docs at docs.protoface.com cover the current REST and SDK shapes.

If your backend is Python, the SDK is a straightforward way to manage avatars and sessions programmatically. A representative flow looks like this:

from protoface import Client

print(session)

from protoface import Client

print(session)

from protoface import Client

print(session)

Again, treat the field names above as illustrative and confirm the exact method signatures in the SDK docs. The point is that session creation stays server-side, and your iOS app only receives the minimal connection data needed to join the live media session.

Practical iOS gotchas

Most integration problems are not about the avatar itself; they are about mobile media plumbing. A few things to test early:

• Backgrounding: decide whether the session should pause, reconnect, or continue when the app goes to the background.

• Network switching: Wi-Fi to cellular transitions can disrupt the peer connection; verify your reconnection logic.

• Audio route changes: plug/unplug headphones, Bluetooth handoff, and speakerphone toggles all need to preserve the session.

• Permission UX: mic permission denial should degrade cleanly and explain what the user needs to enable.

For a fintech experience, there is also a product consideration: not every screen needs the face visible. Sometimes the best pattern is to show the avatar only during high-friction conversational flows, then collapse back to a standard UI once the user has completed the task. That keeps the app efficient and avoids overusing a heavyweight media surface where it is not needed.

Conclusion

Adding a voice-enabled avatar to an iOS app is mostly an exercise in disciplined realtime architecture: keep the UI thin, keep session creation server-side, and make sure your audio/video transport and animation share the same low-latency path. SwiftUI gives you a clean presentation layer, while WebRTC handles the media plumbing that makes the interaction feel live.

If you want to implement this in your own app, start by defining the session lifecycle on your backend, then wire a small SwiftUI surface around the media view, and test the ugly cases first: interruptions, route changes, and reconnects. The docs at docs.protoface.com are the right place to confirm the current API shapes and integration options, and the quickstart repos linked from the project README are useful if you want a working reference before adapting it to iOS.