Introduction

If you’re building a kiosk app in SwiftUI, you usually care about three things: the UI needs to stay simple, the experience needs to feel responsive, and the app has to survive real-world network conditions without turning into a support burden. Adding a talking avatar raises the bar a bit. Now you need live audio capture, streaming speech synthesis, synchronized lip movement, and a video transport that won’t fight with your app’s state model.

This post walks through a practical architecture for adding a talking avatar to a SwiftUI kiosk app using WebRTC for realtime media and streaming TTS for low-latency speech. By the end, you should have a clear mental model for how the pieces fit together, where the latency comes from, and what to keep off the main thread.

The core architecture: one voice pipeline, two realtime streams

A useful way to think about the system is that the avatar is not “just video.” It is the output of a voice agent pipeline that produces two synchronized streams:

• Audio: the synthesized speech, ideally streamed incrementally so the user hears the beginning of the response quickly.

• Video: the avatar face, driven by the same text/audio state so lip sync and expression timing stay aligned.

In practice, your kiosk app needs a transport layer for realtime media and a control plane for session setup. WebRTC is a good fit for the media path because it handles low-latency audio/video over variable networks and gives you congestion control, jitter buffering, and NAT traversal. For the speech side, streaming TTS matters because you do not want to wait for a full utterance before playback begins. A “generate then play” model will feel laggy in a kiosk flow, especially if the user expects interruption or quick turn-taking.

The main constraint is synchronization. If audio starts late relative to video, or if video stalls while audio continues, users immediately notice. So you want a single session boundary that owns both the audio stream and the avatar stream, rather than treating them as separate features glued together in the UI.

SwiftUI integration patterns that actually hold up

SwiftUI is a fine shell for this, but not the best place to put realtime media logic directly. Keep your view layer declarative and isolate the session, transport, and playback code in an observable object or a dedicated service class.

A common shape looks like this:

1. The app requests a session token or session details from your backend.

2. The kiosk view model starts a WebRTC session and subscribes to audio/video tracks.

3. The view renders a native video surface or an embedded player view.

4. User speech or kiosk events are forwarded to the agent as input, and the response streams back as audio plus avatar video.

For SwiftUI specifically, the important detail is that WebRTC renderers are usually UIKit/AppKit views under the hood. You typically wrap them in UIViewRepresentable or NSViewRepresentable instead of trying to render frames directly in SwiftUI. That keeps frame delivery off the main view diffing path and avoids layout churn during high-frequency updates.

import SwiftUI

}

import SwiftUI

}

import SwiftUI

}

That snippet is intentionally minimal. The key idea is that SwiftUI owns placement and sizing, while the realtime media stack owns frames. If you let the view hierarchy decide when to recreate the renderer, you will get avoidable disconnects.

WebRTC, latency, and the parts that usually go wrong

When people first add a realtime avatar, they often optimize the wrong thing. The biggest wins are rarely in the video path itself; they are usually in session startup and audio timing.

There are a few practical issues to plan for:

• ICE negotiation takes time: if the kiosk must establish a fresh peer connection on every interaction, startup feels slow. Reuse sessions when the product allows it, or at least separate session creation from user-facing prompts.

• Audio device setup can block: on macOS, permissions and device enumeration can introduce noticeable delay. Ask for permissions early, not at the moment the kiosk needs to speak.

• Streaming TTS is only useful if playback starts immediately: buffer just enough to avoid stutter, then begin playback. Waiting for “perfect” buffers destroys the benefit of streaming.

• UI work must not compete with frame rendering: do not decode audio, transcode video, or perform NLP in the SwiftUI main actor.

For a kiosk, I would also strongly recommend a fallback state for network loss. When the WebRTC session drops, your UI should present a deterministic “reconnecting” state instead of freezing on the last frame. Kiosk users interpret frozen media as a broken device, not a temporary network issue.

Streaming TTS and lip sync: keep the timing contract simple

The avatar can only look synchronized if the speech timeline is predictable. That means the service producing speech should expose incremental output, and the avatar renderer should consume a sequence that preserves utterance boundaries. You do not need perfect phoneme-level control in your app code, but you do need a stable contract: this text chunk corresponds to this audio chunk and this face motion sequence.

In a kiosk flow, the best pattern is usually:

• Collect user intent or text input.

• Send a single instruction to the agent.

• Let the agent stream the response instead of waiting for the whole message.

• Render audio and avatar frames as one conversation turn.

If you are building around your own speech backend, verify that it can stream partial audio rather than only returning a complete WAV or MP3. File-based TTS is easier to integrate, but it adds avoidable startup latency and makes interruption clumsy. For an interactive kiosk, latency is user experience.

import Foundation

}

import Foundation

}

import Foundation

}

The exact fields depend on your workflow, but the shape should be familiar: create a session server-side, return whatever token or connection metadata the client needs, and keep the API key off the kiosk device if you can avoid it.

Where Protoface fits: session creation and a managed realtime avatar surface

This is the part where a dedicated avatar layer saves time. Protoface gives you a developer-facing way to create and manage realtime avatar sessions without building the avatar rendering stack yourself. For a SwiftUI kiosk app, that means your app can focus on user flow and WebRTC playback while the avatar service handles the synchronized face/video side.

At the integration boundary, the useful pieces are straightforward: create a session through the REST API, keep authentication on the server, and connect the client to the realtime session metadata it receives. If you want to poke at the control plane directly, the API is documented at docs.protoface.com.

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

If you already have a voice agent in Python, the SDK path is similarly straightforward: create the avatar/session programmatically, then hand the connection details to your app. The useful part is that the session lifecycle stays explicit, which makes it much easier to reason about kiosk restarts, reconnects, and usage tracking.

from protoface import Client

print(session)

from protoface import Client

print(session)

from protoface import Client

print(session)

For a kiosk, I would still keep the client thin: the SwiftUI app should not know how to mint API credentials, choose model tiers, or manage avatar inventory. That belongs on the backend or in an operator workflow.

Operational gotchas for kiosks

There are a few non-obvious issues worth planning for before you ship:

• Device restarts: persist enough local state to reconnect gracefully after a power cycle.

• Idle behavior: define what the avatar does when no one is interacting. A silent idle pose is better than a dead video surface.

• Audio routing: kiosks often have fixed speakers. Lock down output selection if the environment allows it.

• Rate and usage limits: if the avatar is customer-facing, enforce session duration and reset rules on the server side, not in the UI.

One practical debugging tip: log session start time, media connection time, first audio packet, and first rendered frame separately. Those four timestamps usually tell you where the bottleneck is. “The avatar is slow” is not a useful metric; “ICE connected in 1.8s, first audio in 2.4s, first frame in 2.6s” is.

Conclusion

To add a talking avatar to a SwiftUI kiosk app, treat the avatar as a realtime media session, not a static view. Keep SwiftUI focused on layout and state, move transport and playback into a dedicated service, and make streaming TTS and WebRTC part of the same timing model so audio and lip sync stay aligned.

If you want to avoid building the avatar control plane yourself, start with the API and docs, then wire the session into your client app. From there, you can refine startup latency, reconnect behavior, and kiosk-specific UX without changing the core architecture. The docs at docs.protoface.com are the right place to check the exact request shapes and session fields before you implement.