Introduction

When you put a conversational agent behind a screen in a lobby, kiosk, retail display, or conference booth, the hard part is rarely the speech model. It’s the visual loop: speech has to drive a face with low latency, the face has to stay synchronized with audio, and the UI has to remain responsive on a touch device that may be running for days.

This post walks through the engineering pieces of a realtime conversation avatar in UIKit for touchscreen signage. By the end, you should have a clear approach for streaming a live avatar into an iOS app, presenting it cleanly in a UIViewController, handling touch-driven UX, and understanding where the realtime transport boundaries actually are. I’ll also show where Protoface fits when you want a developer-facing avatar layer instead of building the avatar stack yourself.

What “realtime avatar” means in practice

A realtime avatar is not a pre-rendered animation loop. It’s usually a streaming video surface whose mouth shapes, head motion, and expression are driven by an active conversation session. In a voice-agent setup, the control path is typically:

• user audio is captured locally or from a mic input,

• speech recognition and/or an LLM produces a response,

• text or audio is passed to the avatar service,

• the service returns a synchronized video stream, often alongside audio.

The important constraint is synchronization. If the avatar is speaking from generated audio, the video has to be aligned to that audio. If it’s lip-synced from text, the rendering pipeline still needs a tight coupling between synthesis, animation, and playback so the face doesn’t “lead” or “lag” the voice in visible ways.

On iOS, that means you should treat the avatar as a media stream, not as a static asset. UIKit is just the container: the real work is in transport, buffering, and lifecycle management.

UIKit layout for signage: keep the avatar simple and dominant

Touchscreen signage is not a normal consumer app. You usually want one dominant visual element, minimal chrome, and deterministic behavior after launch. For the avatar view, that means:

• Use a single full-screen video/container view.

• Avoid table views, complex gesture graphs, and animated transitions unless they serve a specific interaction.

• Respect safe areas only if the screen has overlays; otherwise fill the screen.

• Keep a visible fallback state for “connecting,” “reconnecting,” and “muted/no input” conditions.

A pragmatic UIViewController skeleton looks like this:

final class AvatarViewController: UIViewController {

}

final class AvatarViewController: UIViewController {

}

final class AvatarViewController: UIViewController {

}

That container is where your video renderer, WebRTC view, or embedded browser surface lives. The key is to keep the view hierarchy stable. Replacing the avatar layer repeatedly is a common source of black frames, flicker, and media pipeline resets.

Transport and session lifecycle: treat it like media, not like a simple API call

For a touchscreen sign, the session lifecycle matters as much as the conversation itself. You need a deterministic startup path and a clean teardown path:

1. Authenticate the device or app.

2. Create or fetch a realtime session.

3. Attach the avatar stream to your view.

4. Feed user input or conversation events.

5. Monitor disconnects and reconnect without dropping the UI.

If your avatar arrives as a WebRTC stream, expect the usual concerns: ICE negotiation, network churn, and transient bitrate adaptation. On a kiosk or display, unstable Wi-Fi is normal, so your app should distinguish between a fatal session error and a brief transport interruption. Don’t collapse the whole UI because media paused for a few seconds.

There are a few engineering details worth calling out:

• Warm start matters. If the device boots into a lobby screen, preconnect or prepare the session before the user needs it.

• Audio path matters more than video path. If you have only one shared room audio stream, prioritize stable capture and playback over aggressive UI updates.

• Backpressure matters. If user speech or text input arrives faster than the avatar service can process it, queue intentionally instead of spamming the backend.

• Timeouts should be visible. Signage users interpret silence as failure, so show a clear “please wait” or “reconnecting” state.

For iOS specifically, be careful with app lifecycle events. Kiosk-like deployments often disable normal multitasking assumptions, but you still need to handle backgrounding, audio session interruptions, and view reattachment after process pressure.

Touch interaction: make the screen feel responsive without turning it into a general-purpose UI

Touchscreens invite accidental complexity. For an avatar display, interactions should be limited and predictable: tap to start, tap to mute, tap to restart, maybe a hidden admin gesture if you really need it. Anything beyond that should be justified by the actual use case.

Some practical patterns:

• Single tap to engage. Use it to wake the session or restart a stalled conversation.

• Long press for diagnostics. Useful during installation, not for end users.

• Passive mode by default. If the kiosk is unattended, idle animation should be subtle and lightweight.

• Disable accidental navigation. In a signage context, don’t let UIKit gestures leak into system affordances unless intended.

Also consider the human factors around the avatar itself. If it is speaking continuously, a touchscreen should not look “busy” in every pixel. Leave enough negative space for subtitles, status text, or a call-to-action. If you need captions, render them as a separate layer from the avatar so they remain legible under network variation.

Using Protoface for the avatar layer

This is the point where a developer platform is useful: you want a realtime avatar stream without hand-rolling the avatar pipeline. The REST API and Python SDK are good for session creation and management, while the LiveKit plugin is the right fit if your voice agent already runs in that ecosystem.

For example, you can create a session from a backend service with the REST API using an API key. The exact request body depends on the avatar/session fields in the docs, but the shape is straightforward:

curl https://api.protoface.com/v1/sessions \
}'

curl https://api.protoface.com/v1/sessions \
}'

curl https://api.protoface.com/v1/sessions \
}'

If your app is Python-based, the SDK keeps that workflow programmatic and testable:

from protoface import Client

)

from protoface import Client

)

from protoface import Client

)

For teams already using LiveKit Agents, the plugin path is often the cleanest: drop the avatar into the agent so the voice and face stay synchronized inside the same conversation loop. The integration details and examples live in the GitHub repo and docs, which is the right place to verify current setup and exact package names.

Two practical notes for signage deployments:

• Keep API keys off the device if you can. Use your backend to mint or broker sessions.

• Make session creation idempotent or at least restart-safe, because displays do get power-cycled and networks do drop.

If you’re specifically interested in the LiveKit path, the relevant plugin and examples are in the repo linked from the quickstarts and docs. For the broader API surface, see the documentation before wiring anything into production.

Failure modes you should design for

Realtime avatar projects tend to fail in the same few places:

• Audio drift. The avatar appears to talk out of sync because the media pipeline buffered inconsistently.

• Reconnect loops. The UI reconnects aggressively and becomes unusable on poor Wi-Fi.

• Stale sessions. The app holds onto an expired or invalid session token.

• Hidden latency. The avatar is technically live, but speech-to-response latency is long enough that the screen feels broken.

In UIKit, the fix is mostly discipline: explicit state machine, explicit loading states, and clear ownership of the underlying stream object. Don’t let networking callbacks mutate view state ad hoc. Centralize session state, then have the view controller render that state.

For signage, I’d also recommend logging the following locally or to your observability stack:

• session start time,

• time to first frame,

• time to first audio,

• reconnect count,

• reason for teardown.

Those metrics will tell you whether the problem is transport, backend latency, or the device itself.

Conclusion

Implementing a realtime conversation avatar in UIKit is mostly an exercise in media engineering and state management. Keep the UI simple, treat the avatar as a live stream, handle reconnects intentionally, and make sure the session lifecycle is visible in the app. If you get those basics right, the touchscreen stops feeling like a fragile demo and starts behaving like a dependable kiosk surface.

When you’re ready to wire this into a voice agent or a managed session flow, start with the docs and a quickstart from the project examples. The most useful next step is to build the smallest possible end-to-end path: one screen, one session, one avatar, one reconnect path. After that, you can harden it for production.

See docs.protoface.com for the current API and integration details.