Introduction

If you are building an HR screening flow in iOS, the interesting part is usually not the model itself. It is the user experience around it: a candidate opens the app, joins a realtime session, hears a voice agent ask a question, sees a face that is clearly “speaking,” and responds naturally without waiting for a batch-rendered video or a clunky chat UI.

That is the pattern this post covers. By the end, you should understand how to wire a realtime avatar into a LiveKit-based iOS app, what the streaming and synchronization constraints actually are, and where the avatar layer belongs in your architecture. We will keep the implementation grounded in the mechanics of WebRTC, voice-agent turn taking, and session management rather than hand-wavy “AI video” abstractions.

The core architecture: voice agent first, avatar second

For realtime screening, the cleanest mental model is: the voice agent is the primary participant, and the avatar is a visual representation of that participant. The app should not treat the avatar as a separate experience with independent state. If the agent is listening, speaking, or interrupted, the visual layer should follow that exact state machine.

In practice, the pipeline looks like this:

1. The iOS app joins a LiveKit room.

2. A voice agent runs in your backend or agent runtime and participates in the same session.

3. Audio from the user is transcribed or processed by the agent stack.

4. When the agent speaks, the avatar receives the same turn and timing signals, then emits lip-synced video that is streamed back to the client.

5. The app renders that video as another realtime track in the room.

This matters because lip sync is not just “make a talking face.” It is a synchronization problem. The avatar needs the spoken text or generated speech timing, the client needs low-latency playback, and your agent needs deterministic turn handling so that interruptions, barge-in, and network jitter do not create obvious desync.

Why LiveKit is a good fit for this

LiveKit gives you the right primitives for iOS: room membership, audio/video tracks, active speaker state, and a transport that is already designed for realtime media. For an HR screening app, that means you can keep one session model for user audio, agent audio, and avatar video instead of bolting a separate websocket protocol onto the side.

The important implementation detail is that the avatar should arrive as media, not as a custom animation protocol. If it is a video track, your rendering and lifecycle are boring in the best possible way: subscribe, attach, play, dispose. You get standard buffering behavior, device-native rendering, and fewer platform-specific edge cases than trying to hand-roll a canvas or OpenGL animation stack.

On iOS, that usually means you are working with the LiveKit SDK in the app, while the agent runtime and avatar generation run elsewhere. The mobile client stays focused on session join/leave, permissions, and rendering.

Implementing the iOS side: join the room and render the avatar track

From the client perspective, the work is straightforward: authenticate the user, join the room, and subscribe to the avatar video track when it appears. The exact UI plumbing depends on your app architecture, but the flow is the same whether you use UIKit or SwiftUI.

Conceptually, you want a view model that tracks three things:

• room connection state

• remote audio/video tracks

• agent speaking state, if your backend exposes it

Once the avatar track is available, render it in the same place you would render any other remote video participant.

import LiveKit

}

import LiveKit

}

import LiveKit

}

The exact methods for callbacks and attachment vary by LiveKit SDK version, so treat the above as structural code, not a drop-in snippet. The key point is that the avatar is just one more remote participant from the client’s point of view.

If you are using SwiftUI, keep the video view wrapped in a lightweight representable and avoid recreating it on every state update. Realtime media widgets tend to behave poorly when the underlying view gets torn down and rebuilt during normal SwiftUI re-renders.

Turn taking, interruptions, and candidate experience

The hardest part of an HR screener is not video. It is conversation control. If the agent talks over the candidate, or the avatar keeps mouthing words after the user interrupts, the experience feels broken immediately.

You want these behaviors explicitly designed:

• VAD or speech detection on the user side so the agent can pause when the candidate starts talking.

• Consistent turn state so the avatar’s speaking animation follows the agent’s actual audio state.

• Jitter tolerance so the UI does not flicker between speaking and idle on brief network blips.

• Fast session recovery if the app backgrounds or the network changes.

A useful rule: let the backend own “who has the floor,” and let the client render that state rather than infer it from raw audio volume alone. Raw audio levels are noisy; the agent runtime usually knows when a turn really began and ended.

For screening flows, that backend authority also helps with compliance and analytics. You can capture the questions asked, the candidate’s answers, timestamps, and session metadata without trying to reconstruct the conversation from the mobile client after the fact.

Session setup and security: keep secrets off the device

Mobile apps should not contain long-lived API keys for avatar/session management. The app can hold a short-lived LiveKit token or session credential that your backend mints after authenticating the user. The avatar/session control plane should remain server-side.

If you need to create or inspect avatar sessions programmatically, do that from your backend with the REST API. A minimal request looks like this:

curl https://api.protoface.com/v1/sessions \
}'

curl https://api.protoface.com/v1/sessions \
}'

curl https://api.protoface.com/v1/sessions \
}'

The exact fields depend on the endpoint and your configuration, so check the docs for the current schema. The important part is the boundary: your iOS app should consume session credentials, not manage your API key.

For a backend service written in Python, the SDK is the cleaner option when you need to create sessions, fetch usage, or wire avatar state into an existing orchestration layer:

from protoface import Client

)

from protoface import Client

)

from protoface import Client

)

Again, keep the snippet illustrative. The precise method names and payload shape live in the reference docs.

Where Protoface fits without changing your app architecture

This is the part that tends to matter in practice: you do not need to invent a separate video synthesis stack just to get a speaking face into a LiveKit agent. The Protoface integration surface for this case is the LiveKit Agents plugin, published as livekit-plugins-protoface on PyPI, with examples in the plugin repo. It drops an avatar into the agent so the agent’s spoken output is mirrored as synchronized video.

That is useful because the agent remains the source of truth for conversation logic while the plugin handles the avatar/video side of the session. You keep your LiveKit room, your turn-taking rules, and your iOS client architecture intact.

If you are already on the LiveKit agent path, start from the plugin examples in the repository and the integration guidance in the docs: docs.protoface.com and the relevant GitHub repo are enough to get oriented without changing your mobile app design.

Operational details that matter in production

Realtime avatars are easy to demo and easy to make flaky if you ignore a few basics:

• Latency budget: the avatar should track the agent closely enough that users do not perceive “audio says one thing, face says another.”

• Aspect ratio and cropping: decide early whether the avatar is full-screen, picture-in-picture, or embedded in a card. Different layouts expose different artifacts.

• Backgrounding: iOS may suspend or degrade media behavior when the app goes into the background. Decide whether sessions should pause or reconnect.

• Failure handling: if the avatar stream drops, keep the voice session alive and present a clean fallback instead of forcing the user to restart the interview.

• Observability: log session IDs, join/leave events, and avatar state transitions so you can debug desync issues quickly.

For an HR screening product, I would also keep the avatar visually restrained. The goal is not to distract from the conversation; it is to make a remote interview feel coherent and responsive.

Conclusion

The practical way to add a realtime HR screening avatar to an iOS app is to treat it as a media-track problem, not an animation problem. Keep the conversation state in your agent, stream the avatar as synchronized video over LiveKit, and let the mobile client do what mobile clients are good at: session join, track rendering, and resilient UX.

If you want to implement this with less glue code, start with the LiveKit plugin path and then read the API and SDK docs for session lifecycle and backend control. The documentation at docs.protoface.com is the right place to confirm current endpoint shapes, and the quickstart repos are useful when you want a working reference rather than a blank slate.