Introduction

If you want an iPhone app to feel like a real interview, a static form is usually the wrong interface. For an HR screening flow, the useful pattern is a voice-first conversation with a visible face: the user speaks naturally, the app responds in real time, and the avatar lip-syncs to the agent’s output instead of just playing a canned animation.

This post shows the architecture for adding a talking screening avatar to an iPhone app built with SwiftUI. By the end, you should know how to wire up the UI, stream audio/video with low latency, keep the avatar and agent synchronized, and choose the right integration path for a mobile app.

What you are actually building

At a technical level, this is a real-time media problem, not a “video playback” problem. The app needs to:

• capture microphone audio from the candidate,

• send that audio to a voice agent with low latency,

• receive synthesized audio back from the agent,

• display a talking face that stays in sync with the audio, and

• manage session state cleanly across connection, reconnect, and teardown.

The avatar itself is usually a streamed video track or a WebRTC-delivered media surface, not a local animation. That distinction matters because lip sync and conversational timing are tied to the agent’s output stream. If you treat the face as an independent animation layer, you will eventually get drift, awkward pauses, or mismatched mouth movement.

SwiftUI app structure: keep media and UI separate

In SwiftUI, it helps to treat the interview screen as two layers:

1. a lightweight view model that owns connection state and session lifecycle, and

2. a media rendering view that displays the remote avatar stream.

This keeps the UI reactive without letting SwiftUI redraws interfere with the transport layer.

import SwiftUI

}

import SwiftUI

}

import SwiftUI

}

Your SwiftUI screen can then render the avatar surface conditionally based on connection state:

struct ScreeningView: View {

}

struct ScreeningView: View {

}

struct ScreeningView: View {

}

The important part is not the exact view code. It is the ownership boundary: SwiftUI drives presentation, while the media session owns audio/video transport and timing.

Connecting the candidate’s audio to a live interview agent

For a screening flow, the cleanest model is full duplex: the app streams microphone audio up, the agent processes it, and the agent streams synthesized audio back down. The avatar should be driven by that same response stream so the face and voice remain synchronized.

On iOS, you will typically rely on an RTC stack underneath, because realtime voice agents need jitter buffering, clock sync, and reconnection behavior that ad hoc HTTP polling cannot provide. In practice, this means:

• request microphone permission up front,

• configure an audio session appropriate for voice chat,

• start the session only after the audio pipeline is ready, and

• cleanly stop tracks when the user hangs up or the app backgrounds.

A minimal audio-session setup often looks like this:

import AVFoundation

}

import AVFoundation

}

import AVFoundation

}

That is only one piece of the puzzle. The avatar session still needs to be attached to a media transport that can deliver the remote video track. In a production app, you do not want to invent your own A/V sync logic unless you have to. Use the transport and session semantics provided by the avatar platform, then bind those tracks into your rendering layer.

Rendering the avatar in SwiftUI without fighting the framework

SwiftUI is good at layout, state, and transitions. It is not the place to implement video decoding. For a talking avatar, the usual pattern is to host a UIKit-backed video view inside SwiftUI. That gives you a stable view hierarchy while the underlying renderer handles frames and timing.

import SwiftUI

}

import SwiftUI

}

import SwiftUI

}

A few practical notes:

• Keep the renderer alive across view updates so you do not drop frames or reconnect unnecessarily.

• Do not store session handles in the view body; put them in the view model or a dedicated media controller.

• Expect reconnects. Mobile networks are not stable, and interview sessions are long enough that you will see backgrounding, Wi-Fi switches, and transient packet loss.

Session flow for an HR screening experience

For this use case, the session lifecycle usually has four phases:

1. Provision — create or fetch an avatar/session configuration for the screening bot.

2. Connect — establish the realtime media session and attach the avatar track.

3. Interact — stream microphone audio, receive responses, and keep UI state in sync.

4. Terminate — stop tracks, release the renderer, and record session metadata if needed.

There are a few implementation details worth getting right:

• Latency budget: every extra network hop is visible in conversation timing.

• Backpressure: if the user talks over the agent, your app must handle interruption cleanly.

• State recovery: if the app suspends, resume deterministically or force a fresh session.

• Privacy: treat the candidate’s microphone audio as sensitive data and scope access tightly.

Where Protoface fits

This is the part that can save you from building the avatar pipeline yourself. Protoface gives you a developer-facing realtime avatar surface that plugs into this kind of voice-agent flow. For an iPhone app, the most practical path is usually to create and manage the avatar session on your backend, then hand the app a short-lived session payload or connection details. That keeps API keys off the device and lets the mobile client focus on media and UI.

Protoface exposes a REST API at api.protoface.com for avatar and session management, plus a Python SDK if you want to create sessions from backend code. The exact request shape depends on the avatar/session model you choose, but the pattern is straightforward: authenticate with your API key on the server, create a session, and pass the client only what it needs to join the realtime media stream.

import requests

session = resp.json()

import requests

session = resp.json()

import requests

session = resp.json()

From there, your iPhone app connects to the session and renders the remote avatar stream. The important architectural win is that the app never needs direct access to your long-lived API key, and you do not have to build the avatar generation logic into the client.

Common pitfalls

Three failure modes show up repeatedly in mobile avatar apps:

• Starting the session before audio is ready — this causes missed first words or awkward initial silence.

• Recreating the renderer on every state change — this drops the video surface and makes the avatar flicker.

• Mixing UI state with transport state — this makes reconnect logic brittle and hard to test.

If you keep those concerns separated, the app stays much easier to reason about. The interview UX can be simple: a start button, a live face, a transcript or status line, and an end call action. Everything else should be hidden behind the media session controller.

Conclusion

The practical way to add a talking HR screening avatar to an iPhone app is to treat it as a realtime media integration: SwiftUI for layout and state, a stable video renderer for the avatar surface, and a backend-managed session for security and lifecycle control. Once you respect those boundaries, the implementation is straightforward and maintainable.

If you want to build this with a managed avatar layer instead of assembling the full pipeline yourself, start with the docs at docs.protoface.com. Then wire up a small end-to-end prototype: provision a session on the backend, connect the iPhone client, and confirm that microphone audio, agent speech, and avatar lip sync all move together.