Integrating a Realtime Voice + Video Interview Agent into a Swift iOS App
Integrate a realtime voice and video interview agent into a Swift iOS app with low-latency audio, state sync, and avatar session handling.
Introduction
Adding a realtime avatar to an iOS interview app is mostly an integration problem, not a graphics problem. The hard parts are coordinating low-latency audio, keeping the video face synchronized with the model’s speech, and making the UX feel stable when the network jitters or the user interrupts mid-answer.
By the end of this post, you should know how to wire a Swift client into a voice-and-video interview flow, what happens on the wire in a realtime session, and where the avatar layer belongs in the architecture. I’ll also show where Protoface fits when you want the agent to have a synchronized talking face without building the avatar pipeline yourself.
Start with the right mental model
A realtime interview agent is usually two coupled streams:
Audio path: the microphone input goes to your speech pipeline, which may include VAD, ASR, LLM reasoning, and TTS.
Video path: the avatar renderer consumes the assistant’s speech stream or transcript timing and produces lip-synced frames.
The key is that the avatar is not “just video.” It is a visual output that must track the agent’s speaking state, timing, and interruptions. If your iOS app already uses WebRTC or a realtime media SDK, treat the avatar as another participant in the media graph. That keeps your architecture sane when you later add screen sharing, co-hosts, or human handoff.
For an interview app, the practical constraints are:
Low latency: the user expects fast acknowledgments and natural turn-taking.
Interruptibility: if the candidate speaks over the agent, the agent should stop cleanly.
State consistency: the voice agent and avatar must agree on who is speaking.
Mobile resilience: backgrounding, audio session changes, and transient network loss are normal on iOS.
How the iOS client should be structured
In Swift, keep the UI layer thin and push realtime coordination into a dedicated session controller. That controller should own:
the media transport connection,
audio session configuration,
turn state and transcript state,
avatar session lifecycle, and
retry/reconnect behavior.
A clean shape is:
User taps “Start interview.”
Your app asks your backend for an ephemeral session token or session config.
The app joins the realtime media room or connects to your voice pipeline.
The avatar session is created or attached to the same conversation.
Audio, transcript, and avatar video are rendered together in the UI.
Do not create long-lived credentials in the app. If the flow needs an API key, keep that on your server and exchange it for short-lived, client-safe state. In other words: your iPhone app should authenticate a session, not own your master credentials.
Audio session details on iOS that matter
Most broken realtime experiences on iOS come from audio configuration, not model latency. Before you connect, set the app up for bidirectional voice:
.voiceChat gives you the right processing behavior for conversational audio. The exact category and options can vary depending on whether you want speakerphone, Bluetooth headset support, or lower-latency capture. If you are using a media SDK, let it drive the final audio graph where possible, but still configure the session explicitly so you know what the device is doing.
Two gotchas are worth calling out:
Echo cancellation: if your agent’s audio plays through the speaker while the mic is open, you need the system’s voice processing or the media SDK’s equivalent.
Route changes: users will switch to AirPods, unplug headphones, or receive calls. Observe audio route changes and keep the conversation state intact.
Session orchestration: one state machine, not three callbacks
When you add the avatar, transcript, and agent voice separately, it is easy to end up with three competing sources of truth. Avoid that. Model the call as a state machine:
idle → no active interview
connecting → transport and avatar session are being established
listening → user is speaking or the app is waiting
speaking → agent is synthesizing and avatar is animating
reconnecting → temporary network failure
ended → teardown complete
Driving the UI from these states makes it easier to keep the avatar synchronized with the audio. For example, if the ASR layer detects end-of-turn early, don’t immediately animate the avatar speaking until the TTS or assistant output actually starts. The user will notice mismatches immediately.
Similarly, if the candidate interrupts the agent, you want a fast cut-off: stop audio playback, tell the agent pipeline to cancel generation, and signal the avatar to stop its speaking animation. That interruption path should be first-class, not an edge case.
Where the avatar layer fits in the stack
The avatar service should sit alongside your agent pipeline, not inside the Swift app. Your iOS client should be responsible for UI, media capture, and rendering. The agent backend should own the conversation logic, and the avatar backend should own the face animation and lip sync. That separation matters because it lets you swap LLMs, TTS providers, or transport layers without touching the mobile app.
If you are already running a LiveKit-based voice agent, the cleanest way to add a visual face is to use the LiveKit Agents plugin. The plugin approach is attractive because it keeps the voice and video synchronized inside the same agent runtime instead of bolting a separate renderer onto the mobile client. The repository examples are the fastest way to see the wiring in practice: GitHub examples are especially useful if you are mapping this onto an existing agent stack.
A minimal Python-side shape looks like this:
The important part is the integration boundary: your agent emits speech, the avatar plugin turns that into synchronized video output, and the mobile app consumes the result as part of the realtime session. That keeps the Swift code relatively boring, which is what you want.
Example: creating or managing a session from your backend
If your app needs to create a realtime session on demand, do that from your server. The client asks your backend; your backend talks to the avatar API; the response contains only the information the app needs to join or render the session.
The exact request shape depends on the endpoint and session mode you use; treat this as illustrative. The general pattern is the same: create the session server-side, hand the client a safe join payload, and keep the secret key off the device.
If you prefer Python for orchestration, the SDK is a better fit for job setup and lifecycle management:
Use the SDK when your backend already owns the workflow and you want typed, programmatic control. Use raw REST when you need to inspect or debug the wire contract directly.
Testing the mobile integration
For an interview app, test failure modes early. The happy path is not interesting; the edge cases are what users will remember.
Network degradation: verify that the avatar freezes or reconnects gracefully instead of desyncing from audio.
Rapid turn-taking: short candidate responses should not cause the agent to overtalk them.
Background/foreground: iOS may suspend or reconfigure audio when the app backgrounds.
Device variety: test on older devices, Bluetooth headsets, and different bandwidth conditions.
Also instrument the session. Log timestamps for mic capture, assistant response start, avatar start speaking, and video render attach. When something feels “laggy,” those numbers tell you whether the problem is ASR, generation, synthesis, transport, or rendering.
What Protoface changes in practice
This is the part where the integration gets simpler: instead of building and maintaining the avatar rendering pipeline yourself, you can attach a synchronized talking face to the voice agent using the LiveKit plugin or manage sessions through the REST API and Python SDK. That is useful when your actual product problem is the interview flow, not low-level avatar synthesis.
For the mobile app, the benefit is architectural. Your Swift code does not need to know how lip sync is generated; it only needs to join a realtime session, render the returned media, and keep the audio session healthy. The docs at docs.protoface.com cover the exact session fields, authentication flow, and integration details.
If you are evaluating the stack against an existing agent implementation, the quickest path is usually:
Keep your current agent logic.
Add the avatar layer in the backend.
Return a session payload to the iOS app.
Render the video face in the interview UI and verify turn synchronization.
Conclusion
A realtime interview agent on iOS works best when the app is responsible for media and UX, the backend owns conversation control, and the avatar layer is treated as part of the realtime session rather than a separate video feature. That separation keeps latency manageable and makes interruption handling, reconnects, and turn-taking much easier to reason about.
If you are building this for a production app, start by proving the audio session and state machine on device, then attach the avatar layer, then harden your reconnect and interruption paths. From there, the docs and quickstarts are the fastest way to adapt the exact API shapes to your stack. See docs.protoface.com for implementation details and integration references.