Introduction

If you want to help candidates practice interviews on an iPhone or iPad, the hard part is not the chat loop. It’s making the experience feel present: low-latency audio, a synchronized face, predictable device permissions, and UI that stays responsive while speech is streaming in both directions.

This post walks through a practical way to build an on-device interview practice avatar in SwiftUI. By the end, you should know how to structure the client, where realtime media fits, how to keep the UI and session state separated, and how to connect a mobile app to a live avatar service without turning your app into a pile of ad hoc callbacks.

Understand the shape of the problem

An interview practice avatar is not just “chat with voice.” It is a realtime conversational system with three parallel concerns:

• Transport: audio and video need to move with low latency, usually over a realtime media layer such as WebRTC.

• Turn-taking: the assistant has to know when the user is speaking, when to listen, and when to render a response.

• Presentation: the app needs to show the avatar video, transcripts, state, and controls without blocking playback.

On-device in SwiftUI is a good fit if you want a native mobile interface, local access to the microphone, and predictable lifecycle handling. The key design decision is to keep the avatar session as a separate model layer, then bind SwiftUI to that model. Do not let your views own network state directly.

Design the client architecture first

A simple, maintainable shape is:

1. A session controller that owns the avatar connection, token exchange, and lifecycle.

2. A media layer that handles microphone capture, playback, and the remote avatar stream.

3. A view model exposing small pieces of state for SwiftUI: connection status, current prompt, transcript, and error state.

4. A SwiftUI view hierarchy that only renders state and forwards user actions.

In practice, this makes it easier to handle app backgrounding, reconnects, and cancelled interview sessions. It also keeps the code testable: you can mock the controller in previews and unit tests, and separately test your realtime integration.

A useful rule: if a method touches network, auth, or streaming primitives, it does not belong in a SwiftUI View. Put it behind an ObservableObject or an actor-style service, then publish only the state the UI needs.

Model the interview session as state, not as UI events

For interview practice, the state machine matters more than the animation. A minimal model might look like:

• idle: no active session

• connecting: acquiring credentials and establishing the realtime session

• listening: microphone open, waiting for user speech

• thinking: user finished, assistant is generating a response

• speaking: avatar audio/video is playing

• error: failed session or transport issue

The app should transition between these states based on realtime events, not timers. For example, when the backend detects end-of-utterance or the agent starts speaking, your client should move from listening to thinking or speaking accordingly. That matters because mobile network jitter can make naive timer-driven UI feel wrong very quickly.

Also pay attention to interruption semantics. In an interview practice flow, the candidate should be able to cut off the avatar, ask a clarifying question, or retry an answer. That means your client needs a clean “barge-in” path: stop local capture or signal the agent to yield, then update the UI immediately.

Build the SwiftUI shell around a realtime session object

Here is a stripped-down example of the shape I mean. This is intentionally illustrative; the exact session fields and methods depend on the API you wire up.

import SwiftUI

}

import SwiftUI

}

import SwiftUI

}

Your view can then stay simple:

struct InterviewView: View {

}

struct InterviewView: View {

}

struct InterviewView: View {

}

The missing piece is the video surface. In a native mobile app, that is usually either a UIViewRepresentable or a media SDK view that renders the remote track. The important part is to keep the rendering view dumb: it should render the stream the controller hands it, not decide when a session starts or stops.

Handle audio and permissions like a real mobile app

Most demo implementations underestimate the microphone path. A production-ish interview practice app should:

• Request microphone permission before starting the session.

• Configure the audio session for duplex voice interaction.

• Handle route changes such as Bluetooth headset connect/disconnect.

• Survive background/foreground transitions without leaving the app in a half-open state.

On iOS, that usually means treating audio session configuration as part of startup, not as an afterthought inside the first “Start” button tap. If you need a clean user experience, also consider showing a short explanation before the permission prompt. Interview practice is one of those flows where users are more tolerant of a one-time explanation than of a confusing system dialog.

For WebRTC-style realtime sessions, latency is usually dominated by network, encoding, and turn detection, not SwiftUI rendering. So once your views are reasonably efficient, spend your time on audio-session correctness and state transitions instead of micro-optimizing list updates.

Use Protoface for the avatar session layer

This is where Protoface fits cleanly: you let the API manage the realtime avatar session while your SwiftUI app focuses on native UX. For a mobile app, the practical pattern is usually:

1. Create or configure the avatar session on the backend.

2. Fetch the session details from your app using your server, not by embedding secrets in the client.

3. Connect the app to the realtime media session and render the avatar video track.

If you are programmatically creating sessions, the REST API is the lowest-level surface. A typical request looks like this:

curl -X POST https://api.protoface.com/<exact-session-path> \
}'

curl -X POST https://api.protoface.com/<exact-session-path> \
}'

curl -X POST https://api.protoface.com/<exact-session-path> \
}'

The exact path and fields are documented in the API docs, but the architectural point is stable: create sessions server-side, keep API keys out of the app bundle, and hand the client only what it needs for a specific realtime connection. If you prefer to manage avatars and sessions from Python, the SDK follows the same pattern and is usually the quickest way to automate interview templates, personas, or per-user session creation; see the docs and the Python SDK repo.

If your app already uses a voice-agent backend, you can also integrate through the LiveKit plugin surface so the agent gains a synchronized talking face without rewriting your agent stack. That is useful when the conversational logic already lives in a LiveKit-based service and the mobile app just needs to present the avatar and session UI. The repo with examples is here: GitHub.

Practical SwiftUI gotchas

A few things tend to bite teams building this the first time:

• Avoid view-owned async tasks. If a task starts in a view, make sure it is cancelled when the session stops or the view disappears.

• Debounce transcript updates. High-frequency partial transcripts can cause unnecessary list churn. Batch them or publish only meaningful changes.

• Keep session recovery explicit. If the connection drops, decide whether to reconnect automatically or surface a retry button. Do not hide the failure behind a spinner.

• Test on real devices. Simulator audio behavior is not a reliable proxy for actual Bluetooth routing and network jitter.

Also, if you support a “practice interview” loop with multiple questions, make the question state part of the backend session rather than local UI state alone. That keeps the avatar, transcript, and scoring logic aligned if the app reconnects.

Where to put the product logic

A strong pattern for this kind of app is to let the backend define the interview session and the app define the interaction shell. The backend can own:

• question sequence and rubric

• avatar voice and persona instructions

• session duration and policy

• usage tracking and per-session constraints

The iOS app can own:

• permission prompts and device setup

• session start/stop controls

• transcript rendering and local navigation

• reconnect UX and error handling

That split keeps your SwiftUI code small and your realtime logic easier to reason about. It also makes it straightforward to reuse the same interview session backend across iPhone, web, or a desktop client.

Conclusion

An on-device interview practice avatar is mostly an exercise in good client architecture: isolate realtime session state, treat audio/video as a transport problem, and keep SwiftUI focused on rendering and control flow. Once you do that, the avatar becomes just another streamed surface in the app, rather than a special case that leaks implementation details everywhere.

If you are ready to wire this up, start with the docs at docs.protoface.com, then choose the integration surface that matches your stack: REST API for server-managed sessions, Python SDK for automation, or the LiveKit plugin if your agent already lives there. For SwiftUI specifically, keep the UI thin, the session model explicit, and the media path native.