Building a Realtime AI Interview Avatar in Swift for iOS
Build a realtime AI interview avatar in Swift for iOS with low-latency audio, video sync, and LiveKit/Protoface integration.
Introduction
Building a realtime AI interview avatar in iOS is mostly an integration problem: you need low-latency audio in, low-latency audio out, synchronized video, and a UI that behaves well when network quality changes. The hard part is not rendering a face; it is keeping the conversation feeling continuous when your agent is generating speech, the avatar is lip-syncing, and the user is interrupting or asking follow-ups.
In practice, the architecture looks like this: your iOS app captures microphone audio, sends it to a voice agent, receives synthesized speech back, and renders a talking face whose mouth motion is synchronized to that speech. By the end of this post, you should understand the realtime media flow, the iOS pieces you need, and where a service like Protoface fits when you want to attach a face to an existing voice agent.
What “realtime avatar” actually means
A realtime avatar is not a pre-recorded video loop. It is a live media stream driven by the agent’s output audio, usually over WebRTC or another low-latency streaming transport. The avatar service consumes text or audio from the agent, produces video frames that match phoneme timing, and keeps the visual state aligned with the current speaking turn.
For a developer, the important properties are:
Low end-to-end latency so the avatar starts moving quickly after the agent begins speaking.
Audio-video synchronization so lip motion tracks the output audio rather than drifting.
Turn handling so the avatar can start, stop, and resume naturally as the user interrupts or the agent streams partial responses.
Network resilience so a brief jitter spike does not make the face look broken.
On iOS, this usually means you are integrating with a media stack rather than a simple image-view component. You are dealing with microphone permissions, audio session configuration, playback routing, and a rendering surface for video.
iOS media plumbing: the part most people underestimate
If your app already does voice, the avatar adds one more realtime consumer of the same timing constraints. A few implementation details matter more than the rest:
Configure the audio session early
Use an audio session category that supports both recording and playback, and make sure it is active before joining the conversation. If you get this wrong, you will usually see symptoms that look like “the agent is silent,” when the real problem is an audio route or session mismatch.
Typical concerns:
Record and playback simultaneously.
Handle speaker vs. receiver routing explicitly if your UX requires it.
Respect interruptions, route changes, and Bluetooth devices.
For interview-style flows, you almost always want a stable speaker route and predictable echo behavior. If your app records the user while playing the avatar’s voice, test with real devices and a few headset combinations. Simulator testing is not enough.
Keep the agent turn model simple
For conversational UX, the avatar should track the agent’s speaking turn, not arbitrary UI state. That means your client should react to events such as:
agent started speaking
agent stopped speaking
user interrupted
session reconnecting or failed
This is important for interview avatars because users read visual behavior as intent. If the avatar keeps “talking” after audio has stopped, the system feels broken. If it freezes during a network stall, the system feels dead. Let the media session drive the animation state, not the other way around.
Render video as a first-class stream
For a realtime avatar, the video surface should be treated like live media: attach it to a dedicated view, avoid unnecessary re-renders, and avoid buffering assumptions that only make sense for normal video playback. The practical iOS choice is usually a UIView-backed rendering path that can be swapped in and out as sessions connect and disconnect.
Also account for aspect ratio and cropping. Face video generally looks best when the viewport is stable. If you animate layout aggressively, you make perceived jitter worse even when the underlying stream is fine.
Latency budgeting: where the delay actually comes from
Most “lag” complaints are not one problem; they are a stack of small delays. A realistic budget looks like this:
Mic capture — audio frame collection on device.
Uplink transport — network hop to the voice agent.
LLM / agent inference — response generation, often streaming.
TTS or speech synthesis — turning the response into audio if the agent is text-first.
Avatar render — converting speech into synchronized face video.
Downlink transport — getting audio/video back to the device.
When you design the UX, minimize work in the client between receiving media and presenting it. Avoid waiting for full responses before showing motion. Realtime systems feel responsive when they stream partial output early.
Code: a practical iOS integration shape
On iOS, the cleanest approach is to separate your voice agent transport from your avatar rendering layer. In a typical app, your agent connection owns the conversation, and the avatar view subscribes to the media events from that session.
The exact SDK surface depends on the transport you are using, but the structure usually looks like this:
The important bit is not the scaffolding itself; it is the event model. Your UI should respond to media state, not poll it.
Where Protoface fits: attach a face to an existing agent
If you already have an agent and want the avatar layer without building lip-sync and face rendering yourself, the most direct path is the LiveKit integration using the livekit-plugins-protoface plugin. That plugin drops a video face into a LiveKit voice agent so the agent gains a synchronized talking avatar. For code and examples, start with the relevant plugin repo and the docs at docs.protoface.com.
The useful mental model is: your agent keeps handling conversation logic, while the avatar becomes a downstream media consumer that reflects the agent’s speaking state. That keeps the integration contained. You do not need to rewrite your LLM orchestration just to get a face on screen.
If you are building an iOS client that connects to a LiveKit-based backend, this is especially nice because the client side stays focused on media playback and UI, while the server-side agent handles the avatar session.
If you need direct API control
Sometimes you want to create or manage sessions programmatically from your backend instead of wiring everything through an agent plugin. In that case, the REST API is the right tool. You authenticate with an API key and create sessions server-side; do not expose that key in the app.
This is the pattern to use when the session should be controlled by your own backend lifecycle: create it, attach your app’s realtime agent to it, and clean it up when the call ends. Exact fields vary by endpoint, so use the API reference in the docs when you implement it.
Operational details that matter in production
Interview products tend to fail in boring ways. A few practical habits keep them stable:
Handle reconnects explicitly. If the realtime session drops, show a visible reconnect state rather than leaving the avatar half-rendered.
Watch mic permissions. On iOS, permission flow failures often look like networking bugs.
Log media state transitions. Session start, first audio frame, first video frame, interruption, and teardown are the events that help you debug.
Test on poor networks. A system that works on Wi-Fi can degrade fast on mobile data unless you handle jitter and packet loss gracefully.
Also, keep security boundaries clear. If your app talks directly to a REST API, do that only from trusted backend code. On the client, use short-lived session credentials or a transport designed for browser/app exposure. The iOS app should never embed long-lived API keys.
Conclusion
If you are building a realtime AI interview avatar in Swift, the core problem is not “how do I show a face?” It is how to preserve realtime conversational timing across capture, inference, transport, and rendering. On iOS, that means disciplined audio session handling, a clean media-state-driven UI, and a backend that can produce synchronized speech and video without forcing you to invent the avatar layer yourself.
For implementation details, start with the docs at docs.protoface.com. If you are already running a LiveKit voice agent, the plugin route is the most direct path. If you need programmatic control, use the REST API from your backend and keep keys out of the app. Either way, build the client so it reacts to realtime events, not static assumptions, and your interview experience will feel much closer to a live conversation.