Building a Realtime AI Game NPC in Swift for iOS: A Practical Guide
Build a realtime AI NPC in Swift for iOS with streaming voice, WebRTC, and lip-synced avatar integration using Protoface.
Introduction
Building a realtime NPC is less about “making a character talk” and more about synchronizing three systems under tight latency constraints: speech input, language generation, and a visible face that stays aligned with the generated audio. If any one of those drifts, the illusion breaks. On iOS, that usually means you need a clean path from your game engine or app layer into a streaming voice stack, plus a way to render a lip-synced avatar without turning your client into a media server.
This post focuses on the practical shape of that problem in Swift. By the end, you should have a mental model for how to wire a realtime AI NPC into an iOS app, what to keep on-device versus what to offload, and where Protoface fits when you need the NPC to have a synchronized video face rather than a static portrait or text bubble.
Architecture: what actually needs to happen in realtime
An NPC conversation has four phases that overlap:
The player speaks or sends text.
Your app streams that input to an agent.
The agent produces incremental text and audio.
The UI updates a visible avatar in lockstep with the spoken audio.
The key detail is that “realtime” here means you do not wait for a full response before rendering anything. You want partial tokens, partial audio, and immediate avatar motion. If you wait for the whole turn to finish, the NPC feels sluggish even if the underlying model is strong.
On iOS, the cleanest shape is usually:
SwiftUI or UIKit for game UI and NPC state.
WebRTC or a realtime media transport for bidirectional audio/video.
A voice agent that can stream text and synthesized speech incrementally.
An avatar renderer that consumes the agent audio and emits a talking face.
That separation matters because the avatar should be a presentation concern, not a game logic concern. Keep dialog state, game state, and media state distinct. Your game can decide what the NPC says; the agent and avatar stack decide how it is spoken and displayed.
Choosing a transport: WebRTC, websockets, and where they fit
For low-latency interaction, WebRTC is usually the right mental model. It gives you negotiated media tracks, NAT traversal, and jitter handling that you do not want to reinvent. If your agent stack already speaks WebRTC, your iOS client can subscribe to audio and video tracks with predictable latency. If the agent only exposes text over websockets, you can still build a decent NPC, but you will be stitching together speech synthesis, audio playback, and avatar animation yourself.
For a game NPC, the common trade-off is:
WebRTC media path: lower latency, better synchronization, more moving parts.
HTTP/websocket text path: simpler integration, but you own speech output and lip sync.
If your UX depends on the NPC “being present” while talking, use a streaming media path. The face is part of the turn, not decoration.
Swift-side integration pattern
In Swift, resist the urge to treat this as a single giant async function. Model it as a session object that owns the connection, the current NPC, and lifecycle events. That keeps reconnects, interruptions, and scene transitions manageable.
A minimal shape looks like this:
The important bit is that your UI should react to streamed events instead of waiting on a single response object. For example:
That event-driven design maps well to games, where dialog, animation, and state changes are all time-based. You can trigger idle look-at behavior, gesture changes, or quest updates off the same event stream.
Rendering the avatar without coupling it to game logic
If you are already using a 3D character system in your game, you might ask why you need a video face at all. The answer is often production cost. A talking video face can get you believable eye motion, mouth shapes, and expression transfer faster than building a custom facial rig and animation pipeline.
For a realtime NPC, the avatar should be treated as an independent media surface:
It subscribes to the agent’s audio.
It updates mouth movement and timing from the audio signal.
It can swap expression or framing based on turn state.
From the app’s perspective, that usually means embedding or rendering a video element, then overlaying game UI on top. Avoid making the avatar a hard dependency of the game simulation. If the avatar disconnects, the NPC can still function in text or audio-only mode.
One useful practical rule: keep a fallback path. On mobile, network interruptions happen. If the avatar video stalls, preserve the dialog state and continue the conversation through text or audio. Do not tie quest progression to a perfect video stream.
Where Protoface fits: giving the agent a synchronized face
This is where Protoface is useful: it is a developer-facing realtime avatar API that can attach a lip-synced video face to a voice agent without forcing you to build your own avatar rendering stack. For iOS game work, the relevant pattern is usually to keep your game code in Swift, keep the agent logic in your existing voice stack, and let the avatar layer handle synchronized facial output.
If you are using a Python-based agent backend, the integration is especially straightforward. The LiveKit Agents plugin drops an avatar into the agent so the speech and face stay aligned. The exact wiring depends on your room/session setup, but the shape is familiar:
If you are creating sessions directly from a backend service, the REST API is the right control plane. You create or manage avatars and realtime sessions server-side, authenticate with an API key, and keep secrets out of the app bundle. A representative request shape looks like this:
That is intentionally schematic: the specific endpoint and fields are documented in the docs. The important part is the deployment boundary. Your iOS client should not need long-lived API credentials. It should receive only the ephemeral session information it needs to join the realtime conversation.
Gotchas that matter in production
Three issues show up quickly once you move past a demo:
1. Latency budget
The sum matters more than any single component. Network round trip, speech synthesis start time, avatar startup, and UI decode/render all add up. Measure from the player’s input to the first visible NPC reaction, not just time to final answer.
2. Interruptibility
Players interrupt NPCs. Your pipeline should support barge-in behavior: stop current speech, update state, and let the new turn begin. If the avatar continues animating through a cancelled turn, the character feels broken.
3. Session lifecycle
iOS app backgrounding, route changes, and reconnections are normal. Your session object should know how to recover or fail cleanly. Do not assume a single uninterrupted media session for the entire app lifecycle.
There is also a security consideration that often gets ignored in prototype code: never ship API keys to the client if you can avoid it. Keep session creation on the server, hand the app ephemeral session data, and scope permissions narrowly. That is especially important if the NPC can be triggered from gameplay events or user-generated content.
Testing the conversational loop
Before you animate a full game scene, test the loop in isolation. Start with a single debug screen that shows:
connection status
current NPC state
partial transcript
audio activity indicator
avatar frame or stream status
This makes it much easier to answer the only debugging question that matters: where is the delay coming from? If you cannot see the delay, you will eventually misattribute it to the model when the real issue is buffering, rendering, or a stale session.
For teams that want to move fast on the agent side, the quickstart repos linked from the Protoface GitHub org are a good reference point, especially if you are comparing different realtime voice stack choices. See the examples in the GitHub org and the integration notes in the documentation.
Conclusion
A realtime AI NPC is mostly an integration problem: streaming input, incremental generation, synchronized media, and resilient session handling. In Swift, the right pattern is to keep your game logic separate from your media session, treat the avatar as a realtime presentation surface, and design for interruption and reconnects from day one.
If you want to add a talking face to a voice agent without building the avatar pipeline yourself, Protoface is the shortest path. Start with the docs, wire up a minimal session, and test the latency budget in a narrow debug screen before you drop it into a full game loop. From there, you can decide whether the NPC should stay video-first, fall back to audio-only, or blend into your existing character system.
For implementation details and current API shapes, start at docs.protoface.com.