Introduction

Low-latency avatar speech sync is mostly a systems problem, not a graphics problem. If your agent speaks through a model, you need the audio, captions, and lip motion to stay aligned across several async components: speech generation, transport, playback, and rendering. The failure modes are familiar: captions that arrive late, mouth motion that leads the audio, audio that stutters when a buffer underruns, or a voice agent that feels “alive” in demos but falls apart under real network conditions.

This post walks through the practical architecture for keeping avatar speech and captions synchronized in a Swift client. By the end, you should have a mental model for where latency is introduced, how to structure your client pipeline, and how to handle the common timing bugs that make these experiences feel off by a few hundred milliseconds.

Start with a timing model, not a UI model

The first mistake is treating the avatar as a single animated view. In practice, you have at least three independent streams:

• Audio — the speech payload the user hears.

• Captions — text chunks or tokens that should be displayed relative to the spoken audio.

• Video/lip motion — the avatar render, usually driven by audio features or phoneme timing.

These streams do not arrive simultaneously. Even if they are generated from the same utterance, they traverse different buffers and can be subject to different transport and decoding delays. So the right abstraction in Swift is a shared utterance timeline with timestamps, not a set of view updates triggered by “new text” or “new frame.”

A useful model is:

// Conceptual timing model
}

// Conceptual timing model
}

// Conceptual timing model
}

For captions, you usually care about media time, not arrival time. If a caption chunk arrives late, you still want to display it at the correct point in the utterance, or at least not before the corresponding audio. The same is true for avatar mouth motion: the render loop should be driven by the current playhead of the audio pipeline, not by wall-clock arrival order.

Build a client pipeline with explicit buffering

In Swift, the simplest robust implementation is to keep three queues:

1. An audio queue feeding your player or WebRTC sink.

2. A caption queue keyed by utterance and media timestamp.

3. A visual state queue for lip-sync or talking-state updates.

You then advance all three from the same notion of playback position. If you are using AVAudioEngine or an equivalent audio playback stack, that position can come from the audio player node’s render timeline, not from a timer you hope stays accurate under load.

For captions, the core algorithm is straightforward:

while let next = captionQueue.peek(),
}

while let next = captionQueue.peek(),
}

while let next = captionQueue.peek(),
}

The leadWindowMs matters. A small positive lead, usually on the order of one frame or less, can make captions feel more responsive without visibly getting ahead of the voice. Too much lead and the user reads the answer before hearing it, which feels wrong. Too little lead and captions appear “late” even when the system is technically correct.

For lip sync, your render loop should be even more conservative. If the avatar frame is driven by a phoneme stream or viseme sequence, you generally want to interpolate state based on the audio playhead and tolerate short network jitter by holding the last known mouth shape for a few frames. What you want to avoid is snapping the face to the newest received state on every callback; that makes network jitter visible.

Use timestamps and sequence numbers to survive real-world jitter

If a realtime system only sends “latest text” and “latest animation state,” you will eventually see out-of-order delivery. That can happen even on a healthy network when you mix text chunks, audio packets, and control messages. The fix is to attach both a sequence number and a media timestamp to every event.

In practice, the client should reject any event that is older than the last committed event for that utterance, unless you explicitly support late correction. That rule applies to captions especially. If token 12 arrives after token 13, you do not want to overwrite the already displayed text unless the protocol says token 12 is a revision.

There are two distinct time domains to keep straight:

• Transport time: when the packet arrived on the device.

• Presentation time: when the packet should affect the user experience.

The presentation clock should be derived from the audio clock whenever possible. If the audio pauses, the caption clock pauses too. If you let captions continue on wall-clock time while audio is stalled, they will drift immediately.

Swift implementation details that matter

On iOS and macOS, keep the UI thread out of the hot path. Network callbacks should enqueue events into a thread-safe buffer, and a single scheduler should resolve what to show next. A good pattern is:

• Network layer parses protocol messages off the main thread.

• State manager stores per-utterance buffers and deduplicates by sequence number.

• Audio subsystem owns the authoritative playhead.

• UI layer subscribes to derived state changes only.

If you are rendering video in a UIView or NSView, do not redraw on every packet. Update at the display refresh rate, and use the latest reconciled state for that frame. That gives you a stable 60 Hz render cadence even if the backend emits state at uneven intervals.

Captions are usually easiest if you model them as attributed text segments rather than a single growing string. That lets you manage partial tokens, replacements, and styling without rebuilding the full text tree on every update.

struct CaptionSegment: Identifiable {
}

struct CaptionSegment: Identifiable {
}

struct CaptionSegment: Identifiable {
}

A practical rule: render partial captions only if your product needs token-level streaming. Otherwise, buffer until you have sentence or clause boundaries. Fewer updates means fewer layout thrashes and less visual jitter.

Where Protoface fits: let the avatar pipeline stay on the server side of the boundary

This is the part where Protoface helps if you are building an agent instead of a custom media stack from scratch. The platform exposes realtime avatar sessions and a LiveKit Agents plugin, so the talking face can stay synchronized with the voice agent without you hand-coding lip motion transport.

For example, if your app already uses a LiveKit voice agent, the plugin approach keeps the avatar alongside the agent’s audio lifecycle rather than in a separate rendering pipeline. That reduces the number of clocks you have to coordinate on the client.

A minimal API interaction to create a session looks like this in principle:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

The exact fields depend on the API version, so use the docs for the schema details. The important architectural point is that the session becomes the unit of synchronization: audio, captions, and avatar state all hang off the same server-managed conversation context. That is much easier to reason about than trying to independently start three client-side streams and hope they line up.

If you prefer programmatic access from Python during testing or orchestration, the SDK follows the same general shape: create or load an avatar, start a session, and consume updates. The repository examples are a good place to see the request/response flow in context: https://github.com/protoface-ai/protoface-sdk-python. For the live agent integration path, the plugin repository is the relevant reference: https://github.com/protoface-ai/protoface-plugin-pipecat.

Common gotchas when syncing speech and captions

There are a few bugs that show up repeatedly:

• Using arrival time as display time: this is the fastest way to create drift.

• Driving captions and lips from different clocks: one clock must be authoritative.

• Over-updating the UI: partial token spam can cause visible layout churn.

• No recovery path after packet loss: you need a way to resync when an utterance is interrupted.

• Letting stale events win: always dedupe by utterance ID and sequence number.

Also watch for “buffer too small” optimism. A tiny buffer lowers latency, but if your network or decode path has any jitter, the avatar will stutter or captions will skip ahead. In most real applications, a small controlled buffer is better than chasing theoretical minimum latency. Users notice smoothness more than raw microbenchmark numbers.

If you do support corrections or transcript revisions, model them explicitly. A revision should update an existing caption segment, not append a new one. Otherwise your transcript will visibly contradict itself during the session.

Conclusion

Low-latency avatar speech sync is mostly about disciplined event ordering: one authoritative playback clock, explicit per-utterance timestamps, deduped sequence handling, and a UI that renders derived state instead of raw network chatter. Once you structure the client that way, captions and talking-head animation become predictable rather than fragile.

If you are adding this to a voice agent or interactive avatar app, start with the docs at https://docs.protoface.com, then use the relevant SDK or LiveKit integration path from the examples repo. Build around the timing model first; the visual polish becomes much easier after that.