Synthetic Load Testing for Realtime AI Avatars: Latency, Sync, and Session Stability
Synthetic load testing for realtime AI avatars: measure latency, A/V sync, jitter, reconnects, and session stability under concurrency.
Introduction
When you add a realtime avatar to a voice agent, the obvious failure mode is not “the face looks bad.” It’s that the system drifts: speech starts before the lip motion, the video lags behind the transcript, sessions get dropped under load, or the avatar feels snappy in one region and broken in another. Those are systems problems, not model problems.
This post is about synthetic load testing for that kind of stack: how to probe latency, audio/video sync, and session stability before users do it for you. By the end, you should have a concrete way to design load tests for realtime AI avatars, choose metrics that actually matter, and spot where the bottleneck lives—browser, WebRTC session, avatar pipeline, or your agent backend.
What you should measure, not just “requests per second”
For a realtime avatar, throughput alone is almost meaningless. A system can pass 100 concurrent sessions and still feel unusable if one of these drifts:
End-to-end latency: time from user audio or agent output to visible avatar response.
A/V sync: alignment between speech audio and mouth motion over time.
Session stability: whether long-lived interactive sessions survive reconnects, network jitter, and sustained load.
Jitter: variance in frame delivery and audio chunk timing, which is often more noticeable than mean latency.
In practice, you want to test the full chain:
your client or browser embed starts a session,
the agent generates or relays speech,
the avatar service receives timing information,
video frames are rendered and delivered in real time.
The failure modes show up at the boundaries. If the agent emits speech chunks irregularly, the avatar can’t lip-sync cleanly. If the media transport is underprovisioned, frames arrive but don’t stay aligned. If session state is fragile, everything looks fine in a 30-second demo and falls apart in a 20-minute run.
Build a synthetic harness that resembles the real traffic shape
Load testing realtime systems is easier when you stop thinking in “users” and start thinking in “session scripts.” A good synthetic harness should model:
Session lifecycle: create, connect, interact, idle, disconnect.
Think time: pauses between turns, not just continuous traffic.
Message size: short acknowledgements vs long agent responses.
Concurrency pattern: ramp-up, steady state, and bursty spikes.
Network variability: latency injection, packet loss, and reconnects.
For avatars, a good synthetic workload is often a conversation simulator that alternates between:
short user prompts,
moderate agent responses,
longer “explanatory” turns that stress continuous lip-sync,
idle periods that keep sessions open.
That matters because realtime avatar systems are frequently optimized for the happy path: one clean turn, one stable connection, one user. Synthetic load should deliberately create the edge cases you expect in production: users interrupting mid-sentence, agents speaking for 30–90 seconds, and browsers that reconnect after brief network loss.
Latency: measure the right segments
Mean end-to-end latency is a poor diagnostic by itself. You want at least three timestamps for each turn:
t0: user speech or agent response begins,
t1: avatar pipeline receives the turn or speech stream,
t2: first visible synchronized mouth motion or first rendered frame.
From there, segment the delay:
generation latency: model or agent response time,
avatar prep latency: processing before the first frame,
transport latency: delivery over WebRTC or streaming transport,
render latency: browser decode and paint.
If you can only collect one metric, collect “time to first synchronized frame.” That’s the point where the user perceives the avatar as alive. A system that returns text fast but delays the face is usually worse than one with slightly slower text and a consistent visual response.
Under load, watch for non-linear behavior. A small increase in concurrency can trigger queueing in one stage and suddenly push the whole pipeline over a threshold. This is common when CPU-bound lip-sync, encoding, or session orchestration shares resources with agent logic.
Sync: treat audio and motion as a coupled system
Audio/video sync is usually not a constant offset problem. In a realtime avatar, the offset can drift during a session because the input stream, buffering strategy, and renderer all have their own timing. What you care about is both absolute sync and stability of sync.
A few practical checks:
Voice onset alignment: the first mouth movement should follow, not precede, the first phoneme.
Turn continuity: long utterances should not “reset” facial motion mid-stream unless the speech stream itself breaks.
Pause behavior: short gaps in speech should not cause the avatar to snap to a fully neutral face and then jump back.
Recovery behavior: after jitter or reconnection, the avatar should return to correct sync without manual intervention.
When you’re load testing, capture both sides of the sync boundary. If the system can export audio timestamps and frame timestamps, compare them directly. If not, record the session and analyze it offline. Even a rough visual inspection often surfaces the right class of bug: delayed onset, sawtooth drift, or frame duplication during buffering.
A useful synthetic pattern is to vary speech rate and turn length. Fast, short utterances expose onset latency. Long monologues expose drift and buffer growth. Interruptions expose state machine bugs in the avatar controller.
Session stability: make failures boring and reproducible
For realtime systems, “stable” means more than “didn’t crash.” It means the session survives predictable stressors without silently degrading. Your load test should include:
Long duration runs: 15, 30, 60 minutes to catch memory growth and leaked media handles.
Reconnects: drop and restore the client network path mid-session.
Concurrency ramps: slow increase to find saturation points instead of only testing at full blast.
Rate-limit pressure: especially if you expose customer-managed embeds with per-IP and duration limits.
Track these stability signals:
session creation failures,
unexpected disconnects,
reconnect success rate,
median and p95 session duration,
memory and CPU growth over time,
ratio of sessions that reach a clean terminal state.
If you operate a browser-based experience, remember that the browser is part of the distributed system. A session that works from localhost may still fail when embedded in a page with strict CSP, slow main-thread work, or competing media elements. Synthetic tests should run from the same environment your users actually use.
A minimal harness in Python or curl
For orchestration, the exact API shape depends on the docs, but the pattern is simple: provision a session, attach the avatar or avatar-enabled agent, and run scripted interactions while collecting timing data.
For a quick smoke test from a CI job or a load generator, curl is often enough to prove session creation and authentication work:
If you’re using a LiveKit-based voice agent, the same principle applies: launch a controlled conversation, feed it scripted turns, and measure from speech generation to rendered avatar output. The integration point is usually where you want to place timing probes, because that’s where agent latency and avatar latency meet.
Where Protoface fits in a real load test
For teams already running voice agents, the most direct integration point is the LiveKit plugin, because it lets you add a synchronized avatar to an existing agent without rewriting the agent stack. That makes it a practical place to measure what matters: how the agent’s speech timing interacts with avatar rendering under concurrency.
If you’re using the Python side of the stack, the SDK and REST API are the right surfaces for synthetic session creation and teardown. Keep the tests close to the control plane: create sessions, start conversations, and assert on lifecycle behavior, not just visual output. The relevant examples and API notes are in the docs, and the plugin repo is a good place to see the integration shape in code: https://github.com/protoface-ai. For implementation details, use the documentation.
One pattern that works well is to split tests into two tiers:
Control-plane tests: session creation, auth, rate limits, teardown, and error handling.
Media-path tests: turn-by-turn latency, lip-sync stability, reconnect behavior, and long-running sessions.
That separation keeps you from blaming the avatar when the bug is really in session orchestration, and vice versa.
Practical gotchas
A few issues show up repeatedly in realtime avatar load tests:
Cold starts: first-session latency may be materially worse than warmed sessions.
Hidden buffering: extra buffering reduces jitter but increases lag; both can look “smooth” in short demos.
Overly synthetic speech: perfectly uniform turn timing hides the bugs that real users trigger.
Browser variability: different codecs, devices, and tab throttling policies can change observed sync.
Metrics without logs: if you can’t correlate a bad session with the exact turn and timestamp, you won’t know which stage regressed.
Also, don’t use only average values. p95 and p99 matter a lot more here because a handful of late frames can ruin the experience even when the median looks fine.
Conclusion
Synthetic load testing for realtime AI avatars is mostly about turning subjective UX into measurable system behavior. Focus on segmenting latency, verifying audio/video sync across long and messy sessions, and exercising reconnects and concurrency ramps before production traffic does it for you.
If you want to apply this to a LiveKit agent, a browser embed, or direct API-driven session orchestration, start small: create one scripted session, record the timing chain, then scale to concurrency and fault injection. The docs are the right place to confirm exact request and session fields, and the quickstarts are useful when you want a working baseline before instrumenting it further. See docs.protoface.com to get started.