Introduction

Flaky realtime avatar tests are usually not “random”; they are tests with hidden timing assumptions. In Rust CI, that tends to show up as one of three failure modes: a session never reaches “ready” before the timeout, frames arrive in a different order than the test expected, or the test passes locally but fails under CI load because the event loop, network, or media pipeline is slower than your laptop.

This post is about making those tests deterministic enough to trust. By the end, you should be able to identify where the nondeterminism comes from, write assertions that tolerate legitimate realtime variance, and set up CI so failures indicate real regressions instead of scheduler noise. I’ll also show where a realtime avatar surface like Protoface fits into that workflow when you need an actual avatar session in the test loop.

Why realtime avatar tests get flaky

Realtime avatar systems combine several independent clocks and transport layers: your test process, the agent runtime, signaling, media delivery, and often an upstream model or TTS service. In a voice-agent-with-avatar flow, the test usually waits for some combination of:

• session creation over HTTP,

• WebRTC signaling and ICE negotiation,

• track publication/subscription,

• first audio packet,

• first video frame,

• lip-sync or speaking state transitions.

Any one of these can lag without being incorrect. The mistake is to encode “first frame must arrive within 2 seconds” as if it were a contract. In CI, CPU contention and shared network variance make that assumption brittle.

The other common source of flakiness is asserting on exact ordering in an event-driven system. If your test says “video started before audio” or “the ready callback must fire before session update X,” you are usually checking an implementation detail, not the user-visible behavior. For realtime systems, prefer state-based assertions over sequence-based assertions unless the sequence itself is the thing under test.

Build tests around observable states, not incidental timing

The first shift is conceptual: test the boundaries of the system, not every internal transition. In practice, that means a good avatar test should answer questions like:

• Did the session get created successfully?

• Did the avatar connect and become available for media?

• Did the agent produce audio and an accompanying video track?

• Did the session remain healthy for a short window?

That is much more robust than checking for exact timestamps or exact frame counts.

For Rust integration tests, structure the test around a bounded wait for a state change, then validate the state you care about. If you’re polling an HTTP API, use explicit retry with deadline and backoff rather than a single request. If you’re listening to an async event stream, collect events until the condition is satisfied or the deadline expires.

use std::time::{Duration, Instant};

}

use std::time::{Duration, Instant};

}

use std::time::{Duration, Instant};

}

That pattern does two useful things: it absorbs short-lived variance, and it keeps your test from hammering the API in a tight loop. Use a total deadline that reflects reality, not optimism. For realtime media tests, “a few seconds” is often too aggressive in CI even if it feels fine on a developer machine.

Stabilize the network and the event loop

Most “flaky” failures are really environment failures. Before blaming the app, check whether your test runner is introducing the noise.

Three practical controls help a lot:

1. Isolate test resources. Avoid parallel tests that compete for the same avatar session, port range, or media device emulator. If two tests can observe the same websocket or room state, they can corrupt each other’s assumptions.

2. Use deterministic fixtures. Keep the prompt, voice, and test input fixed. Do not mix the avatar test with model-evaluation randomness if you’re trying to debug transport behavior.

3. Budget for latency variance. WebRTC setup and media startup are not instant. CI runners can be noisy; laptop results are not a baseline.

In Rust specifically, async tests sometimes fail because the runtime is under-provisioned for the work being done. If the test opens a websocket, waits on HTTP polling, and handles media callbacks in the same process, make sure you’re using a Tokio runtime that can actually schedule those tasks concurrently. Starvation looks like a timeout, but the root cause may just be insufficient executor progress.

Another gotcha is cleanup. A test that aborts before it tears down its session can leave behind server-side state that makes the next test behave differently. If the system supports explicit delete/close semantics, call them in a Drop-style cleanup path or a finally-equivalent. In integration suites, I like to record the session ID in logs so a failing CI run can be correlated with backend traces later.

Make failures diagnosable with the right assertions and logs

A test that says “expected false, got true” is not useful when the real bug is “ICE disconnected after track subscription.” When a realtime test fails, you want the log to answer three questions immediately:

• What session was under test?

• Which milestone failed to arrive by the deadline?

• What last-known state was observed before the timeout?

That means logging state transitions as structured events, not just printing text at the end. For example, log session creation time, connection time, first audio time, first video time, and any explicit health/status fields you observe. Even if the test passes, those timings are useful for spotting regressions over time.

tracing::info!(session_id = %session_id, "session created");
tracing::info!(session_id = %session_id, "first media observed");

tracing::info!(session_id = %session_id, "session created");
tracing::info!(session_id = %session_id, "first media observed");

tracing::info!(session_id = %session_id, "session created");
tracing::info!(session_id = %session_id, "first media observed");

When you assert, assert on the semantic contract. Examples:

• session became ready within N seconds,

• at least one audio frame was received,

• at least one video frame was received after the connection became active,

• session stayed connected for a minimum observation window.

Avoid these patterns unless you really need them:

• exact frame counts,

• specific millisecond timestamps,

• ordering between independent signals,

• tests that assume the first callback is always the same callback.

If you must check ordering, constrain it narrowly. For example, “the avatar must not emit media before the session is connected” is a meaningful protocol assertion. “audio callback A must happen before video callback B” usually is not.

How Protoface fits when you need a real avatar session in the loop

When the thing you’re testing includes a real avatar lifecycle, use the REST API or SDK to create a session and then wait on explicit readiness, instead of trying to infer readiness from downstream media alone. That gives your Rust CI a clean separation between control-plane setup and media-plane validation. The API is documented at docs.protoface.com, and the Python SDK is useful when you want a small script to reproduce a failing case outside the test runner.

import os

print(resp.json())

import os

print(resp.json())

import os

print(resp.json())

For Rust CI, the useful pattern is: create a session, poll until the server reports the session is ready, then start the media assertions. If the session creation itself is flaky, that points to auth, configuration, or backend availability. If creation is stable but media observation is flaky, the bug is likely in transport timing, subscription handling, or your test assumptions.

If you’re integrating via a LiveKit voice agent, the same principle applies: validate the agent’s own state transitions, then separately validate that the Protoface plugin produced a synchronized talking face. The plugin itself is meant to be a drop-in integration point, so the test should focus on whether the avatar appears in the agent path and remains synchronized, not on implementation details inside the plugin. If you use that route, the GitHub repo is the right place to inspect examples and expected usage patterns: https://github.com/protoface-ai.

Practical CI tactics that reduce noise

A few boring tactics go a long way:

• Run flaky tests serially until they stabilize. Parallelism is great for throughput and terrible for diagnosing shared-state bugs.

• Record artifacts such as logs, session IDs, and timing summaries on failure.

• Use retries only around external setup, not around the assertion itself. Retrying the assertion can hide a real regression.

• Separate smoke tests from deep media tests. A short session-health check can run on every PR; a full synchronized media validation can run less frequently.

If the same test fails intermittently, categorize the failure. A timeout waiting for a ready state is different from a connection drop after readiness. Don’t treat them as equivalent just because both are “red.” The remediation is often different, and the logs should reflect that.

Conclusion

Flaky realtime avatar tests are almost always a test design problem first and a product problem second. The fix is to model reality honestly: asynchronous setup, variable media startup, and event streams that can legally arrive in different orders. Use deadlines and bounded retries, assert on stable states rather than incidental timing, and capture enough structured logging to understand exactly what failed.

When you need to exercise a real avatar session in CI, use the control-plane surface that matches the job—REST API, SDK, or plugin integration—and keep the test focused on one layer at a time. If you want implementation details or a reference flow, start with the docs at docs.protoface.com.