How to Preload TTS, STT, and Avatar Assets to Avoid ElevenLabs Agent Cold Starts
Preload TTS, STT, avatar assets, and WebRTC sessions to reduce ElevenLabs agent cold starts and first-turn latency.
Introduction
When a realtime voice agent feels “instant” in development but sluggish in production, the problem is usually not the model. It’s the cold path around it: loading TTS voice assets, initializing STT pipelines, fetching avatar media, and waiting for the first WebRTC/media session to fully settle. Those delays show up as awkward startup silence, delayed first audio, or a face that appears late enough to break the illusion.
This post is about how to remove that startup tax. By the end, you should be able to reason about which parts of your agent are truly on the critical path, preload the right assets before the first user turn, and structure your app so the first interaction is fast even after idle periods or deploys.
What “cold start” means in a realtime agent
In a typical agent stack, “cold start” is not a single event. It is the sum of several independent setup costs:
STT warmup: loading acoustic/language models, establishing a streaming connection, and sometimes buffering a few frames before the recognizer emits stable partials.
TTS warmup: creating a voice instance, selecting a model, and preparing synthesis buffers or cached audio assets.
Avatar warmup: fetching the video face asset, initializing lip-sync/render state, and preparing the realtime stream that will carry frames.
Transport warmup: establishing a WebRTC session, ICE negotiation, and media track attachment.
The key point: these are often independent. You can have a fast LLM response and still feel slow because the first synthesized audio or first avatar frame arrives late. For users, the experience is holistic; a voice agent that talks immediately but shows a blank face still feels broken.
Preloading is the practice of moving as much of that work as possible off the user’s critical path. Sometimes that means keeping a session warm. Sometimes it means creating and caching resources ahead of time. Often it means doing both.
Identify what can be preloaded versus what must be created on demand
The right strategy depends on whether the asset is static, semi-static, or truly per-session:
Static assets: avatar media, a fixed TTS voice, or a reusable STT configuration. These are the best candidates for precreation and cache warming.
Semi-static assets: a voice or avatar chosen per user, but reused across many sessions. Preload them when the user enters your app, not when they click “start”.
Per-session state: live transcripts, conversation memory, and media tracks. These cannot be reused, but the session object and transport can still be created before the first utterance.
A practical rule: if a resource is referenced by an ID and reused across sessions, preload it. If it depends on live conversation state, keep only the session wrapper warm and create the state lazily.
Preload the media pipeline before the first turn
For voice agents, the first real optimization is to start the media pipeline before the user says anything. That means establishing your STT stream, initializing TTS voice selection, and attaching the avatar stream early enough that the first token or first phoneme can flow through without extra setup.
Two implementation patterns work well:
Eager connect: open the realtime session when the user opens the page or enters the call lobby, then keep it idle until speech begins.
Speculative preconnect: if a user is likely to speak, create the media session on hover, page load, or after a short inactivity timer.
Eager connect costs a little more upfront but gives you the most predictable latency. Speculative preconnect is cheaper when many sessions never become active, but you need to accept a small chance of wasted setup.
If you are using a voice-agent framework, the important thing is to separate “agent exists” from “agent is speaking.” The session should be live before the first turn, and the first audio packet should not be the same moment you pay initialization costs.
Cache and reuse TTS, STT, and avatar selections
Do not treat voice choice, language config, and avatar selection as ephemeral request parameters. They are often the most reusable part of the stack.
Concretely:
Cache voice identifiers and model settings per tenant or per user segment.
Reuse STT language and endpointing configuration for the duration of the app session.
Keep avatar IDs stable for a given experience so media can be resolved ahead of time.
This is especially important when your app creates a new conversational session on every visit. If each new session also re-resolves the same voice and avatar metadata, you are reintroducing avoidable latency and extra network calls. Instead, load those identifiers once, then attach them to the new realtime session.
There is a subtle but important distinction here: caching the choice is usually more valuable than caching the live conversation. The conversation should be fresh; the media configuration should not.
Warm the path your users actually hit
If your product has a landing page, agent launcher, or call entry screen, that is where preloading should happen. Don’t wait until the first user utterance. By then, you have already lost the race.
A good flow looks like this:
User opens the app or embed.
The client fetches session bootstrap data and creates the realtime connection.
STT, TTS, and avatar configuration are resolved immediately.
The session idles in a ready state until the user speaks or the system emits the first prompt.
First audio and first face frame are already “in flight” when the conversation starts.
For web apps, this often means pairing a lightweight UI load with a background network handshake. For server-driven systems, it means creating sessions before the user is routed to the agent. In both cases, the goal is the same: eliminate setup from the critical path between “user intent” and “first response.”
Also remember that avatars are visual systems, not just metadata. If your avatar face is delivered as streamed media, you want the media transport attached before the first speaking turn so lip-sync starts immediately instead of after a visible gap.
Example: preloading with a realtime voice agent plugin
If you are building on a voice-agent stack, one practical move is to initialize the avatar plugin as part of agent startup rather than on first speech. With the LiveKit Agents plugin, the idea is to create the avatar-facing component early so the agent is already prepared to attach video when the conversation begins. The exact field names depend on the SDK version, but the pattern is straightforward:
The important part is not the exact constructor signature; it is the timing. Create the avatar integration before the first turn so the media pipeline is warm when speech starts. If you want concrete integration details, the plugin examples in the ElevenLabs Agents quickstart and the docs are the right references.
Example: create the session ahead of time from your backend
When you need finer control, create the session from your backend and hand the client only the minimal bootstrap data. That lets you warm the session server-side while keeping your browser or app code simple.
In practice, this is where you would also hydrate the client-side WebRTC or iframe bootstrap so the transport is ready before the user speaks. The main advantage is that you can shift network and allocation work earlier, then reuse the session object when the interaction starts.
If you prefer Python, the SDK follows the same model: create or fetch the reusable resources first, then start the live session when the user is actually present. See the Python SDK and documentation for the current surface area.
Trade-offs and gotchas
Preloading is not free. A few practical caveats matter:
Resource usage: warm sessions consume capacity. If you keep too many warm at once, you trade latency for cost and concurrency pressure.
Stale config: if your voice, avatar, or STT settings change often, make sure your cache invalidation matches the rollout cadence.
Browser lifetime: tab suspension, mobile backgrounding, and iframe visibility changes can tear down or degrade media paths. A warm session may still need reconnect logic.
First-turn prompt design: if the agent needs to generate a long first response before speaking, no amount of asset preloading will save the perceived delay. Keep the first turn short.
The most common mistake is to optimize the wrong layer. If your avatar is preloaded but your STT starts only after the first audible word, the user still experiences latency. Likewise, if your TTS is warm but you create a new avatar session on every turn, you reintroduce startup work repeatedly.
How Protoface fits in
This is exactly the kind of problem Protoface is built to reduce: you can create reusable avatar/session resources through the REST API or SDK, then attach them to the realtime path before the first spoken turn. That makes it easier to keep the face, voice, and transport aligned with the rest of your agent instead of treating avatar setup as an afterthought.
If you are embedding an avatar in an existing voice-agent stack, the LiveKit plugin is the most direct place to apply these ideas. If you are building your own orchestration layer, use the API or Python SDK to create stable avatar/session identifiers ahead of time, then hydrate the live session only when the user is ready. The relevant references are the docs and the repos linked from the quickstarts.
Conclusion
Cold starts in realtime agents usually come from avoidable setup: model initialization, voice resolution, avatar attachment, and transport negotiation happening too late. The fix is to preload the reusable parts, create the session before the first user turn, and keep the critical path as short as possible.
Start by identifying which resources are stable across sessions, move their creation earlier, and measure first-audio and first-frame latency separately. If you are integrating an avatar into a voice agent, check the docs, wire up a warm session path, and test the experience after a fresh deploy, a cold browser tab, and an idle timeout. Those are the cases that usually expose the problem.
For implementation details and current API shapes, see docs.protoface.com.