Introduction

When you wire an ElevenLabs voice agent to a realtime avatar, the hard part is usually not speech synthesis. It’s the session-init path: allocating the agent, establishing media transport, creating the avatar session, and getting the first audio and video frames flowing without making the user wait. If that path is slow or inconsistent, the whole experience feels broken even if the underlying model is good.

This post walks through a practical way to build a fast session-init flow for a Python-based voice or avatar agent. By the end, you should be able to:

• Initialize an ElevenLabs-driven agent with predictable latency.

• Create and attach a realtime avatar session early enough to avoid awkward dead air.

• Keep WebRTC/media setup separate from conversational logic so startup stays reliable.

• Understand where Protoface fits when you want a synchronized talking face for your agent.

I’ll keep the focus on the mechanics that matter: async startup, connection ordering, warm-up strategy, and the places where people usually introduce unnecessary latency.

What “session init” actually means in a realtime agent

For a voice-plus-avatar agent, session init is the short period before the conversation is live. In practice, it often includes four independent operations:

1. Booting your app process or worker.

2. Creating the conversational agent and loading its configuration.

3. Establishing the realtime transport for audio and, if applicable, video.

4. Creating or attaching the avatar rendering session so the face can track the voice.

Those operations do not have the same latency profile. Model initialization, HTTP auth, WebRTC negotiation, and first TTS synthesis all have their own timing. The main design goal is to make them overlap where possible and defer anything non-essential until after the session is already usable.

The biggest mistake I see is serializing everything: wait for the LLM, then start TTS, then connect media, then create the avatar. That creates a visible stall. Instead, you want to create the media and avatar session early, then let the conversation stack fill in behind it.

Keep the init path narrow and asynchronous

In Python, the right shape is usually: do the minimum synchronous work, then fan out the slow pieces concurrently. That includes loading credentials, preparing the ElevenLabs client, and creating the avatar/session handle.

A simple pattern looks like this:

import asyncio

asyncio.run(start_agent())

import asyncio

asyncio.run(start_agent())

import asyncio

asyncio.run(start_agent())

The exact functions depend on your stack, but the rule is stable: do not block on one service before you even begin the next. For a realtime experience, startup is a latency budget, not a checklist.

Two practical details matter here:

• Use async HTTP clients or SDK calls end-to-end. If one step falls back to a blocking request, it becomes your critical path.

• Separate “session creation” from “session activation” if the API allows it. Creating metadata is often cheap; waiting for media readiness is what costs time.

Pre-create what you can, late-bind what you must

For an agent that uses ElevenLabs and an avatar, you generally want to pre-create everything that is stable across sessions:

• Avatar identity and rendering settings.

• Default voice, language, and speaking style.

• System prompt or domain instructions.

• WebRTC or streaming client configuration.

Then late-bind only what is user-specific or session-specific, such as a conversation ID, customer metadata, or a custom greeting. This keeps the init payload small and avoids round-trips to re-fetch static configuration.

A common pattern is to keep a warm pool of ready-to-use session templates in memory. When a request arrives, you clone the template, attach the current user/session identifiers, and immediately start transport negotiation. If your architecture includes a queue or worker, you can also pre-warm a few workers during low traffic so the first request does not pay cold-start cost.

Be careful not to over-warm. If the upstream TTS or avatar service allocates per-session resources, keeping too many sessions half-open can waste capacity and complicate cleanup. The useful middle ground is to precompute configuration, not to pre-open every transport.

Minimize handshake overhead in the first second

Most startup latency in realtime systems comes from handshake chains: auth, transport negotiation, media capability exchange, and first-frame delivery. You can reduce the visible delay by keeping the first second simple.

Concrete tactics:

• Authenticate once and reuse short-lived session tokens where appropriate.

• Start audio first if your UI tolerates it, then attach the avatar video as soon as it is ready.

• Avoid fetching large assets during init; use cached avatar metadata and static prompts.

• Do not wait for “full readiness” if a usable partial state is already available.

For browser-based clients, this matters even more because WebRTC setup competes with page rendering, JS execution, and network variability. For server-side agents, the problem is usually less browser-related but more about cascading RPCs and queueing delay.

If your agent can speak before the avatar is fully visible, you still need the avatar session created early enough that the first utterance is visually synchronized. Otherwise the user hears speech before the face exists, which feels off even if the total latency is acceptable.

Example: attach a Protoface avatar session early

Protoface is one way to solve the avatar side of this problem without forcing you to build the media plumbing yourself. In practice, you create the avatar/session through the REST API or Python SDK, then connect it to your voice agent as part of startup. The exact request fields vary by flow, but the shape is straightforward: authenticate, create a session, then pass the returned session handle into your agent runtime.

A minimal REST example looks like this:

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

curl -X POST https://api.protoface.com/v1/sessions \
}'

And the Python SDK pattern is typically the same idea:

from protoface import ProtofaceClient

)

from protoface import ProtofaceClient

)

from protoface import ProtofaceClient

)

Once you have the session, attach it to the live conversation and start media flow. The important part is not the exact method names here; it’s the ordering. You want the avatar session ready before your agent’s first response so the face can begin moving with the audio immediately.

If you are building on LiveKit, the same idea applies through the plugin surface: drop the avatar into the voice agent, then let the plugin keep the visual stream synchronized with the agent’s speech. That keeps the integration thin and lets you focus on agent behavior instead of media choreography. The plugin and examples are in the ElevenLabs quickstart repository and the main docs at docs.protoface.com.

Gotchas that slow session start more than people expect

There are a few recurring mistakes that add 200–800 ms without being obvious in code review:

• Blocking secret fetches. Pulling API keys or config from a remote store on the request path is often unnecessary. Cache them in process.

• Creating clients per request. Rebuilding HTTP or WebRTC clients on every session adds connection setup and TLS overhead.

• Doing prompt assembly synchronously. If the system prompt is large or templated, build it before the request enters the hot path.

• Waiting for “all subsystems ready.” Start what can start now. Many systems can emit audio while the avatar video is still joining.

• Ignoring cleanup. Leaked sessions make future startup slower by increasing load and retry noise.

Also remember that “fast” is not just average latency. A session-init flow needs low variance. A 400 ms median with a tight distribution is much better than a 250 ms median with random 2-second outliers, especially for interactive voice. If your startup path depends on multiple upstream services, add timeouts and explicit fallback behavior so one slow dependency does not stall the whole conversation.

Instrumentation is worth the effort. Track timestamps for request received, session created, media connected, first audio sent, and first video frame rendered. You will usually discover that the worst delay is not where you assumed it was.

Conclusion

A fast session-init flow is mostly about discipline: keep the hot path small, start independent work concurrently, avoid redundant setup, and attach the avatar early enough that speech and video begin together. For ElevenLabs voice agents, that means treating the avatar session as a first-class part of startup rather than a cosmetic add-on.

If you want a concrete implementation path, start with the public docs at docs.protoface.com, then adapt the REST API, Python SDK, or LiveKit plugin to your agent architecture. The quickest wins usually come from reordering initialization, not from micro-optimizing the media stack.