Introduction

If you’re building a realtime avatar experience, sooner or later you run into the same problem: the browser needs to stream audio and video, but your TTS provider expects a secret API key. If you put that key in frontend code, it will get extracted. If you proxy incorrectly, you can introduce latency, leak user data, or create a brittle session model that’s hard to debug.

This post walks through a practical pattern for proxying TTS requests for realtime avatars without exposing API keys. By the end, you should be able to design a backend relay that:

• keeps provider credentials server-side

• streams synthesized audio with low enough latency for lip-synced avatars

• handles per-session auth, rate limits, and cleanup

• fits into a voice-agent architecture instead of fighting it

The core architecture: browser to your backend, backend to TTS, audio back to the avatar pipeline

The safe pattern is simple: the browser never talks directly to the TTS provider. Instead, it sends a user message, a session token, or a short-lived request to your backend. Your backend validates the request, calls TTS with its own secret key, and streams the synthesized audio into your avatar or voice-agent pipeline.

For realtime avatars, the important detail is that audio is not just an output artifact. It is the source of timing for lip sync, facial pose, and often turn-taking behavior. That means your relay should preserve streaming semantics whenever possible. A request that waits for a full MP3 before forwarding will usually feel noticeably worse than one that emits audio chunks as they arrive.

A typical request path looks like this:

1. User types or speaks a prompt in the browser.

2. Your app sends the text to your backend over HTTPS.

3. Your backend authenticates the user and checks any per-session policy.

4. Your backend calls the TTS provider with a server-side secret.

5. Audio is streamed into the avatar pipeline, which drives lip sync and playback.

Why not call TTS directly from the browser?

Because the browser is not a trusted runtime. Anything you ship to the client can be copied, inspected, or reused. That includes API keys, signed URLs with long TTLs, and backend endpoints that are effectively public because they’re authenticated only by a static token embedded in JavaScript.

Even if your TTS provider supports browser-friendly tokens, direct calls tend to create operational problems:

• Credential leakage: a secret in frontend code is not secret.

• Quota abuse: one leaked key can burn through usage quickly.

• No server-side policy: it’s harder to enforce per-user quotas, content checks, or allowed voices.

• Session coupling: realtime avatars usually need correlated session state, not just a stateless text-to-audio call.

For avatar systems, the backend relay is not just a security boundary. It is also the place where you can normalize latency, attach metadata, and manage session lifecycle.

A practical relay design

The backend can be very small, but it should still make a few decisions explicitly:

• Authenticate the caller with your own session cookie, JWT, or signed request.

• Authorize the action so only allowed users can create synth requests for a given avatar or conversation.

• Forward only what is needed to the TTS provider: text, voice, and request parameters.

• Stream audio back instead of buffering everything in memory when the provider supports it.

• Attach session IDs so logs, usage, and avatar state are easy to correlate.

Here’s a minimal Flask-style sketch. The exact TTS provider parameters will vary, but the structure is what matters:

from flask import Flask, request, Response

return Response(generate(), mimetype="audio/pcm")

from flask import Flask, request, Response

return Response(generate(), mimetype="audio/pcm")

from flask import Flask, request, Response

return Response(generate(), mimetype="audio/pcm")

If your avatar pipeline expects WebRTC audio, you’ll usually convert or ingest the stream into the media layer rather than returning raw PCM to the browser. The core idea stays the same: keep the key on the server, and keep audio moving.

Latency, streaming, and lip sync are tightly coupled

For realtime avatars, the dominant failure mode is not correctness; it’s delay. A text-to-audio round trip that takes two seconds can still be functionally correct, but the avatar will feel disconnected from the conversation.

When you proxy TTS, the main latency costs are:

• network RTT between your backend and the TTS provider

• provider synthesis time for the first audio chunk

• buffering in your relay if you accidentally wait for the full response

• media pipeline handoff into WebRTC or your avatar runtime

Two implementation choices help a lot:

1. Stream early: forward the first bytes as soon as they arrive from the provider.

2. Keep formats simple: use a format your downstream pipeline can ingest without expensive transcoding.

There is also a trade-off between correctness and interactivity when users interrupt themselves or change their mind mid-utterance. In a conversational UI, you may need to cancel the in-flight TTS request, stop the audio sink, and reset the avatar mouth state. Design your relay so cancellation is possible and idempotent.

Session state and rate limits belong on the server

Once you proxy TTS, you have a natural place to enforce policy. That matters more than it first appears, because avatar applications are usually multi-tenant and often public-facing.

Useful server-side checks include:

• per-user and per-IP request limits

• voice allowlists

• maximum text length per synthesis request

• session expiry and cleanup

• per-avatar or per-embed usage caps

These controls are difficult to enforce safely if the browser talks directly to TTS. With a backend relay, you can also record every request with the session identifier that maps back to your avatar runtime or conversation agent. That makes debugging much easier when a lip-sync issue turns out to be an audio chunking issue, or when a rate-limit spike comes from a single client retrying aggressively.

How this looks in a voice-agent stack

If you’re using a voice agent framework, the cleanest setup is often to let the agent own the conversation flow while your backend owns secrets and media policy. For example, in a LiveKit-based agent, the agent can generate text, your backend can synthesize audio, and the avatar layer can render a synchronized face on top of the voice session.

The important part is that the agent should not need the TTS key. It should call your internal service or plugin boundary, and your server should mediate access to the provider. That keeps the secret out of process memory in client code, reduces blast radius, and makes credential rotation tractable.

from livekit.plugins import protoface

)

from livekit.plugins import protoface

)

from livekit.plugins import protoface

)

If you’re using the LiveKit plugin, the implementation details are documented in the integration guide and examples. The general principle is the same: the plugin becomes the bridge between your agent and the realtime avatar surface, while your backend remains the only place that ever sees private credentials. If you want a reference implementation, the plugin repository is a good place to start: https://github.com/protoface-ai/protoface-plugin-pipecat.

A note on Protoface and when you may not need your own proxy

There are cases where you do not want to build this plumbing yourself. If your requirement is simply “put an interactive avatar on a website without exposing an API key,” customer-managed iframe embeds are the right abstraction: the browser loads an iframe, the parent-origin allowlist controls who can embed it, and per-embed voice, instructions, and rate limits stay on the server side. No backend is required in the customer app, and no key is ever shipped to the browser.

For more custom voice-agent integrations, the REST API and Python SDK give you server-side control over avatars and realtime sessions, so your secret stays on your backend while you create or manage sessions programmatically. The docs at https://docs.protoface.com cover the exact request shapes and supported fields.

For example, a backend service can create a session with its own API key and hand the client only a short-lived session reference:

import os

print(session.id)

import os

print(session.id)

import os

print(session.id)

That pattern is usually preferable to exposing TTS credentials because it preserves a single trust boundary: all secrets and policy live server-side, while the browser only gets ephemeral session data.

Conclusion

Proxying TTS for realtime avatars is mostly about discipline: keep secrets on the server, stream audio instead of buffering it, and make session policy explicit. If you do those three things, you get a design that is safer, easier to reason about, and much more compatible with lip-synced realtime media.

If you’re building on a voice-agent stack, keep the boundary narrow: the browser talks to your backend, your backend talks to TTS, and the avatar/media layer consumes the resulting stream. If you want to see the exact Protoface integration points, start with the docs at https://docs.protoface.com and the relevant quickstarts in the GitHub org.