Introduction

Routing a realtime avatar stream through NGINX or Envoy is a good idea in a lot of production setups: you get a stable public endpoint, TLS termination, access control, and a place to centralize routing without exposing your media services directly. The catch is that avatar sessions are usually not “just HTTP.” They often involve WebSockets for signaling, potentially WebRTC for media, and strict timing between audio and video. If your proxy buffers the wrong thing, times out idle connections, or rewrites headers incorrectly, the avatar may still connect but drift out of lip-sync.

This post shows what actually matters when proxying realtime avatar traffic, how to configure NGINX or Envoy without introducing latency or connection churn, and how to debug the failure modes that look like “video works but mouths are off by a few hundred milliseconds.” By the end, you should be able to put a reverse proxy in front of a realtime avatar service and keep the stream stable enough for production use.

Start with the transport, not the brand name

The right proxy configuration depends on what is being proxied. For realtime avatars, the control plane is usually HTTP, while the media plane may be WebRTC or an equivalent low-latency stream. Those two classes of traffic behave very differently:

• REST and dashboard traffic: normal HTTP requests. Reverse proxying is straightforward.

• WebSocket signaling: long-lived upgraded connections. You must preserve upgrade headers and avoid buffering.

• WebRTC/media paths: often not suitable for a generic HTTP reverse proxy at all. If the avatar uses WebRTC directly, the proxy should usually front the API/signaling layer, not sit in the media path unless the vendor explicitly supports that.

Lip-sync problems often show up when the media path is technically “working” but the proxy adds enough delay or jitter to disturb the timing relationship between audio frames and rendered video frames. In practice, that means you need to avoid any proxy behavior that batches, buffers, retransmits unnecessarily, or tears down quiet-but-live connections.

What breaks lip-sync at the proxy layer

There are a few common mistakes that are easy to miss because they do not look like obvious failures:

1. Response buffering: if the proxy buffers upstream responses, it can hold back chunks that should reach the client immediately. For streaming video or incremental media metadata, that adds visible latency.

2. Idle timeouts: realtime avatar sessions can go quiet at the transport layer for short periods while audio is still being processed. Default proxy timeouts may close “idle” sockets too aggressively.

3. Incorrect WebSocket upgrade handling: if the proxy does not forward Upgrade and Connection headers properly, the signaling channel may fall back to polling or fail entirely.

4. Header rewriting or protocol downgrades: media systems are often sensitive to origin, host, and scheme. Mismatched X-Forwarded-Proto or Host headers can break session setup, cookies, or signed URLs.

5. Compression and re-chunking: do not gzip a realtime stream unless you know the endpoint is meant for it. Compression can increase latency and distort delivery timing.

The key point is that lip-sync is not only a model problem. If audio and video are generated on time but delivered inconsistently, the user still perceives drift. Proxies are often the hidden source of that inconsistency.

NGINX: keep it transparent, long-lived, and boring

For NGINX, the safest pattern is to treat the avatar service as an upstream that needs raw pass-through for upgraded connections and minimal interference for streaming endpoints. A typical configuration for the API or WebSocket signaling looks like this:

location / {
}

location / {
}

location / {
}

Two details matter more than most people expect:

• proxy_buffering off keeps NGINX from collecting upstream chunks before forwarding them.

• Long proxy_read_timeout and proxy_send_timeout values prevent sessions from being killed while the user is speaking or while the avatar is rendering.

If you have separate endpoints for REST and realtime signaling, split them into separate location blocks. Keep the REST side conventional and reserve the “no buffering, no surprises” profile for the streaming path. That makes it easier to tune caching, rate limiting, and logging independently.

Envoy: control the timeouts and upgrade behavior explicitly

Envoy gives you more granular control, but the same principles apply. You want a route that preserves upgraded connections, avoids request/response buffering for streaming traffic, and sets timeouts high enough to match the session duration rather than the request duration.

For a WebSocket-style route, the important knobs are typically route timeout, idle timeout, and any protocol-specific upgrade settings. Exact YAML varies by deployment, but the intent is the same:

route_config:
idle_timeout: 3600s

route_config:
idle_timeout: 3600s

route_config:
idle_timeout: 3600s

Setting timeout: 0s is common for long-lived upgraded connections because you do not want Envoy to impose a normal request deadline on a stream that is supposed to stay open. If you are proxying REST calls too, keep those on a separate route with ordinary request timeouts so you do not accidentally make every API call “infinite.”

For WebRTC itself, be careful not to assume Envoy should sit in the media path. In many real deployments, the proxy only fronts the control plane while the actual media connections use direct peer-to-peer or managed relay infrastructure. If your architecture does require a proxy in the media path, validate it with packet capture and latency measurements rather than assuming an HTTP configuration is enough.

Operational checks that catch timing problems early

When lip-sync looks wrong, the root cause is often easier to find with a few concrete checks than with generic “watch the logs” advice:

1. Measure upstream and downstream latency separately. If the upstream avatar service is returning frames on time but the client sees them late, the proxy is the likely bottleneck.

2. Confirm upgraded connections stay upgraded. A surprising number of “realtime” issues are just WebSockets being downgraded, retried, or closed by an intermediary.

3. Inspect buffering and chunking. If your proxy logs show large response chunks instead of a steady stream, you are probably buffering somewhere.

4. Look for reconnects during idle speech gaps. If the connection is reset during brief pauses, raise the read timeout and check any intermediary load balancers as well.

5. Test with real audio, not synthetic pings. Lip-sync depends on end-to-end timing under load, not just whether the socket stays open.

Also make sure your proxy preserves the client-facing origin and scheme if the backend uses signed session URLs, cookie-based auth, or parent-origin checks. A lot of seemingly random failures are actually caused by a mismatch between what the browser thinks it connected to and what the backend sees.

How Protoface fits in

This is exactly the kind of deployment detail you want to solve once at the edge instead of in every app. Protoface exposes both a REST API and developer-facing realtime surfaces, so you can front the service with NGINX or Envoy while keeping the timing-sensitive parts as transparent as possible. If you are wiring avatars into a voice agent, the LiveKit plugin is the most relevant integration point, and the repository examples are the best place to see how the session is created and managed in practice: https://github.com/protoface-ai.

A minimal REST call shape looks like this; exact fields depend on the session or avatar object you are creating, so use the docs for the real schema:

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","voice":"en-US"}'

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","voice":"en-US"}'

curl -X POST https://api.protoface.com/v1/sessions \
-d '{"avatar_id":"avt_123","voice":"en-US"}'

If you are building the session lifecycle in Python, the SDK is a cleaner fit than hand-rolling requests, especially once you start passing through proxy-aware endpoints and handling retries. See the package and examples in the SDK repo: https://github.com/protoface-ai/protoface-sdk-python.

The practical takeaway is that Protoface gives you a clean upstream to proxy: you can keep your own edge stable, while the avatar session mechanics stay in the vendor API or plugin layer rather than leaking into your application code.

Conclusion

If your realtime avatar stream loses lip-sync behind NGINX or Envoy, the fix is usually not “add more retries.” It is to stop the proxy from behaving like a normal HTTP cache or request router and instead treat it like a transparent transport layer for long-lived upgraded connections. Disable buffering where needed, preserve upgrade headers, keep timeouts aligned with session duration, and avoid putting generic HTTP middleware in the media path unless the protocol explicitly allows it.

Once the edge is boring, the avatar stack is much easier to reason about. For implementation details, supported fields, and up-to-date integration guidance, start with https://docs.protoface.com. If you are using LiveKit, Python, or direct API integration, the examples there will map cleanly onto the proxy patterns described above.