Troubleshooting Realtime Avatar Latency on Interactive Kiosks and Public Displays
Debug realtime avatar latency on kiosks: measure ASR, TTS, network, and browser rendering bottlenecks.
Introduction
Realtime avatar latency on an interactive kiosk or public display usually shows up as one of three failures: the avatar starts speaking noticeably after the user finishes, the lip sync drifts behind the audio, or the video freezes while audio continues. In a controlled demo, these issues are annoying. In a lobby screen, retail kiosk, or museum display, they make the system feel broken.
This post is about debugging those delays systematically. By the end, you should be able to identify where the latency is coming from, separate network problems from rendering problems, and make practical trade-offs between quality, responsiveness, and reliability.
The core idea is simple: a realtime avatar is not “just video.” It is a pipeline with multiple queues—mic capture, speech recognition, LLM reasoning, TTS or speech synthesis, avatar rendering, transport, and browser playback. The latency you perceive is the sum of those stages, plus any jitter introduced by the public display environment.
Start by measuring the right latency
Before changing settings, define what “latency” means in your app. Different symptoms map to different root causes:
Turn-taking latency: time from user speech end to avatar speech start.
Video lead/lag: difference between audio onset and lip movement onset.
Frame delivery latency: time from render to display on the kiosk browser.
Interaction lag: delay between a tap, button press, or wake word and visible reaction.
For kiosks and public displays, “feels slow” often comes from turn-taking latency, not raw video generation time. A 300 ms render path can still feel sluggish if your ASR waits for long end-of-utterance detection, your LLM blocks on a long prompt, or your TTS doesn’t stream early audio.
A useful debugging habit is to timestamp each boundary in the pipeline:
If you can log those markers, you can stop guessing. In practice, the biggest wins come from reducing the largest gap rather than trying to shave 20 ms off every stage.
Latency usually comes from buffering, not compute
Most realtime systems are optimized for throughput, and kiosks punish that choice. Any component that buffers for quality can add perceptible delay.
Common culprits:
Endpointing on the ASR side: waiting too long to decide the user is done speaking.
Non-streaming LLM calls: generating the full reply before anything can be spoken.
Batch-oriented TTS: requiring the whole response before the first audio chunk is emitted.
Video frame accumulation: buffering several frames before playback to smooth jitter.
Browser autoplay policies or decoding stalls: especially on locked-down public devices.
For interactive avatars, you want the opposite bias: start early, stream continuously, and tolerate small imperfections. That usually means:
Use an ASR that can emit partial transcripts and finalize quickly.
Generate the response incrementally or at least avoid large preambles.
Use streaming TTS so audio starts as soon as the first phrase is ready.
Keep the avatar transport path low-jitter and avoid unnecessary transcoding in the browser.
One subtle issue on kiosks is that the display machine is often underpowered relative to a developer laptop. A browser tab may be competing with signage playback, analytics scripts, idle-screen animations, or OS-managed power saving. If frames are decoded on a low-end CPU or the GPU is clocked down, you can see stutter even when the network is fine.
Separate network latency from rendering latency
When public displays are remote from your backend, it is tempting to blame the network for all lag. Sometimes that is correct; often it is not. The easiest way to distinguish the two is to compare audio arrival time with visual update time.
If audio starts on time but the face animation lags or freezes, the problem is likely in the client playback path: decode, compositing, browser event loop, or GPU contention. If both audio and video are late together, the delay is upstream in the agent or transport path.
Practical checks:
Measure round-trip RTT from the kiosk network, not from your office network.
Check packet loss and jitter; realtime media handles latency much better than burst loss.
Verify the kiosk uses wired Ethernet when possible; public Wi-Fi is a latency multiplier.
Disable aggressive power-saving modes that downclock the browser process or network adapter.
Avoid heavy DOM reflows around the avatar container.
If you are using an iframe-based embed, remember that the host page still matters. Even with a clean network path, a busy parent page can starve the browser main thread and delay paints. Keep the kiosk page simple. One big iframe is easier to keep smooth than a dashboard-style shell with several animated panels around it.
Tune the interaction loop for public-facing environments
Public displays have a different latency budget than agent-to-agent workflows. Users expect an immediate acknowledgment. Even if the full answer takes a moment, the system should visibly react right away.
Good patterns:
Immediate visual acknowledgement: the avatar looks at the user, blinks, or shows a listening state as soon as input begins.
Short backchannels: a quick “one moment” or nonverbal cue while the system is processing.
Partial response streaming: start the answer with the first stable clause instead of waiting for the perfect full paragraph.
Bounded context: keep prompts concise so the agent does less work per turn.
For kiosks specifically, it helps to design for interruptions. A user may walk away mid-turn, another user may step up, or the microphone may pick up ambient speech. Your avatar should recover quickly without waiting for a stale response to drain through the pipeline.
Also watch for “latency amplification.” A 150 ms delay in each of four stages is already 600 ms. Add a 500 ms ASR endpointing window and a 300 ms TTS startup delay, and you are now in the zone where users start repeating themselves. The goal is not perfection; it is keeping the total loop comfortably below the threshold where the interaction feels asynchronous.
How to inspect the system in practice
When I troubleshoot these setups, I start with a minimal reproduction. Remove everything not essential to one turn of speech. Then reintroduce complexity until the latency appears.
A few concrete tests:
Network-only test: play a prerecorded response through the kiosk network path. If playback is smooth, the transport layer is probably fine.
Agent-only test: use a short canned response from the agent. If this is fast but live responses are slow, the issue is ASR/LLM/TTS.
Render-only test: render the avatar with a fixed audio clip. If this stutters, the client display path is the culprit.
Idle test: leave the kiosk up for an hour. Some browsers degrade over time because of memory pressure, throttling, or background tasks.
On locked-down devices, browser configuration matters more than people expect. If the kiosk runs in a managed Chromium shell, confirm that hardware acceleration is actually enabled, the tab is not being background-throttled, and display refresh settings match the panel. A 60 Hz screen with a 30 fps render loop is usually fine; a 60 Hz screen with an unstable 15-25 fps loop reads as janky immediately.
Using Protoface in a live voice agent pipeline
This is where Protoface fits naturally: it adds the talking face layer to an existing voice agent without forcing you to re-architect the rest of the stack. If you are using a LiveKit voice agent, the plugin can drop in a synchronized avatar so the agent’s speech and facial animation stay aligned. The relevant examples live in the plugin repository and in the docs at docs.protoface.com.
A minimal LiveKit-style integration looks like this:
That snippet is intentionally generic. The point is not the exact constructor shape; it is that the avatar becomes part of the same realtime pipeline as the voice agent, which reduces glue code and helps keep timing aligned. If you are building a kiosk, this is usually preferable to stitching together separate services in the browser.
For lower-level control, the REST API can be useful when you want to create or manage avatars and sessions server-side. A typical flow is to create a session, then connect your client to it. Authentication uses API keys, so keep that on the backend:
If you are building the kiosk itself as a simple browser client, the iframe embed is often the fastest path because you do not expose backend credentials in the browser. That is especially relevant for public displays, where you want a constrained surface with parent-origin allowlisting and session limits rather than a fully custom client.
Operational fixes that actually move the needle
In practice, the best improvements on public displays come from a short list:
Prefer wired network over Wi-Fi.
Keep the kiosk page minimal and avoid expensive UI work around the avatar.
Use streaming ASR/TTS and reduce endpointing delays.
Limit prompt size and response length so the agent starts speaking sooner.
Enable hardware acceleration and verify the browser is not throttled.
Instrument the pipeline so you can see whether the delay is upstream or downstream.
Also set expectations honestly. A kiosk that needs to answer highly variable free-form questions will never feel as snappy as one that handles a constrained task flow. If the interaction matters more than the richness of the answer, optimize for responsiveness first and quality second. Users forgive shorter answers much more readily than they forgive dead air.
Conclusion
Realtime avatar latency is rarely one problem. It is usually several small delays stacked across ASR, reasoning, synthesis, transport, and rendering. The way to debug it is to measure each boundary, remove buffering where you can, and keep the display machine as boring as possible.
If you need a quick path to a production-grade avatar pipeline, check the examples and integration notes in the docs and the relevant GitHub repos, especially if you are working with LiveKit or a browser embed. Start with a minimal turn, instrument it, and only then add complexity. That approach will tell you very quickly whether your kiosk is slow because of the network, the agent, or the screen itself.
For implementation details, examples, and integration options, see docs.protoface.com.