Introduction

In a daily-powered conversational video agent, the audio path is usually the first thing that goes wrong. You connect a microphone, a speech model, a TTS engine, and a lip-synced avatar, then suddenly users hear themselves echoed back, the agent starts re-triggering on its own output, or the whole conversation feels “roomy” and unstable. The root cause is almost always the same: you have multiple live audio sources, at least one speaker in the loop, and no strict separation between what is meant to be listened to and what is meant to be played.

This post breaks down the feedback mechanisms that show up in realtime avatar systems and how to control them. By the end, you should be able to reason about the audio graph, decide where echo cancellation belongs, and configure your agent so it can speak and listen without self-interference.

Start by mapping the audio loop, not the avatar

Feedback problems are easier to debug if you ignore the video layer initially. The avatar is usually a consumer of synthesized speech, not the source of the bug. The loop you need to model is:

user microphone → voice activity detection / ASR → agent → TTS → speaker/output device → room acoustics → microphone

Once the avatar is added, you often also have a second path:

TTS audio → avatar lip sync/rendering

The video face itself does not create acoustic feedback, but it can make the system feel “more live,” which makes any audio instability more noticeable. If the avatar speaks with a delay, cuts off, or re-triggers its own turn detection, users will perceive it as a broken conversation even when the root cause is an audio routing issue.

The important distinction is between:

• Acoustic feedback: the speaker output is physically picked up by the microphone.

• Digital loopback: your software routes the agent’s own output back into its input path.

• Turn-detection contamination: VAD or barge-in logic treats agent speech as user speech.

Eliminate digital loopback first

Digital loopback is the easiest bug to reproduce and the easiest to fix. It usually comes from one of three mistakes:

1. You subscribe to the wrong audio track and feed mixed output back into ASR.

2. You play the agent’s TTS into the same stream you are recording for the agent.

3. Your media pipeline does not distinguish user tracks from agent tracks, so both are processed as input.

The rule is simple: the agent should only listen to the user’s inbound track, and only speak on its own outbound track. If your runtime supports separate input and output channels, use them. If it does not, introduce a mixer boundary and explicitly tag each frame by origin.

A practical guardrail is to log the source of every audio frame in development. If you ever see a frame generated by the agent re-enter the ASR path, you have a routing bug, not an acoustic problem.

For WebRTC-based systems, also be careful about browser defaults. A local preview that works with speakers muted can fail in production if the same page captures and renders audio without isolating tabs, devices, or streams. If you are testing in a browser, use headphones during development and verify that your code subscribes only to the remote user track intended for the agent.

Use echo cancellation, but treat it as a last line of defense

Acoustic echo cancellation (AEC) helps when the speaker output is physically leaking back into the microphone. It is useful, but it is not a substitute for correct routing. Good AEC can suppress a lot of room echo, but it cannot reliably fix a software loop that sends your own output back into your input pipeline.

In general:

• AEC helps when the mic hears the speaker in the same room.

• Noise suppression helps with HVAC, fans, and constant background noise.

• Voice activity detection helps decide when the user is actually speaking.

• Barge-in control determines whether the agent should stop talking when the user interrupts.

These systems interact. AEC reduces false positives in VAD, which reduces unnecessary interruptions, which in turn reduces the chance that your agent keeps reacting to itself.

One subtle gotcha: some browser and device stacks apply automatic gain control or built-in echo cancellation in ways that vary across platforms. That means a session that is stable on macOS with headphones can behave differently on a mobile browser or on a laptop speaker. If your product depends on spoken conversation in open rooms, test with actual speakers, not just headphones.

Design your turn-taking so the agent does not fight itself

Even when the audio routing is correct, turn-taking can still create a “loopback” symptom at the application layer. For example, if your VAD is too sensitive, the agent may interrupt itself because its own synthesized speech bleeds into the user-input detector. Or if you stream partial transcripts too aggressively, the language model may respond to fragments that are actually just residual audio from the previous turn.

A robust conversational design usually includes:

1. Input gating: only accept audio from the user input track, and only during the active listen window.

2. Speaker gating: mute or pause user input processing while the agent is generating a response, unless you explicitly support barge-in.

3. VAD thresholds: tune onset and offset so short echoes do not count as speech.

4. Barge-in policy: decide whether interruption is allowed, and if so, what stops first: TTS playback, avatar animation, or the agent’s text generation.

The implementation details depend on your stack, but the principle is consistent: the agent should have one authoritative state machine for “listening,” “thinking,” and “speaking.” Most feedback bugs appear when those states overlap in ways you did not intend.

Keep the media graph clean in realtime video avatars

A conversational video agent adds a visual consumer of speech, but the visual layer should not be allowed to alter the audio signal path. The safest architecture is:

• capture user audio once, at the edge of the session;

• process that audio through ASR/VAD only once;

• send the model’s response to TTS once;

• play the synthesized audio only to the user output track;

• feed the same synthesized audio to the avatar renderer for lip sync, but never back into user input.

That separation sounds obvious, but it is easy to violate accidentally when different services own different parts of the pipeline. If you are using a WebRTC agent framework, inspect both your subscription logic and your playback logic. If you are using a browser embed, verify which side is responsible for capturing microphone input and which side is responsible for rendering audio.

Another practical issue is session lifetime. When a session restarts or reconnects, stale tracks can remain attached if your cleanup logic is incomplete. That can create a phantom feedback path that appears only after a reconnect. Treat reconnects as first-class test cases, not edge cases.

Where Protoface fits

In a LiveKit voice agent, the cleanest way to keep the avatar visually synchronized without disturbing the audio graph is to add the avatar as a separate realtime surface rather than mixing it into your speech pipeline. The quickstart examples and the docs show the expected integration pattern: your agent continues to own ASR, LLM, and TTS, while the avatar consumes the synthesized speech for lip sync.

If you are using the LiveKit agent stack, the plugin path is the most direct. The Protoface LiveKit plugin is designed to drop a synchronized talking face into an existing voice agent without changing the underlying audio routing. That matters because the avatar should not become part of the microphone input chain. Keep the agent’s inbound audio separate, and let the avatar render the agent’s output visually.

# Illustrative only; exact setup fields are in the docs

# Illustrative only; exact setup fields are in the docs

# Illustrative only; exact setup fields are in the docs

If you are wiring things up by API instead of a plugin, the same principle applies: create or manage a session server-side, keep credentials out of the browser, and make sure your frontend only receives the scoped runtime data it needs. A minimal REST call to create a session might look like this:

curl -X POST https://api.protoface.com/sessions \

curl -X POST https://api.protoface.com/sessions \

curl -X POST https://api.protoface.com/sessions \

The exact request shape will depend on the endpoint, but the operational idea is the same: manage sessions centrally, keep audio ownership clear, and avoid browser-side code that can accidentally subscribe to its own output.

Debugging checklist that actually helps

When feedback or loopback shows up, work through the problem in this order:

1. Mute the speaker. If the problem disappears, you have acoustic echo or playback leakage.

2. Inspect track routing. Confirm the ASR path only receives user input.

3. Disable barge-in. If the issue disappears, your turn-taking logic is too permissive.

4. Lower mic gain. If this helps, your input chain is too sensitive or your room is too loud.

5. Test with headphones. If the problem persists, it is probably digital, not acoustic.

6. Log frame origin. The first audio frame that re-enters the input path usually tells you where the loop starts.

If you want a more systematic approach, record short sessions and inspect them offline. A ten-second transcript with timestamps for user audio, agent audio, VAD events, and turn boundaries is often enough to pinpoint the defect. Most teams debug this faster when they can see exactly when the agent started speaking relative to the last user frame.

Conclusion

Reducing feedback in a conversational video agent is mostly about discipline in the audio graph. Keep user input and agent output separate, use echo cancellation as support rather than a crutch, and make turn-taking state explicit. The avatar layer should be visually synchronized with the agent, but never entangled with the microphone path.

If you are building this with a realtime avatar platform, check the session and integration docs, then validate your routing in a small end-to-end test before you ship. For more implementation details and integration examples, see the docs and the relevant quickstarts in the GitHub org.