Introduction

Echo cancellation is one of those problems that looks trivial in a lab and becomes annoying the moment you deploy a voice agent into a real room. The core issue is simple: the microphone hears not only the user’s voice, but also the speaker output from the assistant. If that leaked audio gets fed back into the ASR or turn-taking logic, you get doubled words, false interruptions, unstable VAD, and a conversational loop that feels broken.

If you are building realtime voice AI with an avatar layer like Protoface, the video face is usually not the hard part. The hard part is making the audio chain behave across laptops, earbuds, conference rooms, and browser tabs with different acoustic paths. By the end of this post, you should be able to reason about where echo comes from, what echo cancellers can and cannot do, and how to set up your app so the avatar stays synchronized without making audio quality worse.

What echo cancellation actually does

Acoustic echo cancellation, or AEC, is not a generic “remove all background noise” feature. It is specifically designed to subtract a known render signal from the microphone signal after accounting for the room’s impulse response and latency. In a voice AI stack, the far-end signal is your agent’s audio playing through the speaker; the near-end signal is the user speaking into the mic.

Conceptually, the mic stream looks like:

mic(t) = user_voice(t) + room_response ∗ speaker_audio(t - delay) + noise(t)

An AEC system estimates the delayed, filtered speaker contribution and removes it. Good AEC depends on three things:

• Accurate knowledge of what was rendered to the speaker.

• Reasonable latency estimation between render and capture paths.

• A stable acoustic path, or at least a path that changes slowly enough to track.

That last part is why the room matters so much. A tiny laptop in a quiet office is much easier than a laptop docked to a monitor speaker in a glass meeting room.

Microphone and speaker choices change the problem shape

The hardware itself strongly affects how much cancellation you need and how hard the problem is. A headset with closed-back headphones and a close-talk mic almost sidesteps acoustic echo because the speaker is effectively isolated from the mic. A laptop microphone paired with built-in speakers does the opposite: the speaker output is loud, physically close to the mic, and reflected by the desk and display.

A few practical observations from real deployments:

• Headsets are the easiest path for desktop agents. They reduce acoustic echo before software has to touch it.

• Speakerphone mode is the hardest path because the mic and speaker often share the same device and the acoustic coupling is strong.

• Beamforming arrays help, but only if the array can spatially reject the speaker and the user is positioned reasonably well.

• Built-in laptop speakers can be surprisingly problematic because they excite the desk and screen, not just the air.

Software AEC can handle some of this, but not all of it. If your product relies on the browser’s capture pipeline, remember that the browser can only cancel what it can model. A poorly chosen microphone/speaker pair can exceed the cancellation budget even when the DSP is working correctly.

Room acoustics determine how “predictable” the echo is

AEC works best when the acoustic path between speaker and mic is mostly linear and time-invariant over short windows. Real rooms violate that assumption in a few ways:

• Reverberation stretches the speaker signal into many delayed reflections. Longer reverberation time means the echo tail is harder to model.

• Early reflections off desks, walls, and monitors create strong, short-delay copies that are especially confusing because they overlap with speech onset.

• Nonlinear distortion from clipped speakers or overdriven laptops creates harmonics that are not easy for standard cancellers to remove.

• Moving objects like people, laptop lids, or a turned head change the room impulse response mid-call.

In practice, this means a clean office with carpet and absorptive surfaces is significantly friendlier than a conference room with a large table, hard walls, and one speakerphone sitting in the center. Even a decent AEC implementation can struggle when the echo path is rapidly changing because the mic is picking up a mix of direct sound, delayed reflections, and the user’s own voice at the same time.

If you are debugging an agent that keeps hearing itself, do not start by blaming the model. First check the room: speaker volume, mic placement, desk reflections, and whether the device is using an actual headset profile or just the default system input.

What matters in the software pipeline

For a realtime agent, the audio pipeline typically has four stages: capture, echo reference, cancellation, and transport. The important detail is that the echo reference must match exactly what was sent to the speaker path, including sample rate conversion, gain changes, and any jitter introduced by the render schedule. If your reference signal is stale or mismatched, the canceller will subtract the wrong waveform and leave residual echo behind.

There are also a few developer-facing trade-offs worth calling out:

1. Lower latency helps turn-taking, but it also tightens the timing window for AEC alignment.

2. Aggressive noise suppression can hurt speech fidelity and sometimes removes low-energy consonants the ASR needs.

3. Double-processing is risky. Stacking browser AEC, OS-level processing, and an external DSP can create artifacts if each layer makes different assumptions.

4. Speaker playback level matters. More volume increases echo energy faster than AEC capacity.

For voice agents, there is a subtle failure mode that often looks like “the model interrupted itself.” What is actually happening is the agent’s own output is leaking into the mic, the VAD or endpointing logic misclassifies it as user speech, and the conversation stack reacts to its own voice. That is why echo cancellation is not just about audio cleanliness; it is part of the control loop.

How to instrument and debug it

When a call sounds wrong, I recommend a simple sequence:

1. Test with headphones. If the issue disappears, the problem is acoustic coupling, not the model.

2. Reduce speaker volume by 6–12 dB and retest.

3. Move the mic farther from the speaker or rotate the device.

4. Disable extra “enhancement” layers temporarily so you can identify which component is introducing artifacts.

5. Check whether the app is using the correct input device and whether echo cancellation is enabled in the browser or OS path.

For browser-based flows, you can also inspect whether the media constraints are selecting echo cancellation and noise suppression. Exact support varies by browser and device, but the intent is the same: give the platform enough signal processing to handle common room setups before your app has to compensate.

const stream = await navigator.mediaDevices.getUserMedia({
});

const stream = await navigator.mediaDevices.getUserMedia({
});

const stream = await navigator.mediaDevices.getUserMedia({
});

That is not a cure-all, but it is a sensible baseline for browser capture. If you are driving a voice agent from the browser and the user is on speakers, these constraints often improve the default experience materially.

Where Protoface fits in a realtime voice stack

Protoface sits on the avatar side of the pipeline, so the main job is to keep the face synchronized with the audio stream without making audio debugging harder. If you are using the LiveKit Agents plugin, for example, the avatar is added as a visual surface to an existing realtime agent. The important point is that the voice agent still owns audio quality, AEC behavior, and turn-taking; the avatar should not be treated as an excuse to loosen audio discipline.

A minimal Python-oriented setup would look conceptually like this, with the exact fields following the docs:

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

If you are wiring into LiveKit, the plugin approach keeps the integration close to your agent runtime. See the relevant package and examples in the repo here: github.com/protoface-ai/protoface-plugin-pipecat. For the broader API surface, use the docs when you need the exact session, avatar, or embed parameters.

The practical win is that you can debug audio independently of the avatar. If the agent is echoing, fix the audio path first. Once the audio is stable, the synced face usually just works.

Conclusion

Echo cancellation in voice AI is mostly an acoustics problem, not an AI problem. Microphone type, speaker placement, room reflections, and render/capture timing all shape how much of the agent’s own voice leaks back into the mic. The better your hardware and room setup, the less your software has to guess.

If you are building a realtime avatar experience, keep the audio pipeline disciplined: use sensible capture settings, avoid unnecessary processing layers, and test in the same kind of room your users will actually have. Then add the avatar on top of that stable foundation.

For implementation details, quickstarts, and API shapes, start with docs.protoface.com. If you want a working agent integration path, the LiveKit and Pipecat examples in the linked repos are the fastest way to see the pieces assembled end to end.