Introduction

If you are building a realtime avatar app, the core architectural choice is not “which model is best?” It is whether you want to own the entire speech pipeline yourself or delegate part of it to a platform like Protoface. That decision matters more once you add multilingual support, because the failure modes multiply: recognition latency, language detection errors, synthesis quality across accents, and timing drift between audio and lip motion.

This post compares Vapi-style managed voice-agent orchestration with a custom STT/TTS pipeline for realtime avatars. The goal is practical: by the end, you should be able to decide which architecture fits your latency budget, your language coverage requirements, and how much infrastructure you are willing to operate.

What actually changes when you add a face

A voice agent without a face is mostly a text-and-audio problem. A realtime avatar turns it into a synchronization problem.

In the browser or in a video stream, the avatar must keep three clocks aligned:

• ASR clock: partial and final transcription timing from STT.

• Generation clock: response tokenization, tool calls, and any model-side delay.

• Playback clock: synthesized audio duration and the corresponding mouth motion / video rendering.

If these drift, you get obvious artifacts: the face starts speaking before the audio, lip flaps continue after the utterance ends, or the avatar “snaps” between states on turn boundaries. For multilingual apps, drift gets worse because language identification and TTS voice selection can add branching and buffering.

So the real question is not only “can it transcribe and speak multiple languages?” but “can the system preserve temporal alignment across languages, devices, and network conditions?”

Managed orchestration vs owning STT/TTS

In a managed voice-agent stack, the platform typically handles some combination of transport, speech recognition, turn detection, synthesis, and agent orchestration. That is attractive because you can ship faster and spend less time wiring services together. For many English-only voice apps, that is enough.

For realtime avatars, though, the platform boundary matters. Once you need precise control over the speaking face, you need to know where audio is generated, where it is buffered, and how turn state is exposed to the avatar renderer. If the platform treats audio as just another output channel, you may not have enough hooks to keep video synchronized under low-latency conditions.

A custom STT/TTS pipeline gives you maximum control:

• You choose ASR per language, per accent, or even per domain.

• You can route synthesis to the best voice provider per locale.

• You can tune streaming chunk sizes, VAD thresholds, and interruption behavior.

• You can keep the avatar renderer close to the playback timeline instead of reverse-engineering it from opaque agent events.

The downside is operational complexity. You own provider failover, latency budgets, language routing, observability, and all the weird edge cases that show up when a user switches from Spanish to English halfway through a sentence.

Where custom STT/TTS is worth it

Custom pipelines are justified when one or more of these are true:

1. You need explicit language control. For example, a support app that must speak Japanese, French, and Arabic with different voices and terminology.

2. You care about provider selection per segment. One ASR provider may be strongest for one language, another for another language.

3. You need deterministic interruption behavior. In voice agents, barge-in and partial transcripts are not nice-to-haves; they define the conversational feel.

4. You want to own latency budgets end to end. Realtime avatars are sensitive to 100–300 ms differences because the user can see them.

That said, “custom” does not mean “single-provider.” A sane multilingual architecture often uses one STT provider for most languages, a second for fallback, and a TTS provider selected per locale or voice. The complexity comes from routing and synchronization, not from the providers individually.

Minimal architecture for a multilingual avatar

A clean production design usually looks like this:

1. Receive live microphone audio over WebRTC or another realtime transport.

2. Run streaming VAD and partial STT to detect language and intent early.

3. Route the transcript into an agent loop that can call tools and produce text incrementally.

4. Select a TTS voice based on detected language, user preference, or tenant policy.

5. Stream synthesized audio back while the avatar renderer consumes the same turn timeline.

6. Keep interruption handling unified so the avatar and the audio stop together.

The important implementation detail is that the avatar should not be treated as a separate “post-processing” step. It needs to be attached to the same turn lifecycle as speech generation, otherwise you will spend a lot of time compensating for skew between audio and video.

For realtime apps, the transport choice also matters. WebRTC is usually the right default because it is designed for low-latency media delivery and bidirectional interaction. If you are building a browser-facing avatar, avoid architectures that require the client to poll for state or wait for full text responses before rendering motion.

Gotchas specific to multilingual realtime systems

There are a few recurring failure modes that are easy to miss in early prototypes:

• Language detection lag. If you wait for a full utterance before choosing a locale, the first second of speech may be routed incorrectly.

• Voice mismatch. A great English voice can sound unnatural when forced onto another language, even if the transcription is correct.

• Prosody mismatch. TTS quality is not just phoneme coverage; pacing and intonation need to fit the language or the avatar will feel off.

• Turn fragmentation. Streaming ASR often emits partials that get revised. If you animate the avatar off unstable text, you may create visible jitter.

• Latency compounding. A few extra hundred milliseconds in STT, plus another in routing, plus another in TTS buffering is enough to make the interaction feel robotic.

The usual fix is to separate what the user has said from what you are willing to commit to visually. Let partial transcripts drive agent reasoning, but only commit the avatar to a speaking state when your pipeline has enough confidence that the turn has actually started.

How Protoface fits without forcing the whole stack

Protoface is useful when your main problem is the avatar surface itself: you want a synchronized talking face attached to an existing voice agent or embedded in a web app without building your own video layer. In practice, that means you can keep your custom STT/TTS choices and still hand off the synchronized avatar portion to a dedicated realtime avatar API.

If you are already using LiveKit for voice agents, the LiveKit-style integration path is the most relevant mental model: the agent continues to own speech logic, and the avatar becomes another realtime participant that stays synchronized with the spoken turn. That is a better split of responsibilities than trying to make the speech stack also solve video timing.

A minimal REST flow looks like this:

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

curl -X POST https://api.protoface.com/sessions \
}'

The exact request fields depend on the session shape in the docs, but the pattern is straightforward: create an avatar/session server-side, then connect your realtime client or agent to it. For programmatic workflows, the Python SDK is the cleanest option; see the examples in the Python SDK repo.

One practical advantage here is operational isolation. You can keep your own multilingual STT/TTS routing and still avoid building custom lip-sync and video rendering infrastructure. For web deployment, customer-managed iframe embeds are also useful when you need a browser-safe integration with no backend and no API key exposed in client code.

When a fully custom stack still wins

Even with a good avatar API, there are cases where owning the whole pipeline is still the right answer:

• Your product depends on deep customization of the rendering pipeline or nonstandard media transport.

• You need to run all media processing inside a tightly controlled environment.

• You already have a mature realtime stack and only need a very small avatar component that can be embedded as a local rendering primitive.

In those cases, custom STT/TTS plus your own avatar timing layer can be justified. Just be honest about the maintenance cost. The hidden work is not the first demo; it is the second language, the third vendor, the browser autoplay edge case, and the production support burden when network jitter shows up at scale.

Conclusion

If your product is mostly a voice agent, managed orchestration can get you to market quickly. If your product is a multilingual realtime avatar, the harder problem is not text generation; it is preserving low-latency synchronization across speech, language routing, and video motion.

A custom STT/TTS stack gives you the most control, especially for language-specific quality and latency tuning. A dedicated avatar layer reduces the amount of media infrastructure you need to own, which is often the right trade-off once lip sync and realtime rendering become first-class requirements.

If you want to see how the pieces fit together, start with the docs at docs.protoface.com and the relevant quickstarts in the GitHub org. The useful next step is not to redesign everything at once; it is to prototype one multilingual turn end to end and measure actual round-trip latency, interruption behavior, and visual sync under realistic network conditions.