Text-to-Speech in Unity for Talking Avatars: On-Device Audio vs Cloud TTS Compared
Unity text-to-speech for talking avatars: compare on-device vs cloud TTS, streaming latency, lip sync timing, buffering, and interruption handling.
Introduction
Building a talking avatar in Unity is mostly an audio systems problem disguised as an animation problem. The avatar only looks “alive” if speech arrives with low latency, the lip sync stays aligned to the audio stream you actually render, and the implementation does not fight Unity’s frame loop or your networking stack.
If you are deciding between on-device text-to-speech and cloud TTS for a Unity avatar, the useful question is not “which sounds better?” It is “which path gives me predictable latency, acceptable quality, manageable memory/CPU cost, and the right deployment properties for my app?” By the end of this post, you should be able to choose a TTS architecture for a realtime avatar, wire it into Unity, and understand where a managed avatar service like Protoface fits when you want to stop building the streaming and synchronization plumbing yourself.
How TTS fits into a realtime avatar pipeline
A talking avatar pipeline usually has four stages:
Text generation or user input.
TTS synthesis into an audio stream or audio file.
Playback in Unity, with lip sync driven from the same audio timing.
Optional turn-taking / interruption handling so speech can be stopped or replaced mid-utterance.
The key technical constraint is that audio and facial animation must stay coupled. If the lip sync system reads a different clock, or if the audio buffer underflows and shifts timing, the face will visibly drift. That is true whether the speech comes from a local engine or a cloud service.
In practice, cloud TTS and on-device TTS differ in where synthesis happens and how much of the transport you own:
On-device TTS: synthesis runs inside your app or on the user’s machine. You control the call site and often get immediate access to PCM samples, but you pay in binary size, memory, and platform-specific integration.
Cloud TTS: you send text to an API and receive audio back. You get better quality and less local complexity, but you add network latency and must engineer around partial audio arrival and failure modes.
On-device audio: when control matters more than model quality
On-device TTS is attractive for offline support, privacy-sensitive applications, and tightly constrained latency budgets where you can keep the engine warm. In Unity, the practical upside is that audio never has to cross the network before playback. If your TTS library exposes PCM frames directly, you can drive both audio output and lip sync from the same buffer.
The trade-offs are real:
Quality ceiling: local engines are often behind the best cloud voices, especially for prosody and expressiveness.
Memory footprint: voice models can be large, and multiple voices are expensive.
Platform fragmentation: what works on Windows may not work on iOS, consoles, or WebGL.
Runtime cost: synthesis can compete with gameplay, animation, and rendering for CPU/GPU budget.
For Unity specifically, the safest integration pattern is to synthesize off the main thread, then enqueue audio buffers for playback on the main thread. Avoid blocking the frame loop while waiting for a full utterance. If your engine only emits whole clips, you can still build a decent avatar, but you lose responsiveness for barge-in and mid-turn interruption.
Cloud TTS: better voice quality, but you own the streaming edge cases
Cloud TTS is usually the right choice when the avatar is part of a customer-facing product and voice quality matters. You get better models, consistent upgrades, and less device-specific maintenance. The cost is that your app now depends on network round trips and API semantics.
The most important implementation detail is whether the provider supports streaming audio. For a talking avatar, streaming matters more than raw latency numbers, because you want the first phonemes to start playback as soon as they are available. If you wait for the full clip, your app will feel sluggish even if the total synthesis time is reasonable.
With cloud TTS, the normal failure modes are:
Cold-start latency: the first request after idle takes longer.
Chunk timing jitter: audio packets arrive unevenly and must be buffered.
Turn cancellation: a user interrupts the agent, so the current synthesis must stop cleanly.
State drift: the avatar animates from text timing, but playback starts later than expected.
For Unity, the implementation pattern is usually:
Request a streaming synthesis response.
Append audio chunks into a jitter buffer or circular buffer.
Start playback once the buffer reaches a small threshold.
Drive lip sync from the same sample clock used by the audio source, not from wall clock timestamps alone.
A minimal cloud TTS request from a backend might look like this. The exact payload fields depend on the provider and your setup, but the shape is representative:
For Unity, you would typically keep the API key off the client and let a backend mint the session or provide a short-lived token. That is true for any cloud TTS or avatar service; do not bake long-lived credentials into a shipped build.
Lip sync accuracy depends on the audio clock, not the speech model
Developers often over-focus on TTS quality and under-focus on timing. For a talking avatar, a very good voice with poor sync looks broken, while a decent voice with stable timing looks acceptable.
The rule is simple: use the audio output clock as the source of truth. In Unity, that usually means measuring playback position from the AudioSource rather than assuming the synthesis timestamps match wall-clock time exactly. If you are doing viseme or blendshape driving, map phoneme or viseme events to the exact sample offsets used for playback. If you only have text, you can still estimate mouth motion from audio amplitude, but that is a fallback, not a precise lip sync strategy.
Two practical gotchas:
Resampling changes timing: if the engine emits 24 kHz and Unity outputs at 48 kHz, make sure your timing model accounts for conversion.
Buffer size affects perceived latency: larger buffers reduce stutter but delay mouth motion and first audio.
If your avatar supports interruption, you also need a clean stop path. Stopping playback immediately is not enough; you should clear queued chunks, invalidate any pending speech marks, and transition the face back to a neutral or listening state.
Where Protoface fits: offloading the avatar synchronization layer
If your goal is not “build a TTS engine” but “ship a realtime avatar inside a voice experience,” a managed avatar layer can save a lot of glue code. Protoface provides the avatar/session side of the problem: you create a session, attach it to your voice flow, and get synchronized talking video back without having to build your own video-face transport, timing pipeline, or browser embedding layer from scratch.
For server-side orchestration, the REST API is the most direct integration point. A backend can create a session, attach your chosen voice settings, and hand the session data to Unity or a web client. If you are already using a Python backend, the Python SDK is the same idea with less request plumbing. The exact object names and fields are in the docs, but the workflow is straightforward: authenticate with an API key, create an avatar/session, then stream or join the realtime experience.
A very small Python example might look like this:
If you are wiring Unity to a backend voice agent rather than calling TTS directly from the client, this separation is usually cleaner: Unity handles rendering and local input/output, while the backend owns credentials, session creation, and speech orchestration.
Choosing the right approach for Unity
Use on-device TTS when you need offline support, strict data locality, or a fully self-contained client build and you can live with voice quality trade-offs. Use cloud TTS when the avatar is part of a production product and voice quality, naturalness, and rapid iteration matter more than local autonomy.
For most talking-avatar applications, the real bottleneck is not synthesis itself; it is keeping audio transport, mouth animation, and interruption handling coherent. If you are building that stack in Unity, start by deciding where audio will originate, then design around the audio clock and buffer model before you touch the visuals.
If you want a reference implementation or need to connect Unity to a broader realtime agent system, the docs at docs.protoface.com are the right place to start. From there, you can pick the integration surface that matches your architecture: backend-driven sessions, a Python service, or a realtime voice agent workflow that hands off avatar synchronization to the platform instead of implementing it yourself.
Conclusion
The practical decision is simple: on-device TTS gives you local control, while cloud TTS gives you better voice quality and less platform-specific work. For Unity talking avatars, the hard part is not just generating speech; it is keeping the generated audio, the playback clock, and the facial animation aligned under real network and runtime constraints.
Once you choose your synthesis path, build around streaming, buffering, and interruption from the start. If you want to avoid wiring all of the avatar/session plumbing yourself, use the docs and quickstarts to get a realtime avatar session running, then integrate Unity against that stable interface. Start with the documentation, verify the transport behavior early, and treat lip sync timing as a first-class systems problem, not a cosmetic one.