Introduction

If you are building a conversational triage agent, the hard part is not the dialog policy. It is keeping the user experience coherent while the system is streaming text, audio, and visual feedback at different latencies. A useful triage agent has to acknowledge the user quickly, ask follow-up questions, and maintain a believable face that tracks the voice without visible drift or buffering artifacts.

This guide walks through the architecture and implementation choices for adding a streaming lip-synced avatar to a voice-driven triage flow in Unreal Engine. By the end, you should understand how to structure the media path, where synchronization usually breaks, and how to wire an avatar service into a realtime agent so the character speaks naturally instead of “rendering after the fact.”

What a conversational triage agent actually needs

“Triage” usually means short, structured interactions: collect intent, ask one or two clarifying questions, decide whether to route, escalate, or hand off. In practice, the agent needs four things to feel responsive:

• Low-latency turn taking: the user should hear and see an acknowledgment within a few hundred milliseconds.

• Streaming speech synthesis: you do not want to wait for a full response before starting playback.

• Continuously updated lip sync: the mouth should follow the current audio buffer, not a stale transcript.

• Stateful conversation control: triage often needs slots, confirmations, and barge-in handling.

The important consequence is that the avatar is not a separate “video asset.” It is part of the realtime media pipeline. If your agent can speak but the face updates on a delayed batch render, the user will notice immediately.

Recommended architecture in Unreal Engine

In Unreal, keep the rendering and the conversation engine decoupled. Think in three layers:

1. Conversation layer: your voice agent, LLM orchestration, and triage logic.

2. Media layer: audio stream, token stream, and avatar session state.

3. Presentation layer: the Unreal scene, camera, UI, and avatar mesh/materials.

The conversation layer should produce incremental outputs: partial transcripts, interim responses, and final actions. The media layer converts those outputs into audio playback and animation control signals. The presentation layer only consumes normalized state such as “speaking,” “listening,” “thinking,” and the current mouth/viseme parameters.

This separation matters because Unreal is good at rendering frames deterministically, but voice AI is inherently jittery. You want your animation code to tolerate variable network latency, audio chunk sizes, and occasional token stalls without resetting the character every time the model pauses for a beat.

Streaming lip sync: what to synchronize, and what not to

For realtime avatars, synchronization is usually audio-first. The mouth pose should be driven by the audio that is actually being played, not by the transcript text or by an estimated “speech started” timestamp from the LLM. Transcripts are useful for semantics; they are not a reliable animation clock.

In practice, you want the following sequence:

1. The agent decides to speak.

2. Audio begins streaming as soon as possible.

3. The avatar receives animation updates tied to the same audio session.

4. Unreal renders the avatar using the current mouth state and other speaking cues.

That also implies a few constraints:

• Do not buffer too aggressively. Large audio buffers make lip sync look late, even if the speech sounds clean.

• Use the playback clock, not wall clock. If playback drifts, the animation should follow the actual audio position.

• Handle interruptions explicitly. If the user barges in, stop or fade the current speech and reset the avatar state in the same turn.

• Expect partial failure. If the avatar stream drops briefly, keep the conversation alive and reconnect the media path without losing session state.

Implementing the triage loop in Unreal

In Unreal Engine, the cleanest pattern is to treat the avatar as a networked actor with a small state machine. A minimal state model looks like this:

• Idle: listening, no speech playback.

• Thinking: user has spoken, agent is composing a response.

• Speaking: audio playback active, lip sync enabled.

• Interrupted: user barged in, current utterance cancelled.

The animation side can then drive blend shapes or pose curves from a normalized phoneme/viseme stream. If your avatar service provides higher-level mouth positions, consume those directly. If it exposes only audio-driven sync, let the service determine the mouth state and keep Unreal responsible for applying the resulting parameters to the mesh.

A practical implementation detail: keep the avatar state outside the main character blueprint if possible. A dedicated component or actor makes it easier to reconnect sessions, swap avatar definitions, and test the media pipeline without touching movement or gameplay code.

Networking and session lifecycle

For triage, session lifecycle is usually more important than avatar selection. You need a reliable way to create a session, attach a voice identity and prompt/instructions, and tear the session down when the interaction ends. The session should survive transient transport issues, but not linger forever after the user closes the app.

Use short-lived runtime sessions with explicit cleanup. In a browser or client app, the common failure mode is letting a session continue after the user navigates away. In Unreal, the equivalent is leaving the avatar actor alive while the audio device or signaling channel has already disconnected. Make teardown idempotent and call it from both the normal exit path and any watchdog timeout.

If you are managing sessions yourself, the REST flow is straightforward: create a session, pass the relevant avatar/session parameters, then stream media through that session. The exact payload fields depend on the docs, but the pattern looks like this:

curl -X POST https://api.protoface.com/<session-endpoint> \
}'

curl -X POST https://api.protoface.com/<session-endpoint> \
}'

curl -X POST https://api.protoface.com/<session-endpoint> \
}'

That kind of API shape is useful when Unreal is not the system of record. For example, you might create sessions from a backend service, then hand the client only the ephemeral connection details it needs to render the avatar.

Where Protoface fits

Protoface is the piece that handles the avatar side of this pipeline so you do not have to build custom lip-sync transport from scratch. For Unreal-centric builds, the useful mental model is: your agent owns the conversation, while the avatar session owns the realtime face and its synchronization with speech.

If you are already using a voice agent stack, the simplest integration path is often to keep your agent logic where it is and connect the avatar layer through the available developer surface. The docs are the right place for the exact session and avatar fields, but the engineering shape remains the same: create or select an avatar, open a realtime session, stream speech, and let the avatar service keep the visual mouth motion aligned with the active audio.

If you prefer Python for orchestration, the SDK is a reasonable place to start. The code below is intentionally schematic; the exact method names and payload keys are documented in the package and docs:

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

from protoface import Client

print(session.id)

For Unreal, that usually means your backend or game server owns the API key and session creation, while the client receives only the runtime token or connection metadata it needs. Do not ship long-lived secret keys in the game client.

Common pitfalls in realtime avatar triage

1. Animating from text instead of audio. Text is semantically useful, but if you drive the mouth from tokens, you will see visible mismatch whenever the model pauses, revises, or streams unevenly.

2. Overusing frame-perfect sync. You do not need every jaw pose to line up with every waveform peak. You need believable coarse alignment at conversational latency.

3. Letting UI and media state diverge. If the UI says the agent is speaking but audio has already stopped, users notice. Treat audio playback as the source of truth.

4. Ignoring barge-in. Triage is interruption-heavy. A user saying “wait, this is about billing” should immediately stop the current utterance and reschedule the agent state.

5. Mixing session ownership across layers. Pick one place to create, refresh, and end sessions. Otherwise reconnect logic becomes impossible to reason about.

Conclusion

A good conversational triage agent is mostly an exercise in realtime systems design: stream audio early, synchronize the face to what is actually playing, and keep the conversation state separate from the render loop. Unreal Engine can present the experience cleanly, but it should not be responsible for inventing lip-sync timing or managing voice-agent state transitions on its own.

If you want to implement this without building the avatar pipeline yourself, start with the Protoface docs and adapt the session flow to your Unreal architecture. From there, add the triage logic, barge-in handling, and cleanup semantics that match your product. A small, reliable media loop will beat a flashy but desynced avatar every time.