Guide to Streaming a Realtime HR Screening Agent in SvelteKit
Build a realtime HR screening agent in SvelteKit with streamed mic audio, secure server-side sessions, and synchronized avatar playback.
Introduction
If you are building a realtime HR screening agent in SvelteKit, the core problem is not “how do I get an LLM to answer questions?” It is how to keep the experience responsive while the user is speaking, manage audio and video streaming correctly, and avoid turning your frontend into a bucket of mutable session state.
By the end of this post, you should be able to structure a SvelteKit app that starts a realtime interview session, streams microphone audio to your agent, plays back low-latency responses, and renders a synchronized avatar face without exposing secrets in the browser.
For the avatar layer, Protoface gives you a developer-facing realtime avatar API that fits into voice-agent architectures rather than fighting them. The important part is not the branding; it is the separation of concerns: your app owns the interview UX, your agent owns conversation state, and the avatar service owns the talking face.
What “realtime” actually means here
In practice, “realtime HR screening” usually means a duplex audio session with optional video rendering of the agent. The browser captures microphone audio, streams it to a backend agent over WebRTC or another low-latency transport, and receives synthesized audio back as soon as the agent starts generating a response. The avatar is tied to that same response stream so the lips match the voice closely enough to feel coherent.
Two implementation details matter:
Turn-taking: you need a clean way to detect when the candidate starts and stops speaking so the agent does not talk over them.
Session affinity: the avatar, audio pipeline, and conversation context all need to belong to the same session, not a stateless HTTP request.
That means you generally do not “poll for answers.” You establish a session, stream media, and keep a persistent state machine on the server side.
SvelteKit architecture for a screening flow
A practical SvelteKit setup usually has three pieces:
Client page: handles permissions, audio capture, and rendering the agent UI.
SvelteKit server routes: create sessions, mint any short-lived tokens, and keep API keys out of the browser.
Agent backend: runs the actual interview logic, conversation memory, and optional avatar integration.
The browser should know as little as possible. In particular, do not ship your realtime vendor API key to the frontend. If you need to create a session from the browser, have SvelteKit call your own server route first, and have that route talk to the external API.
For a screening agent, I would keep the frontend focused on UX: consent, device selection, transcript display, and a visible “listening / thinking / speaking” state. Everything else belongs on the server.
Streaming audio from the browser
In SvelteKit, the audio capture path is straightforward but the details matter. Use getUserMedia for microphone access, then feed the stream into whatever realtime transport your backend uses. If you are using WebRTC, the browser can publish audio tracks directly. If you are using a custom websocket pipeline, you will need to encode and chunk PCM or Opus frames consistently.
For a voice interview agent, WebRTC is usually the least painful option because it already solves jitter buffering, NAT traversal, and media timing. It also lets you attach additional tracks later if you want screen share or the avatar video surface.
Client-side state should be minimal and explicit. A good rule is to model only a few session states:
idle— no permissions or no active sessionconnecting— setting up the room or transportlistening— mic live, waiting for the candidatethinking— backend has end-of-turn and is generatingspeaking— receiving synthesized response audio and avatar animationended— interview completed or terminated
That state machine is more important than the specific framework hooks. SvelteKit will happily render a polished UI around it, but the state transitions need to be deterministic or your conversation feels broken.
Server-side session creation and secure token handling
The cleanest pattern is to let SvelteKit create the realtime session server-side, then return only the minimum data needed by the client. If your backend requires a token, mint it in a +server.ts route and scope it to a single interview session with a short TTL.
Example: a minimal server route that forwards a session creation request to your own backend or API:
Two important gotchas:
Never call the external API directly from the browser with a long-lived secret.
Keep session identity stable across reconnects so the transcript and avatar state do not reset if the network blips.
If you are building a screening workflow, you probably also want rate limits and timeboxing. A candidate interview should not be able to run forever, and a broken reconnect loop should not create a surprise bill.
What the avatar layer changes in a voice agent
The avatar is not just decoration. It changes your latency budget and your session model. A talking face needs to be synchronized to the same turn boundaries as the audio. If the mouth animation starts too early, the system feels uncanny; if it starts too late, the face looks disconnected from the voice.
From an engineering perspective, the avatar service should consume the agent’s speaking events rather than infer speech from browser audio. That keeps the animation aligned with the authoritative agent output. In other words: the browser handles capture and playback, the agent owns the conversational turn, and the avatar renders the speaking state for that turn.
That is where a dedicated avatar API is useful. You want an integration that plugs into the agent runtime, not a separate frontend animation layer that tries to guess what the model is doing.
Using the LiveKit agent path for synchronized video
If your voice agent already runs on LiveKit, the lowest-friction path is the LiveKit Agents plugin from the Protoface ecosystem. It attaches a realtime face to the agent so the avatar stays synchronized with the spoken response. The plugin is packaged for Python, which makes it easy to drop into an existing agent process without rewriting your media stack.
A sketch of the shape looks like this:
The value here is architectural, not cosmetic. Your agent keeps its existing conversation logic, but it emits an additional synchronized video surface. That is the right place to add a face if you are already invested in LiveKit.
If you want the concrete package and examples, the plugin repo is the place to start: https://github.com/protoface-ai/protoface-plugin-pipecat is for the Pipecat integration, and the LiveKit-related quickstarts are linked from the main developer README. For setup details, stick to the docs rather than guessing at field names.
Practical SvelteKit UI concerns
Once the streaming path is working, the remaining work is mostly product engineering:
Permission UX: ask for mic access before connecting, and explain why.
Reconnect behavior: a dropped network should resume the session, not restart the interview.
Transcript rendering: show partial transcriptions separately from final messages.
End-of-interview handling: preserve the summary and scoring state after the session ends.
One subtle issue is backpressure. If the frontend cannot render or play audio as fast as the backend emits it, you need a strategy for dropping, buffering, or resynchronizing media frames. For interviews, a small controlled buffer is usually better than aggressive low-latency tricks that produce choppy playback.
Another issue is privacy. HR flows often involve sensitive candidate data, so keep logs intentionally sparse. Store only what you need for evaluation, and separate operational telemetry from interview content.
Where Protoface fits in this workflow
In this specific architecture, Protoface is useful when you want the agent to have a synchronized face without building and maintaining a custom lip-sync pipeline yourself. The REST API and SDK are the right surfaces if your SvelteKit app or backend needs to create sessions programmatically, while the avatar integration belongs on the agent side where the audio turn events already exist.
If you need a quick way to validate the flow end to end, use the developer docs and one of the quickstarts to confirm the session lifecycle, then wire that into your SvelteKit server route. The important part is to keep the API key on the server and treat the browser as a media client, not a control plane.
For programmatic access from Python, the SDK follows the same pattern: create or manage avatars and sessions server-side, then return only the data the client needs.
Conclusion
A realtime HR screening agent in SvelteKit is mostly an exercise in correct media and session architecture. Keep audio streaming persistent, keep state explicit, keep secrets server-side, and make sure the avatar is tied to the agent’s actual speaking turn rather than a separate animation guess.
If you already have a LiveKit or Python-based agent, the remaining work is mostly integration glue. If you are starting from scratch, read through the docs, then build the smallest possible loop: session creation, microphone streaming, response playback, and a synchronized face. Once that works reliably, you can layer on scoring, transcripts, analytics, and the rest of the screening workflow.