vocence/ultravad

Audio-native end-of-turn detection pod for Vocence voice agents, built on fixie-ai/ultraVAD (an 8B Ultravox model that fuses the recent dialog text with the user's audio and returns P(<|eot_id|>) — the probability the user has finished their turn).

Follows the same Vocence pod shape and WebSocket contract as vocence/turn-detection (the /v1/smart-turn audio endpoint in particular): X-API-Key auth, auth-free /healthz + /metrics, start → ready → probability/end_of_turn → close lifecycle.

Endpoints

Endpoint	Auth	Purpose
GET /healthz	none	warming/ok status + model loaded/progress
GET /metrics	none	Prometheus metrics
WS /v1/ultravad	X-API-Key	streaming audio + dialog context → EOU events
POST /v1/ultravad/batch	X-API-Key	one-shot scoring (testing)

Run

PYTHONPATH=src ULTRAVAD_API_KEY=dev uvicorn ultravad.server:app --host 0.0.0.0 --port 8121

WS protocol (/v1/ultravad)

Mirrors turn-detection's /v1/smart-turn — client sends start first, server replies ready (not the reverse):

1. Client opens WS with X-API-Key header (bad/missing → close 4401).
2. Client → {"type":"start","sample_rate":16000,"encoding":"pcm_s16le","history":[...],"emit_every_ms":200} (bad fields → error event + close 1003).
3. Server → {"type":"ready","session_id","model","sample_rate"}.
4. Client streams binary PCM frames (16k mono s16le, 80–1600 samples; larger → close 4413).
5. Server → {"type":"probability","p_end_of_turn":0.x} every emit_every_ms.
6. Server → {"type":"end_of_turn","p_end_of_turn":0.x} once when p crosses threshold_fire; re-arms below threshold_reset.
7. {"type":"reset"} clears the turn buffer; {"type":"context","history":[...]} updates dialog history; {"type":"close"} (either side) ends → close 1000. At capacity → close 4429; model crash → 1011; shutdown → 1001.

Config (env)

Var	Default	Effect
ULTRAVAD_API_KEY	(required)	shared bearer; refuses to start without it
ULTRAVAD_PORT	8121	TCP port
ULTRAVAD_MAX_CONCURRENT	16	concurrent WS cap (over → 4429)
ULTRAVAD_MODEL	fixie-ai/ultraVAD	HF repo id
ULTRAVAD_THRESHOLD_FIRE	0.5	p to fire end_of_turn
ULTRAVAD_THRESHOLD_RESET	0.2	p below which it re-arms
ULTRAVAD_WINDOW_S	16	max audio (s) scored per turn
ULTRAVAD_LOG_LEVEL	info	logging level

GPU is pinned by the dashboard via --gpus '"device=N"'; inside the container the model uses cuda:0.

Weights

Downloaded on first start into HuggingFace's cache (HF_HOME=/cache/hf, the shared ops_hf_cache volume every Vocence pod mounts) — not baked into the image. fixie-ai/ultraVAD (1.3 GB projector) pulls a separate 8B base LM (fixie-ai/turntaking-pretraining-it-multilingual-3c, ~15 GB), which is too large to bake on GitHub-hosted runners. First cold start ≈ 4–6 min download + load; restarts hit the cache (~60 s). /healthz reports warming until the model is on GPU, then ok.