Windows single-GPU serving: WDDM pin-budget, split expert residency, tiered KV storage — design notes + reference branch

[brz6699]

Problem

We run FreeToken on a single consumer GPU (RTX 5090 32 GB) with lots of system RAM (~250 GB), serving large MoE checkpoints with long context and multimodal history. On that setup we hit four concrete issues with the upstream path:

1. Windows specifics: the POSIX-only resource module is used for page locking, and expandable_segments / zmq event-loop behavior differs under the Windows Proactor loop — startup degraded or failed on plain Windows.
2. WDDM pin ceiling: the driver caps GPU-accessible pinned host memory at roughly half of physical RAM, so when expert weights exceed that budget, the naive "pin everything" strategy OOMs on the last bank.
3. KV pool size vs. context length: at bf16 (2048 B/token/KV-layer), a 1M-token paged pool alone eats ~13+ GiB of VRAM, starving the MoE expert cache.
4. Repeated multimodal content: without content-keyed prefix reuse, the same image/video is re-prefilled on every request.

Proposed approach

What worked for us, in one causal chain — tiered KV storage frees GPU room for the expert LRU; a larger expert cache keeps more hot experts on-GPU; content-keyed mm reuse avoids re-prefill:

• Windows compat: ctypes-based VirtualLock replacing the resource module; working-set/pin-budget handling under WDDM; graceful degradation paths for expandable_segments and the Proactor loop differences.
• Split residency for expert banks: when the pin budget is smaller than the expert weights, head/tail MoE layers stay OS-locked in RAM and decode on a multi-threaded CPU executor (AVX-512 BF16 + VNNI, NVFP4 W4A8), remaining banks are pinned for GPU streaming. Budget tunable via FREETOKEN_PIN_BUDGET_GB.
• Storage-side KV quantization tiers for the paged pools (QSA / MLA-DSA families), measured round-trip against bf16 reference attention (max-cosine over sparse + split-k paths): bf16 2048 B/tok (cos 1.0000) / fp8_e4m3 1024 B/tok (cos 0.9994) / turbo4 ~516 B/tok (cos 0.9887). Storage is compressed; attention compute stays bf16; indexer keys always bf16.
• Vision behind an explicit --vision-on flag with content-hashed prefix keys (same image → KV reuse, different image → zero false hits) and an mm RAM tier that swaps against the GPU pool.

Measured on RTX 5090 + 254 GiB RAM (Qwen3.8-Flash-Next-NVFP4): aggregate throughput peaks at concurrency 4 (~67 tok/s, +155% over serial; ceiling is PCIe-bandwidth-bound, not compute); MoE cache 1024→3072 lifts long-text decode +36% with zero quality loss; needle retrieval 6/6 exact at 322k/450k tokens; image requests cost only +8 ms over text.

Reference implementation

All of the above is implemented and gated by a verification/ regression suite on our branch:

https://github.com/brz6699/freetoken-win-qwen3.8-flash-next (windows-build branch, based on upstream 58f4b9e / release 0.1.2+g816c324d0)

Happy to upstream the pieces incrementally — suggestions on how you'd like them sliced would be welcome.