den_llama.cpp

A llama.cpp fork tuned for one job: Qwen3.8-27B at up to 196K context with native MTP speculative decoding on a single 16 GB GPU (RTX 5070 Ti, Blackwell GB203) — plus a MoE serving lane (gemma-4-26B-A4B) that runs on the same card via partial CPU+GPU expert offload. This repo hosts a frozen reference engine (build2) and the development engine (build3).

Part of Project Den — a framework for persistent AI companions; this repo is its neural execution engine.

Builds

Tree	Role
build2 (abd589d4c)	Frozen reference engine. Read-only.
build3 (build3 branch)	Development engine — build2 plus the FA fix, the quantized-KV codecs, SER, the MoE placement fix, and the KV headroom guard.
build-fa	Dev twin of build3 (I:\llama-fa).

What this fork is

• Qwen3.8 GDN-hybrid focus. Built around the Qwen3.8-27B hybrid architecture (gated-delta-net + attention). This tree carries a delta-net conv-state snapshot bound fix (src/models/delta-net-base.cpp) and the Qwen3.5/3.8 model path (src/models/qwen35.cpp, src/models/qwen35moe.cpp).
• Quantized KV cache — the build3 advance. GGML_TYPE_NVFP4 as a KV type (matched + mixed pairs, KLD 0.0151 vs q4_0) and a new GGML_TYPE_Q4_G64 KV codec (signed INT4 + one FP16 scale per 64 = 4.25 bpv; unit-verified byte/bit-exact quantizer + dequantizer; KLD 0.0163). Plus layer-wise split-KV (--kv-high-precision-layers N: the first N full-attention layers keep -ctk/-ctv, the rest drop to NVFP4) and adaptive KV streaming (see below).
• KV headroom guard. --kv-headroom-warn/-block/-force report the projected free device memory at context creation and surface the VRAM-ceiling cliff (a silent prefill collapse today), naming the maximum safe context.
• Smart Expert Reduction (SER). -ser MIN,THRESH + --ser-top-p + --ser-ramp: a dense-sanitized expert-reduction tool (correct + perf-neutral; a quality/allocation instrument, not a speed lever). Root-caused a CUDA fault inherited from the reference implementation (the mm_ids_helper density invariant).
• MoE expert placement. --fit on (without -ngl) auto-places experts in the fast zone on a 16 GB card after a MoE-aware margin fix; partial CPU+GPU expert offload overlaps and beats full residency by ~8× on a 26B-A4B.
• Native MTP / NextN head work. The pinned golden master adds an MTP carryover-reset fix (common/speculative.cpp: zero stale pending_h/verify_h on a new prompt) plus server-side MTP hidden-state plumbing (--embeddings-nextn, llama_set_embeddings_nextn(), per-token h_nextn emitted in the non-OAI embeddings JSON).
• FlashAttention quant-pair enablement. A 2-file fix (CMakeLists.txt + fattn.cu, fa_fix.diff, +4/−1) compiles the q8_0-q4_0 FA instance. With GGML_CUDA_FA_ALL_QUANTS=OFF — the golden build cache value — the live -ctk q8_0 -ctv q4_0 pair is not compiled and FA silently falls back to f16-f16. The fix lives in a separate build tree (same pinned commit + 2-file FA fix), not in this repo's pinned commit; see Results.
• IQ3_XXS RCO quant pipeline. Ships a 9.75 GiB IQ3_XXS trunk (~3.06 bpw) with a Q6_K MTP draft head, packed from ISTA-DASLab's published per-tensor RCO allocation with a custom OrcaRouter-native importance matrix.
• Gated benchmark suite. Nothing lands on main without passing the repo's gates: needle retrieval, toolcall-v2, coherence, and MTP-ladder speed, with medians-of-N and no-regression checks.
• Dreya serving target. The production consumer is a local long-context assistant served on one 16 GB GPU; the adaptive KV streaming path below is what makes 196K fit.

Results

Only golden-build (build2) numbers with recorded conditions are listed. FA-quant-pair numbers are explicitly labeled experiment (build-fa, not golden). Numbers whose conditions are not fully pinned (GPQA-Diamond, toolcall) are intentionally omitted — see the model repo for those.

Metric	Value	Conditions
Needle retrieval	6/6	golden build2; Qwen3.8-27B-OrcaRouter-GSQ-RCO-IQ3_XXS-v2.0.gguf; 15K-word haystack, depths 0.1–0.9, temp 0.0.¹
WikiText-2 perplexity	6.1745 ± 0.14889	golden build2 llama-perplexity.exe; ship file; 40×512 chunks; -fa on; KV default (no -ctk/-ctv passed).²
Serve t/s @ 32K ctx	82.73 t/s (median; 79.4–89.6)	golden build2; ctx 32768; 5 prompts × 300 tok × 2 reps; temp 0.6; seed 424242; np=1; MTP n-max 2; accept 0.6703; prefill 706 t/s.³
Serve t/s @ 65K ctx	64.16 t/s (median ALL; 51.11–72.48)	golden build2 (FA A/B control); ship file; ctx 65536; -b 2048 -ub 2048 -fa on -ctk q8_0 -ctv q4_0 -ngl 99 -t 8 -tb 8; MTP n-max 2; median accept ALL 0.7675 / CUM 0.7010.⁴
FA build — t/s	79.48 t/s (median)	experiment (build-fa, not golden): HEAD abd589d4c + fa_fix.diff; same flags as the 65K control; +23.9% vs control.⁵
FA build — acceptance	0.7018 ALL / 0.6768 CUM	experiment (build-fa, not golden); control 0.7675 / 0.7010.⁵
FA build — ppl / needle	6.1745 ± 0.14889 / 6/6	experiment (build-fa, not golden); ppl Δ 0.00% vs control; control needle 5/6 (needle-05 miss).⁵

build3 (working engine) — additions

Metric	Value	Conditions
NVFP4 KV — decode / KLD	88.6 t/s / 0.0151	build3; matched -ctk nvfp4 -ctv nvfp4; vs q4_0-KV control (KLD 0.104 bound); ~1 GiB KV saved @131K.⁶
Q4_G64 KV — NIAH / KLD	6/6 / 0.0163	build3 feat/q4-g64-kv; matched + mixed pairs; unit harness: quantizer byte-identical vs CPU ref, dequant bit-identical.⁷
Layer-wise split-KV — NIAH	6/6	build3; --kv-high-precision-layers 8 @64K; prefill 1059 t/s.⁸
MoE 26B-A4B — decode (offload)	94–117 t/s	build3; gemma-4-26B-A4B Q4_K_M; -ncmoe 4 (full-resident collapses to 11–15 t/s at the VRAM ceiling).⁹
MoE KV×offload — decode	129 t/s	build3; -ctk q8_0 -ctv q4_0 -ncmoe 2 (+11% vs f16/ncmoe4); knife-edge — 6 MiB more VRAM (ncmoe 1) collapses to 21.5 t/s.¹⁰
KV headroom guard	verified	build3; dense 32K → 4585 MiB free (clean); dense 196K → 653 MiB free, warns + reports max safe ctx; offloaded MoE → accurate, warn-only.¹¹

⁶ NVFP4-KV validation record (docs/superpowers/specs/2026-09-24-nvfp4-kv-validated.md). ⁷ Q4_G64 record (…/2026-09-24-q4-g64-kv-type.md) + unit harness. ⁸ Split-KV record (…/layer-wise-split-kv.md, 64K needle result). ⁹ MoE placement record (…/2026-09-24-moe-expert-placement-build3.md). ¹⁰ KV×MoE record (…/2026-09-24-kv-moe-interaction.md). ¹¹ Headroom-guard commits c1a0fa815 + 9cd4febbb.

Notes from building these:

• V precision vs MTP acceptance is context-state dependent, not a fixed property. With a fixed prompt and seed on a fresh short context the figure is stable (q4_0 V 0.7563 vs q8_0 V 0.7417, identical across runs), but after a long-context ingest the direction flips (q8_0 V 0.819 vs q4_0 V 0.742). Pick V precision per deployment by measuring; q8_0 V fits ≤64K, q4_0 V for long context.
• MoE + CUDA graphs is high-risk (dynamic expert destinations break capture / replay stale pointers). Capture dense-lane graphs only, if at all.
• Thread count does not move the MoE offload path (114–117 t/s across -t 4/8/16).
• Each of these results was reproduced on the same card; numbers without recorded conditions are not listed.

¹ Model card (Hugging Face / ModelScope). ² Project perplexity log (build2 llama-perplexity.exe, 40×512-chunk run on the ship file). ³ Project A/B record: model file at run time was …v2.0-s1-orcaim.gguf, later consolidated into v2.0.gguf; short-ctx KV type was not recorded in the AB source. ⁴ FA A/B battery log (control leg, median-of-3). ⁵ Project FA A/B verdict record §2. Disposition: the pre-registered gate recorded TESTED-NEGATIVE (acceptance CUM −0.0242, exceeding the 0.02 bound) and build2 stayed golden; on 2026-09-10 an executive override promoted build-fa to >128K-regime golden, retaining build2 as the immutable ≤128K default (override record).

Models

All repos verified live via the Hugging Face and ModelScope public APIs.

Artifact	File	Size	SHA256 (prefix)
Main quant — Hugging Face · ModelScope	Qwen3.8-27B-OrcaRouter-GSQ-RCO-IQ3_XXS-v2.0.gguf	10,466,439,424 B (9.75 GiB)	41ad7dfb…
Vision projector (BF16)	mmproj/mmproj-Qwen3.8-27B-BF16.gguf	931,146,528 B (~0.87 GiB)	13cb7beb…
Vision projector (Q8_0)	mmproj/mmproj-Qwen3.8-27B-Q8_0.gguf	629,247,648 B (~0.60 GiB)	b280c2cb…
Chat template	froggeric-qwen3.8-tool-use.jinja	28,234 B	e57684ba…
RCO allocation map	REF-IQ3_XXS-mtp.rco-allocation.txt	24,583 B	2e690030…
imatrix	imatrix.dat	13,642,656 B	e0fcab28…

Base and reference repos: orcarouter/Qwen3.8-27B-Uncensored (uncensored base), Qwen/Qwen3.8-27B (architecture + pretrained weights), ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (RCO allocation source), froggeric/Qwen-Fixed-Chat-Templates (tool-use template). All hashes are from the model repo's SHA256SUMS.txt; REF-… and imatrix.dat prefixes are from the upload record.

Quick start

Golden build (Windows, CUDA 13.3 + VS2022 + Ninja)

Pin the golden master commit, then configure with GGML_CUDA_FA_ALL_QUANTS=OFF (this is the value the golden build2 was actually built with):

git clone https://github.com/RentedNoodle/den_llama.cpp.git
cd den_llama.cpp
git checkout abd589d4cf6d4d2d221b9532d3861fadbdb7b046

cmake -S . -B build2 -G Ninja `
  -DCMAKE_BUILD_TYPE=Release `
  -DGGML_CUDA=ON `
  -DGGML_CUDA_FA_ALL_QUANTS=OFF `
  -DGGML_CUDA_GRAPHS=ON `
  -DGGML_CUDA_NCCL=ON `
  -DGGML_CUDA_COMPRESSION_MODE=size `
  -DGGML_NATIVE=ON `
  -DGGML_LLAMAFILE=ON `
  -DGGML_OPENMP=ON `
  -DGGML_CPU_REPACK=ON `
  -DGGML_BUILD_EXAMPLES=OFF `
  -DGGML_BUILD_TESTS=OFF

cmake --build build2 --target llama-server --config Release -j 8

Ship command (RTX 5070 Ti 16 GB)

build2\bin\llama-server.exe `
  -m Qwen3.8-27B-OrcaRouter-GSQ-RCO-IQ3_XXS-v2.0.gguf `
  --spec-type draft-mtp --spec-draft-n-max 2 `
  --ctx-size 262144 -fa on -ctk q8_0 -ctv q4_0 -ngl all `
  -b 512 -ub 512 -np 1 --kv-stream-stage-mib 2048 `
  -t 8 -tb 8 --cache-ram 8192 --reasoning-budget 256

For depth-0.5 retrieval fidelity, use -b 2048 -ub 2048 (a documented chunking artifact of the model, not this fork).

Build3 (working engine) — same command, plus the new flags

build3 is source-compatible with the ship command above; it adds:

# quantized KV + split-KV (memory / capacity levers)
-ctk q8_0 -ctv nvfp4                       # NVFP4 KV (validated <=96K; KLD 0.0151)
-ctk q8_0 -ctv q4_g64                      # Q4_G64 KV (4.25 bpv, KLD 0.0163)
--kv-high-precision-layers 8               # first 8 full-attn layers keep -ctk/-ctv, rest NVFP4

# the KV headroom guard (surfaces the VRAM-ceiling cliff and the max safe ctx)
--kv-headroom-warn 1536                    # default; warns when projected free < this
--kv-headroom-block 512                    # opt-in refusal (0 = off, warn-only)
--kv-headroom-force                        # continue anyway (warn only)

# MoE lane (gemma-4-26B-A4B) — partial expert offload; do NOT add --fit together with -ngl
-ngl 999 -ncmoe 4                          # ~94-117 t/s; full-resident collapses at the VRAM ceiling

Build it the same way (or use the snapshot binaries under DEN_CANONICAL/engine/build3_snapshots/2026-09-24/bin), and read the KV headroom report at load with -lv 4.

What's in the fork

Location	What
common/speculative.cpp	MTP carryover-reset on a new prompt (zeroes stale pending_h/verify_h), plus an env-gated (LLAMA_DUMP_MTP) debug dump.
common/arg.cpp, common/common.h, common/common.cpp	--embeddings-nextn flag and common_params::embeddings_nextn.
include/llama.h, src/llama-context.cpp	llama_set_embeddings_nextn() / llama_get_embeddings_nextn_ith(); NULL-safe wrapper and env-gated MTP tensor dump.
tools/server/server-task.{h,cpp}, tools/server/server-context.cpp	Server emits per-token MTP head input hidden as "nextn" when embeddings_nextn is enabled.
src/models/delta-net-base.cpp	Gated-delta-net conv-state snapshot bound fix (K = min(n_rs_seq, n_tok)+1).
benchmarks/	Adaptive KV streaming benchmark driver (benchmark_kv_stream.py) — sweeps context capacity 8K→max, auto-selects the largest practical KV pool, and emits CSV/PNG/SVG. See benchmarks/README.md.
BUILD2_GOLDEN.md	Rebuild/repro record for the golden engine (toolchain, configure command, patch stack). Note: its "Source of truth" commit field names 0875eab41, one commit behind the current golden HEAD abd589d4c.
FA quant-pair work	Not in this repo's pinned commit: same HEAD abd589d4c + 2-file FA fix (fa_fix.diff, +4/−1), maintained in a separate build tree. See Results.
ggml/src/ggml-cuda/mmid.cu, mmq.cu, mmf.cu, mmvq.cu, quantize.cu	SER v2 — dense-sanitized expert reduction; mm_ids_helper density-invariant fix (the reference implementation's CUDA follow-up never landed).
ggml/src/ggml-quants.c, ggml-common.h, ggml-cuda/convert.cu, fattn-common.cuh, fattn.cu, cpy-utils.cuh, set-rows.cu + template-instances/fattn-vec-instance-q4_g64-*.cu	GGML_TYPE_Q4_G64 KV codec (INT4 + FP16 scale per 64).
src/llama-kv-cache.{h,cpp}, src/llama-context.cpp, src/llama-graph.cpp	Layer-wise split-KV (--kv-high-precision-layers) + the KV headroom guard + SER plumbing.
common/fit.cpp	MoE-aware margin cap — --fit auto-placement lands in the fast zone.

Branches:

Branch	Role
main	Golden master HEAD abd589d4c ("MTP carryover fix + server MTP/reasoning plumbing + delta-net fixes + build repro doc").
build3	Development engine (a22b5fea0) — FA fix + NVFP4/Q4_G64 KV + split-KV + SER + --fit MoE fix + KV headroom guard. Tag build3-golden-20260924.
feat/q4-g64-kv	Build3 + the GGML_TYPE_Q4_G64 codec + the KV headroom guard (9cd4febbb).
archive/stage1-nvfp4-abd589d4c	Pre-consolidation archive (42bf457d5).
den-legacy	Retired engine history (a99ff8f9…).

Adaptive KV streaming (contributed)

This branch adds an experimental, block-granular KV cache streaming path to the CUDA llama-server. It is intended for running long contexts when model weights leave too little VRAM for the complete KV cache.

With --kv-stream-stage-mib N, the authoritative KV tensors are stored in pinned host memory while a bounded CUDA pool is shared by resident KV pages and a transfer ring. The runtime adapts that split as the context grows: it keeps as many pages resident as the budget allows, reclaims resident space for staging when more streaming is required, and prefetches later layers while the current layer computes. This avoids relying on uncontrolled Unified Memory page thrashing and preserves exact attention over the full context.

Detailed project story, design, implementation, and benchmark results are in Running Qwen 27B on 16G VRAM with Full Context Length: Building Adaptive KV Cache Streaming for llama.cpp.

Warning

This is research code tailored to our current NVIDIA CUDA configuration: an RTX 5070 Ti with 16 GB VRAM, unsloth/Qwen3.8-27B-GGUF UD-Q3_K_XL, a 262144-token context, Flash Attention, a Q8_0 K cache, a Q4_0 V cache, and one server slot. Other models, KV cache quantization combinations, parallel slots, and non-CUDA backends are not yet supported or validated. Expanding model and KV quantization support is follow-up work.

Build the modified server

Install a C++ compiler, CMake, and the CUDA toolkit, then run this command from the repository root:

cmake -S . -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build --config Release --target llama-server -j

The executable is created at build/bin/llama-server.

Example using the tested cache configuration:

./build/bin/llama-server \
  --model /path/to/model.gguf \
  --ctx-size 262144 \
  -fa on \
  -ctk q8_0 \
  -ctv q4_0 \
  -ngl all \
  -np 1 \
  --kv-stream-stage-mib 2304

The best value for --kv-stream-stage-mib depends on the model, context capacity, GPU, and other VRAM consumers. Start conservatively and increase it while checking startup and peak VRAM use.

Optional Unified Memory for model weights

Adaptive KV streaming works with or without Unified Memory. Leave GGML_CUDA_ENABLE_UNIFIED_MEMORY unset for ordinary CUDA device allocations. To make GPU-offloaded model buffers CUDA managed allocations, launch the same server with the environment variable enabled:

GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 \
./build/bin/llama-server \
  --model /path/to/model.gguf \
  --ctx-size 262144 \
  -fa on \
  -ctk q8_0 \
  -ctv q4_0 \
  -ngl all \
  -np 1 \
  --kv-stream-stage-mib 2304

With this flag, CUDA-backed model buffers, including GPU-offloaded weights, are allocated with cudaMallocManaged and their pages can migrate between VRAM and host memory. The adaptive resident-page and transfer-ring pool is intentionally different: it is still allocated with cudaMalloc, so that fixed-size pool remains physically allocated in VRAM instead of becoming managed memory. UVM is therefore optional for this branch and does not change the KV streaming pool into pageable storage.

Recreate the benchmark graph

The benchmark driver automatically selects the largest practical adaptive KV pool for each configured context capacity, sweeps from 8K through the requested maximum, and generates the CSV, PNG, and SVG results:

python3 -m pip install matplotlib

python3 benchmarks/benchmark_kv_stream.py \
  --model /path/to/model.gguf \
  --max-context 192K

The only required arguments are the model GGUF and maximum context. See benchmarks/README.md for the pool-probing algorithm, generated files, optional settings, and resumable output directories.

---

This fork builds on ggml-org/llama.cpp (MIT). The upstream README is included unmodified below for attribution.

Upstream llama.cpp README (unmodified)

Upstream llama.cpp README

llama.cpp

LLM inference in C/C++

ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API

Quick start

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

VLM session with llama cli

Built-in web UI against llama serve

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Backend	Target devices
BLAS	All
BLIS	All
CANN	Ascend NPU
CUDA	Nvidia GPU
HIP	AMD GPU
Hexagon [In Progress]	Snapdragon
IBM zDNN	IBM Z & LinuxONE
MUSA	Moore Threads GPU
Metal	Apple Silicon
OpenCL	Adreno GPU
OpenVINO [In Progress]	Intel CPUs, GPUs, and NPUs
RPC	All
SYCL	Intel GPU
VirtGPU	VirtGPU APIR
Vulkan	GPU
WebGPU	All
ZenDNN	AMD CPU

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models
• Release process

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
• sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain