[ICML 2025 Spotlight] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
-
Updated
May 1, 2025 - Python
#
cpu-offload
Here are 17 public repositories matching this topic...
An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU (DAC ‘26)
-
Updated
Sep 29, 2026 - Python
-
Updated
Jul 9, 2026 - PHP
CPU-side action routing, context compression, and causal memory for AI agents — matches an LLM-everything agent at 58% fewer LLM calls and ~45% lower cost. Glues busyBee-cpu, honey-comb, and rust-brain.
agent mcp nvidia grace jetson honeycomb edge-ai cpu-offload busybee llm context-compression llm-cost agent-memory action-routing rust-brain
-
Updated
Sep 28, 2026 - Python
-
Updated
Sep 23, 2026 - PHP
Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs
-
Updated
Sep 29, 2026 - Python
DACAN (Дацан): Qwen3.8-Flash-Next (125B MoE) on any NVIDIA RTX 20/30/40/50 card and any x86-64 CPU with AVX2, AVX-512 not required. One or two cards, one or two sockets, context up to 524K.
cuda inference nvidia moe avx2 numa avx512 turing ampere ada-lovelace cpu-offload blackwell llm local-llm qwen qwen3 nvfp4
-
Updated
Oct 6, 2026 - C++
Run a closure in a child process and return the result over a promise
-
Updated
Jun 27, 2026 - Makefile
Run closures in a child process messenger or pool
-
Updated
Jul 16, 2026 - Makefile
[arXiv] ColA: Collaborative Adaptation with Gradient Learning
-
Updated
Apr 23, 2024 - Python
QSA HiSparse for SGLang: 256K KV offload, CUDA Graph benchmarks, and patches tested on dual RTX 4090 48GB
qsa cpu-offload tensor-parallelism sparse-attention long-context fp8 gpu-inference llm-inference qwen sglang qwen3 rtx-4090 cuda-graphs sm89 256k-context hisparse rtx-4090-48gb 4090-48gb dual-rtx-4090 kv-cache-offloading
-
Updated
Oct 5, 2026 - Python
Measurement-first research on local Mixture-of-Experts inference under a hardware contract. 3 measured laws, 4 falsified ideas, and paper site.
reproducibility google-colab mixture-of-experts cpu-offload negative-results kv-cache llama-cpp llm-inference
-
Updated
Aug 14, 2026 - Jupyter Notebook
Engine for two MoE models only - DeepSeek-V4.1-Flash (main) and GLM-5.3-Flash - on one 96 GB GPU, experts offloaded to CPU RAM, 262K context. Pieced together from what we had (DDR4, PCIe 4); DDR5 would do better. Sleeps/wakes in seconds to share the GPU. OpenAI/Anthropic API, works behind LiteLLM.
cuda moe glm inference-engine mixture-of-experts cpu-offload single-gpu llm-inference litellm deepseek openai-compatible
-
Updated
Oct 7, 2026 - C++
Run DeepSeek-V4.1-Flash on 8x RTX 5090 + 503 GiB RAM. 5.6x output throughput vs patched eager mode, with vLLM fixes, CPU offload & reproducible benchmarks.
cuda benchmarks quantization inference-optimization mixture-of-experts cpu-offload blackwell sparse-attention vllm llm-inference deepseek flashinfer rtx-5090 sm120 cuda-graphs deepseek-v4-1-flash
-
Updated
Sep 16, 2026 - Python
Recall-aware tiered KV cache engine for long-context local LLM inference.
-
Updated
May 19, 2026 - Python
DeepSeek-V4.1-Flash at TP3 on 3x RTX PRO 6000 (SM120): EXL3 vs official checkpoint + UVA offload, same-box benchmarks, failed attempts, reproducible configs.
benchmark uva quantization cpu-offload blackwell tensor-parallelism vllm llm-inference deepseek sm120 rtx-pro-6000 exl3 expert-offload
-
Updated
Oct 7, 2026 - Python
Add this topic to your repo
To associate your repository with the cpu-offload topic, visit your repo's landing page and select "manage topics."