MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert cache with LRU hot experts, OpenAI-compatible proxy, GGUF multi-shard.
-
Updated
Sep 12, 2026 - Python
#
expert-caching
Here are 4 public repositories matching this topic...
Run AI models too large for your Mac's memory — at near-full speed. Intelligent expert caching, speculative execution, and 15+ research techniques for MoE inference on Apple Silicon.
python macos rust machine-learning inference moe quantization mlx speculative-execution mixture-of-experts memory-optimization apple-silicon llm metal-gpu ssd-streaming expert-caching
-
Updated
May 11, 2026 - Python
Flash-backed mixture-of-experts inference on Apple devices with Swift/MLX.
swift ios moe quantization memory-efficient mlx mixture-of-experts edge-ai on-device-ai mobile-inference flash-storage apple-silicon llm-inference qwen mlx-swift on-device-llm expert-caching model-offloading
-
Updated
Sep 24, 2026 - Swift
Run large MLX models on Apple Silicon with flash weight streaming, using native precision beyond RAM limits
menubar clustering optimization transformer quantization speculative-execution swiftui bpe memory-optimization apple-silicon bpe-encoder llm-training ssd-streaming weight-streaming expert-caching
-
Updated
Sep 25, 2026 - Python
Add this topic to your repo
To associate your repository with the expert-caching topic, visit your repo's landing page and select "manage topics."