llm-d

Helm charts for llm-d

Self-hosted, distributed, multi-model LLM inference

Kubernetes controllers for fast model actuation using vLLM sleep/wake and launcher-based model swapping

Distributed inference configuration tools

Accelerate reproducible inference experiments for large language models with LLM-D! This lab automates the setup of a complete evaluation environment on OpenShift/OKD: GPU worker pools, core operators, observability, traffic control, and ready-to-run example workloads.

Demo - Distributed Inference on OpenShift (aka llm-d)

Experimental recovery coordination for expert-parallel inference groups in llm-d.

Stack to serve LLM inference based on llm-d, Envoy AI gateway

一个 Kubernetes 原生的高性能分布式大语言模型推理框架

Prove which open model can replace your closed one — on your traffic, your hardware, your budget. Then move you there without a risky cutover.

Step-by-step llm-d deployment on GKE with GPU acceleration

A light weight vLLM simulator, for mocking out replicas.

Self-hosted LLM inference platform on AWS EKS — vLLM, llm-d, Cosign, ArgoCD, Kyverno, Kagent+MCP

RecursiveCharacterTextSplitter and context cache with llm-d

Open Source software code for use with PCIe card-based hardware AI accelerators catering to both inference and training use cases

LLM inference stack on Kubernetes: Envoy + EPP router + vLLM on minikube, CPU-only — benchmarked against a naive HF server

llm-d benchmark scripts and tooling

End-to-end Bielik model implementation on RHOAI with llm-d

TOA vLLM/LLM-D Hackathon — attendee guide, Brev Launchable configs, and GPU cost estimates

Verifiable, tenant-isolated LLM inference controls for regulated AI infrastructure.