Add MIG support:"mixed strategy"
Add support for kubernetes v1.22+
Bugs fixed
a pod can be scheduled to a node where its core usage is 100 - Fixed
cudevshr.cache can't be modified with non-root users - Fixed
Add custom resource name
A task with cores=100 will allocate all device memory(virtual device memory excluded)
nvidia.com/gpucores will limit the GPU utilization inside container Prior than v1.0.1.3, nvidia.com/gpucores will not limit utilization inside container, we have fixed it in v1.0.1.3
Add nvidia.com/gpumem-percentage resource name This resource indicates the device memory percentage of GPU, can not be used with "nvidia.com/gpumem". If you want an exclusive GPU, specify both the "nvidia.com/gpucores" and "nvidia.com/gpumem-percentage" to 100
Add GPU type specification You can set "nvidia.com/use-gputype" annotation to specify which type of GPU to use. "nvidia.com/nouse-gputype" annotation to specify which type of GPU to avoid.
Fix an monitor "desc not found" error
Add "devicePlugin.sockPath" parameter to set the location of vgpu.sock
Major Update: Device Memory will be counted more accurately several device memory usage, including cuda context, modules, parameters, reserved addresses will be counted in v1.1.0.0
Update to be compatible with CUDA 11.6 and Driver 500+
Rework monitor strategy Monitor will mmap control file into address space instead of reading it in each query.
Fix segmentation fault when invoking cuMallocAsync
Core Utilization Oversubscribe and priority-base scheduling Currently we have two priority, 0 for high and 1 for low. The core utilization of high priority task won't be limited to resourceCores unless sharing GPU node with other high priority tasks. The core utilization of low priority task won't be limited to resourceCores if no other tasks sharing its GPU. See examples/nvidia/example.yaml for more details
Add Container Core Utilization policy See details in Configure HAMi
Update device memory counting mechanism to compat with CUDA 11.3+ task sometimes vgpu-scheduler won't able to collect device memory usage when running cuda11.3+ compiled tasks in v1.x version of vgpu-scheduler. We solve this problem by reworking device memory counting mechanism.
Use node annotation instead of grpc to communicate between scheduler and device-plugin In v1.x version of vgpu-scheduler, we use grpc to communicate between scheduler and device-plugin, but we reimplement this communication in v2.x by using node annotation, to make it more stable and readable.
modified nvidia-container-runtime is no longer needed We remove self-modified nvidia-container-runtime in v1.x, because we now use node lock to track pod and container information. so this nvidia-container-runtime is no longer needed.
BUG fix
fix tasks with "gpumem-percentage" not working properly
fix dead lock when a process die with its lock not released
Adjust certain logs
update go modules to more recent version in order to support k8s v1.25
BUG fix
fix vGPUmonitor not working properly with containerd
fix installation error on k8s v1.25+
BUG fix
fix non-root user in container can't access /tmp/vgpulock, result in "unified lock error"
Rework device registration
device registration used to be done in gRpc between device-plugin and scheduler. However, in some cluster, this communication may be blocked by firewall or selinux configuration. So, we reimplement device registration mechanism by using node annotations: A-device-plugin will put its usable device and its status in "Node-A-device-register" annotation scheduler will read from this annotation and acknowledge this registration. So, gRpc will no longer be used.
Optimization in code
Put nvidia-device-plugin related code in a separate directory "nvidiadevice"
Libvgpu log adjusting
Downgrade the following API from LOG:WARN to LOG:INFO cuFuncSetCacheConfig, cuFuncSetCacheConfig ,cuModuleGetTexRef, cuModuleGetSurfRef
BUG fix
fix process can't initialize properly in driver 440
fix cuCtxCreate failed in some tensorRT task
fix env CUDA_VISIBLE_DEVICES not working properly sometimes.
BUG fix
fix initialization failed with non-root users
fix core limitation not working properly on A30
Downgrade core control output from LOG:WARN to LOG:DEBUG
Adjust default memory to 0, which means use 100% device memory
Move cache file directory from /tmp/vgpu/containers to /usr/local/vgpu/containers
Fix device memory calculation error after container crashloop
Fix env cuda_oversubscribe not set properly when MemoryScaling < 1
Fix MemoryScaling not working when set < 1
Move shared-memory from from /tmp/xxx.cache to /usr/local/vgpu/xxx.cache inside container
Add Deviceidx to scheduler monitor apis(31993)
Fix crash during initlization in vGPUmonitor
Fix oom can't be triggered when loading module
Update device-plugin version to v0.14.0
Fix a bug where a cuda process can't be launched properly
Remove node selector for scheduler
Fix an issue where mlu device-plugin can't be launched properly
Major rework on devices-related code
Add support for hygon DCU device
Fix an issue where pod pending on nodes with multi-architect devices.
Fix an issue where 31993 port can't list all GPU nodes
Add a switch on cuda_control by set env "CUDA_DISABLE_ENV=true" in container
Fix initialization error when using ray
Fix error when "N/A" is shown in command "nvidia-smi topo -m" Fix core utilization not working on some cases Adjust some documents
Fix device-plugin launch error on driver version < 500
support manual config MutatingWebhookConfiguration failurePolicy
add metrics bind address flag for scheduler
Improved log messages
fix: loss of metrics after device restart
bugfix: device-plugin monitor serves too slowly in big cluster
Add support for iluvatar GPU devices
Fix issue on "get_host_pid" func in HAMi-core
Regular devices API, make it easier to add new devices
Fix issue where device-plugin failed to start
Add support for Ascend910B3 device
Add "NVIDIA_VISIBLE_DEVICES=none" to none-gpu tasks
New features
- Add support for Ascend 910P device
- Support multiple cudevshr versions in vGPUmonitor
- Filter devices by UUID or index when registering nodes
- Support Ascend custom config for NPU virtualization
- Add event handler registration
- Support arm architecture
Bug fixes
- Fix duplicate resource keys in ConfigMap
- Fix data race when reading pod info
- Fix device ConfigMap errors
New features
- Support MetaX scheduling optimization and topology awareness
- Support Moore Threads sGPU
- Add unified ConfigMap (hami-scheduler-device) for all HAMi configuration
- Refactor Helm admission webhook config
Bug fixes
- Fix error when allocating Iluvatar device
- Fix pod assignment when pod already has a node assigned
- Fix array out-of-bounds when GPU containers are placed between non-GPU containers
- Fix wrong device assignment when one pod has multiple containers requesting GPU
New features
- Support dynamic MIG partitioning
- HAMi reinstall no longer crashes running GPU tasks
- All configuration moved to a single ConfigMap
- Add device plugin DaemonSet update strategy
- Add informer-based pod cache to reduce API server load
Bug fixes
- Fix HAMi-core stuck on tasks using cuMallocAsync
- Fix HAMi-core stuck on high glibc images
- Fix device filter registry to node
- Fix vGPUmonitor deviceIdx always reporting 0
- Fix Kubernetes version string handling with metadata
Bug fixes
- Fix passDeviceSpecsEnabled defaulting to wrong value
- Fix scheduler ignoring KUBECONFIG env variable
- Fix device filter initialization order
- Fix parseNvidiaNumaInfo index out of range
- Fix Cambricon pods not recognized by HAMi scheduler
- Fix device memory count error on cuMallocAsync
- Fix error handling for nvml.Init in device plugin
Bug fixes
- Fix device usage metrics endpoint (port 31992) not accessible
Bug fixes
- Fix multiple scheduler and vGPUmonitor stability issues
New features
- Support Enflame GCU sharing
- Support MetaX GPU and MetaX sGPU
- Support RuntimeClass with NVIDIA devices
- Add NVIDIA GPU topology score registry to node
- Add MIG info metrics to vGPUmonitor
- Add profiling support via net/http/pprof
- Add Helm chart checksum annotation for ConfigMap-triggered restarts
Bug fixes
- Fix stuck in NVIDIA driver 570+
- Fix device memory not counted properly in ComfyUI tasks
- Fix Cambricon devices not allocated properly
- Fix vgpu-devices-allocated annotation inconsistency
- Fix dynamic GPU partitioning lacking single-GPU granularity
- Fix device memory count error on cuMallocAsync
- Fix scheduler crash when MIG task runs on HAMi-core GPU
- Fix multi-process device memory count
Bug fixes
- Fix multiple scheduler and node lock stability issues
New features
- Add NVIDIA resource quota enforcement in webhook
- Support Kunlunxin topology-aware scheduling and vXPU
- Support Enflame GCU topology awareness
- Support AWS Neuron device and core allocation
- Support MetaX sGPU topology awareness
- Add aggregated scheduling failure events
- Make node lock timeout configurable
- Support MIG mode change
- Add option to disable admission webhook via Helm
- Add option to disable device plugin per device type via Helm
- Use scoped-down RBAC role for scheduler
Bug fixes
- Fix MIG partitioning NVML suppression before execution
- Fix multi-node scoring inaccuracy
- Fix error when creating Iluvatar pod
- Fix scheduler name overwrite option
Bug fixes
- Update HAMi-core to fix vLLM-related issues
- Fix quota calculation error
- Fix ClusterRoleBinding failure when changing release or chart name
- Fix nil pod check in ReleaseNodeLock
- Fix concurrent map read/write fatal errors
- Fix device plugin still active after removal from GPU node
- Upgrade nvidia-mig-parted to v0.12.2 for security fix
New features
- Support DRA (Dynamic Resource Allocation) via HAMi-DRA
- Enable leader election among multiple schedulers
- Support CDI mode on NVIDIA devices
- Sync NVIDIA device plugin with upstream v0.18.0
- Add hami_build_info metrics and version output
- Watch and hot-reload updated TLS certificates
Bug fixes
- Fix vXPU feature not working properly on P800 nodes
- Fix scheduler allocating incorrect MIG instance
- Fix concurrent map iteration and write fatal errors
Bug fixes
- Fix vLLM with version above 0.18 failing to launch with multiple GPUs
Bug fixes
- Fix device monitor not working properly
Bug fixes
- Fix HAMi-core monitoring not working properly
- Fix device utilization watcher not launching when GPU cores are not specified
- Fix GetMemoryInfo error on unified memory GPUs
New features
- Add HAMi-core mode for Ascend devices
- HAMi-DRA for NVIDIA is ready for use
- Sync Volcano vGPU device plugin with v0.19, add CDI support
- Add Prometheus ServiceMonitor to Helm charts
- Add resource quota check in webhook
- Support module-pair allocation for Ascend 910C in SuperPod environments
- Add support for VastAI devices
- Add namespaceSelector and objectSelector config for webhook
- Add Ascend core resource for HAMi-vNPU-core virtualization
- Add enableGetPreferredAllocation flag
- Add local-deploy target for minikube/kind clusters
Bug fixes
- Fix initialization error when using tensor parallelism on vLLM above 0.18
- Fix multiple device typos