The Intel Gaudi AI Operator is designed to manage Habana AI hardware on a Kubernetes cluster. This document provides instructions for installation, compilation, and removal of the operator.
- SPEC
- Daemon sets
- How to update the controller image
- Working with Helm
- Working with OpenShift
apiVersion: habanalabs.habana.ai/v1
kind: ClusterPolicy
metadata:
name: example-name
spec:
accelerator_allocation_mode: device-plugin # one of: device-plugin, dra
image_registry: example-registry.com
driver:
driver_loader:
images:
os1: example-os1
repository: example-registry.com/driver-loader-repo
tag: "example-tag"
os2: example-os2
repository: example-registry.com/driver-loader-repo
tag: "example-tag"
resources:
limits:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
requests:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
repo_server: example-repo.com
repo_path: path/to/repos
mlnx_ofed_repo_path: path/to/mlnx/ofed/package/optional
mlnx_ofed_version: mlnx-ofed-version-optional-x86_64.tar.gz
hugepages: hugepages_number_int_optional
external_ports: turn_on_external_port_bool_optional
driver_runner:
image:
repository: example-registry.com/driver-runner-repo
tag: "example-tag"
resources:
limits:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
requests:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
device_plugin:
image:
repository: example-registry.com/device-plugin-repo
tag: "example-tag"
resources:
limits:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
requests:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
dra_resource_driver:
image:
repository: example-registry.com/dra-resource-driver-repo
tag: "example-tag"
resources:
limits:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
requests:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
runtime:
runner:
image:
repository: example-registry.com/runtime-repo
tag: "example-tag"
resources:
limits:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
requests:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
configuration:
container_engine: container_engine_name
engine_container_runtime_configuration: container_engine_configuration_optional
habana_container_runtime_configuration: container_runtime_configuration_optional
metric_exporter:
runner:
image:
repository: example-registry.com/metric-exporter-repo
tag: "example-tag"
resources:
limits:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
requests:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
port: 41611
interval: 20
feature_discovery:
runner:
image:
repository: example-registry.com/feature-discovery-repo
tag: "example-tag"
resources:
limits:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
requests:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
nfd_plugin: boolean_nfd_installed
bmc_monitoring:
image:
repository: example-registry.com/bmc-monitoring-repo
tag: "example-tag"
resources:
limits:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
requests:
cpu: cpu_str_or_int_optional
memory: memory_str_optional
node_selector:
key_optional: value_optionalThe operator installs the following six daemon sets on the Kubernetes cluster.
This daemon set exposes information about the node that the pod is running on as Kubernetes pod labels. These labels contain information such as the existence of Habana hardware and driver version. Based on these labels, the other daemon sets are loaded.
This daemon set does not have any label selectors/affinities other than what is requested on the cluster policy node_selector field.
This daemon set loads the driver on the cluster’s node. It is done by running the habanalabs-installer.sh installation script:
habanalabs-installer.sh install --type k8sThe daemon set has an affinity to a label provided by the Habana Feature Discovery daemon set to ensure that the driver will be installed only on servers containing Habana hardware.
Because the installer script installs the driver differently based on the Operation System it runs on, the Driver daemon set needs to be able to detect which operation system the node has and run the appropriate image for the installer to run on to avoid miss-configurations, for example, if the node’s Operation system is RedHat 9.2 the image will be based on UBI 9.2, and if Ubuntu 22.04 then the image will be based on Ubuntu 22.04.
The Habana Runtime exposes the Habana hardware network and uverbs interfaces to the pod that acquired the device. The daemon set copies the runtime binaries into the host, configure the engine’s default runtime and restart the container engine to load the new configuration.
This daemon set has an affinity to a labels indicating that there is Habana hardware on the node and that the driver is loaded on the server.
This daemon set lists the existing Habana hardware and exposes the devices to the kubelet as a resource available to workloads. For Habana, this resource will be exposed as habana.ai/gaudi.
For a pod to use Habana hardware it needs to request the resource type habana.ai/gaudi like it would request cpu/memory.
This daemon set has an affinity to a labels indicating that there is Habana hardware on the node and that the driver is loaded on the server.
As an alternative to the Device Plugin, the operator can install a DRA resource driver daemon set.
Use spec.accelerator_allocation_mode to select the allocation mode:
device-plugin(default): install the Device Plugin daemon set.dra: install the DRA resource driver daemon set.
These two modes are mutually exclusive. The operator reconciles only the selected mode and deletes the other daemon set.
This daemon set exports metrics about the node and the Habana devices on it.
This daemon set has an affinity to a labels indicating that there is Habana hardware on the node and that the driver is loaded on the server.
This daemon set exports metrics by utilizing the Redfish protocol to scrape the node’s BMC. To access the BMC a config map containing the connection credentials needs to exists in the Operator’s namespace with the name bmc-metric-conf in the structure:
{
"username": "ADMIN",
"password": "ADMIN",
"port": "4001",
"servers": [
{
"password": "ADMIN",
"username": "ADMIN",
"hostname": "IP_ADDRESS"
}
]
}This daemon set has an affinity to a labels indicating that there is Habana hardware on the node and that the driver is loaded on the server.
The entire code sits in a github repo. When committing changes to the repo it will automatically build and test the changes using a built-in CI flow. But it can be done manually on a local cluster as well.
make docker-build IMG=[image-repository]:[image-tag]make docker-push IMG=[image-repository]:[image-tag]make testThe preferred way to install the operator on a cluster is by deploying a helm chart. The chart is generated based on the artifacts located in the config directory.
The helm template is auto-generated as well using the go “helmify” package. This package creates the templates and value file for all resources required for installing the operator, such as the deployment, service account, roles, etc... But it does not template the CRD spec itself, because of that, inside the hack/helm folder there are a custom files with the CRD template and values, and after the helmify package runs, the files from the hack/helm folder are being concatenate to the generated templates and values.
make helm IMG=[image-repository]:[image-tag]make helm-install-local IMG=[image-repository]:[image-tag]make helm-push ARTIFACTORY_HELM_REPO=[helm-repository] VERSION=[helm-version]make helm-install ARTIFACTORY_HELM_REPO=[helm-repository] VERSION=[helm-version]When running under OpenShift two things need to be taken into consideration:
- OpenShift Security Context Constraints (SCC)
- OLM Bundles
OpenShift SCC prevents workload from running using a privileged security policy. As part of the operator's job, it installs the driver, runs the device plugin, restart the container engine with the runtime, etc...
To enable the creation of privileged pods two things were required:
- Creation of a role binding for the service accounts to the cluster role system:openshift:scc:privileged
- Labelling the daemon sets namespace with the following labels
pod-security.kubernetes.io/audit: privileged pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/warn: privileged
OpenShift exposes operators to the users using an “Operator Hub”. This operator hub is like a marketplace where a user can install operators through.
To add an application to the Operator Hub a custom "Bundle" package needs to be created. Like a helm chart, this bundle contains all the resources required to deploy the operator; the controller deployments, service account, roles...
After the creation of the bundle, it needed to added to the operator hub. This is done by another object called “Catalog” which is a list of one or more bundles. This list is what seen in the operator hub. The name of the catalog which the bundle comes from is shown in the top right corner of each cube in the operator hub.
make bundle-build IMG=[image-repository]:[image-tag]The image will be built as [image-repository]-bundle:[image-tag]
make bundle-push IMG=[image-repository]:[image-tag]Deployment of the operator from a bundle image is done using operator-sdk CLI, install it using the following commands:
export ARCH=$(case $(uname -m) in x86_64) echo -n amd64 ;; aarch64) echo -n arm64 ;; *) echo -n $(uname -m) ;; esac)
export OS=$(uname | awk '{print tolower($0)}')
export OPERATOR_SDK_DL_URL=https://github.com/operator-framework/operator-sdk/releases/download/v1.35.0
curl -LO ${OPERATOR_SDK_DL_URL}/operator-sdk_${OS}_${ARCH}
chmod +x operator-sdk_${OS}_${ARCH}
sudo mv operator-sdk_${OS}_${ARCH} /usr/bin/operator-sdkAnd deploy the operator using the following command:
operator-sdk run bundle [image-repository]-bundle:[image-tag]make catalog-build IMG=[image-repository]:[image-tag]The image will be built as [image-repository]-catalog:[image-tag]
make catalog-push IMG=[image-repository]:[image-tag]Creates a custom catalog to OpenShift create an object of type CatalogSource.
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: [catalog-name]
namespace: openshift-marketplace
spec:
displayName: [catalog-display-name]
image: [image-repository]-catalog:[image-tag]
publisher: [publisher-name]
sourceType: grpcThe operator bundles will be added to the GUI and be installed from there. To install using the kubectl CLI the following objects needs to be created; a namespace, an operator-group and a subscription:
---
apiVersion: v1
kind: Namespace
metadata:
name: [namespace]
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: [operator-group-name]
namespace: [namespace]
spec:
targetNamespaces:
- [namespace]
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: [operator-subscription-name]
namespace: [namespace]
spec:
channel: stable
installPlanApproval: Automatic
name: [operator-name]
source: [catalog-name]
sourceNamespace: openshift-marketplaceThis will create a ClusterServiceVersion object at the target namespace, and after it reaches a success status, it will be possible to deploy the habana.ai/clusterpolicy custom resource.