fix(slurm/ib): build slurm against HPC-X SDK PMIx 5 (OMPI 5)

DOCA-OFED's openmpi.deb does not expose a link-clean libpmix: its
libpmix.so has 21 unresolved references to opal_libevent2022_* and
opal_hwloc201_* symbols inside libopen-pal, so slurm's autoconf
AC_LINK_IFELSE on -lpmix fails ("unable to locate pmix installation").

Switch the IB image to NVIDIA HPC-X SDK v2.26 (CUDA13 build) instead.
HPC-X ships a self-contained OMPI 5 + PMIx 5.0.10rc3 stack at
/opt/mellanox/hpcx/ompi5/. Its libpmix.so has clean DT_NEEDED entries
(libevent_core, libevent_pthreads, libhwloc) and DT_RUNPATH=$ORIGIN/.
so it links and runs cleanly without dragging in libopen-pal.

slurm is now configured --with-pmix=/opt/mellanox/hpcx/ompi5, which
makes slurm's autoconf bake -Wl,-rpath,/opt/mellanox/hpcx/ompi5/lib
into mpi_pmix_v5.so. slurmstepd dlopens libpmix.so.2 via that RPATH at
MPI step launch — no /etc/ld.so.conf.d entry for HPC-X is needed (and
adding one would make slurmd link against HPC-X's bundled libhwloc.so.15
during dh_shlibdeps and break the .deb build).

DOCA-OFED apt repo is still used for IB userland (rdma-core,
ibverbs-providers, infiniband-diags) — only the openmpi package is
dropped in favor of HPC-X.

Locally validated end-to-end: configure detects PMIx 5 at
/opt/mellanox/hpcx/ompi5, mpi_pmix_v5.so / mpi_pmi2.so / gpu_nvml.so
all build, dh_shlibdeps passes, 17 slurm-smd debs produced.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: bobcao3

schedmd/slurm/25.11/ubuntu24.04/Dockerfile

@@ -225,26 +225,35 @@ ldconfig
 EOR
 
 ################################################################################
-# base-ib — DOCA userland IB verbs + DOCA OpenMPI 4.
+# base-ib — DOCA-OFED userland IB verbs + NVIDIA HPC-X SDK (OMPI 5 + PMIx 5).
 #
-# Installs only guest/userland components (no kmod, no metapackage bloat).
-# Open MPI comes from DOCA repo (4.1.x, Mellanox-tuned for IB).
+# IB userland (rdma-core, ibverbs, infiniband-diags) comes from the DOCA repo.
+# We deliberately do NOT install DOCA's `openmpi` apt package: its libpmix.so
+# has 21 unresolved references to symbols inside libopen-pal (opal_libevent2022_*
+# / opal_hwloc201_*), which makes slurm's autoconf AC_LINK_IFELSE on -lpmix fail.
+# The DOCA OMPI is also OMPI 4 + PMIx 3.2.5, which is not the modern stack.
 #
-# Deliberately NO Ubuntu libpmix-dev/libpmix2t64. DOCA's openmpi deb installs
-# a complete standalone PMIx 3.2.5 tree under /usr/mpi/gcc/openmpi-4.1.9a1/
-# (libpmix.so.2 + pmix.h + pmix.pc) which is the PMIx that DOCA OMPI itself
-# is linked against. The slurm-build stage builds slurm --with-pmix pointing
-# at this tree, so slurmd's mpi/pmix server plugin and DOCA OMPI's PMIx
-# client share one ABI-identical libpmix.so.2 — the single-vendor invariant
-# NVIDIA HPC-X requires. Co-installing apt libpmix-dev (PMIx 5.x) would only
-# put a second pmix.h on the build search path and a second libpmix.so on
-# the loader path, with no benefit (DOCA OMPI was not built --with-pmi=slurm,
-# so PMI-2-only launch is not an option on this fabric).
+# Instead we lay down NVIDIA HPC-X SDK at /opt/mellanox/hpcx. HPC-X ships both
+# ompi4 (OMPI 4 + PMIx 3) and ompi5 (OMPI 5 + PMIx 5.0.10rc3) trees, plus UCX,
+# HCOLL, SHARP, UCC, and the NCCL-RDMA-SHARP plugin. We expose only the OMPI 5
+# tree to slurmd so the mpi/pmix plugin links against PMIx 5 cleanly. The OMPI 5
+# libpmix.so.2 has self-contained DT_NEEDED entries (libevent_core,
+# libevent_pthreads, libhwloc) and `$ORIGIN/.` RUNPATH, so it is portable.
+#
+# Deliberately NO Ubuntu apt libpmix-dev/libpmix2t64: HPC-X's PMIx is the only
+# PMIx installed on the image, keeping the loader search path unambiguous.
 FROM base-cuda AS base-ib
 
 SHELL ["bash", "-c"]
 
 ARG DEBIAN_FRONTEND=noninteractive
+# HPC-X SDK pin. Update by querying the downloader API at
+#   https://downloaders.azurewebsites.net/downloaders/hpcx_downloader/helper.php
+# (see slinky-containers/docs/hpcx-pin.md if added later) and verifying SHA256.
+ARG HPCX_VERSION=v2.26
+ARG HPCX_TARBALL=hpcx-v2.26-gcc-doca_ofed-ubuntu24.04-cuda13-x86_64.tbz
+ARG HPCX_URL=https://content.mellanox.com/hpc/hpc-x/v2.26_cuda13/hpcx-v2.26-gcc-doca_ofed-ubuntu24.04-cuda13-x86_64.tbz
+ARG HPCX_SHA256=57656486279f6bcf44bfc60228bac5c34e25366e6d3b7f7d1d0a46427a8836ff
 
 RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
   --mount=type=cache,target=/var/lib/apt,sharing=locked <<EOR
@@ -263,22 +272,39 @@ apt-get -qq -y install --no-install-recommends \
   libibverbs-dev \
   librdmacm1 \
   librdmacm-dev \
-  infiniband-diags \
-  openmpi
-# DOCA openmpi installs under /usr/mpi/gcc/openmpi-*/ but does not create
-# /usr/bin symlinks. Expose a version-agnostic MPI_HOME symlink + /usr/bin
-# entries so the downstream slurmd stage and operators can resolve mpirun.
-OMPI_DIR=$(ls -d /usr/mpi/gcc/openmpi-* | head -1)
-ln -sfn "${OMPI_DIR}" /usr/mpi/gcc/openmpi
+  infiniband-diags
+EOR
+
+# Lay down HPC-X SDK at /opt/mellanox/hpcx. Versioned dir is the actual extract
+# target; /opt/mellanox/hpcx is a symlink to it so paths in slurm builds and
+# ld.so.conf.d entries are stable across HPCX_VERSION bumps.
+RUN <<EOR
+set -xeuo pipefail
+cd /tmp
+curl -fsSL --retry 3 -o "${HPCX_TARBALL}" "${HPCX_URL}"
+echo "${HPCX_SHA256}  ${HPCX_TARBALL}" | sha256sum -c -
+mkdir -p /opt/mellanox
+tar -xf "${HPCX_TARBALL}" -C /opt/mellanox
+HPCX_DIR=$(ls -d /opt/mellanox/hpcx-* | head -1)
+ln -sfn "${HPCX_DIR}" /opt/mellanox/hpcx
+rm -f "${HPCX_TARBALL}"
+# Deliberately NOT adding /opt/mellanox/hpcx/*/lib to /etc/ld.so.conf.d/.
+# HPC-X is fully self-resolving:
+#   - slurm's mpi_pmix_v5.so plugin is built --with-pmix=/opt/mellanox/hpcx/ompi5,
+#     and slurm's autoconf bakes -Wl,-rpath,$pmix_libdir into the plugin
+#     (PMIX_V5_LDFLAGS), so slurmstepd dlopens the right libpmix.so.2 at MPI step
+#     launch without depending on ld.so.cache;
+#   - HPC-X's libpmix.so itself has DT_RUNPATH=$ORIGIN/. so its sibling libs
+#     (libevent_core, libevent_pthreads, libhwloc) resolve relative to its own
+#     directory;
+#   - HPC-X mpirun / mpiexec carry DT_RPATH=$ORIGIN/../lib so they self-resolve
+#     when invoked via the /usr/bin symlinks below.
+# Putting HPC-X libs on /etc/ld.so.conf.d/ would shadow Ubuntu's libhwloc.so.15
+# during the slurmd link, causing dh_shlibdeps to fail with "no dependency
+# information found" (HPC-X libs are not from any Debian package).
 for b in mpirun mpiexec mpicc mpicxx; do
-  ln -sf "/usr/mpi/gcc/openmpi/bin/${b}" "/usr/bin/${b}"
+  ln -sf "/opt/mellanox/hpcx/ompi5/bin/${b}" "/usr/bin/${b}"
 done
-# Expose DOCA OMPI's lib dir to the system loader so libpmix.so.2
-# (and the OMPI runtime libs) resolve via ldconfig. Required for slurmd
-# step launch: mpi_pmix.so is built --with-pmix=/usr/mpi/gcc/openmpi
-# and dlopens libpmix.so.2 at MPI step bootstrap.
-printf '%s\n' "/usr/mpi/gcc/openmpi/lib" > /etc/ld.so.conf.d/200_doca-ompi.conf
-ldconfig
 EOR
 
 ################################################################################
@@ -289,7 +315,7 @@ EOR
 # from patches/common/ then patches/${FABRIC}/ so EFA and IB paths receive the
 # correct debian/control and debian/rules hunks (EFA drops apt libpmix-dev and
 # points --with-pmix at /opt/amazon/pmix; IB drops apt libpmix-dev and points
-# --with-pmix at /usr/mpi/gcc/openmpi where DOCA's PMIx 3.2.5 lives).
+# --with-pmix at /opt/mellanox/hpcx/ompi5 where HPC-X's PMIx 5 lives).
 ARG FABRIC=ib
 ARG FABRIC_BASE=base-${FABRIC}
 FROM ${FABRIC_BASE} AS slurm-build
@@ -338,7 +364,7 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
   --mount=type=cache,target=/var/lib/apt,sharing=locked <<EOR
 set -xeuo pipefail
 mk-build-deps -ir --tool='apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends' ${SLURM_DIR}/debian/control >/dev/null
-( cd ${SLURM_DIR} && debuild -b -uc -us )
+( cd ${SLURM_DIR} && debuild -b -uc -us >/dev/null )
 EOR
 
 ################################################################################

schedmd/slurm/25.11/ubuntu24.04/patches/ib/debian-control-ib.patch

@@ -5,11 +5,11 @@ Description: IB fabric build-deps: add cuda-nvml-dev, drop apt libpmix-dev.
  enable the mpi/gpu_nvml autodetect plugin at build time (runtime
  libnvidia-ml.so.1 is injected by nvidia-container-toolkit from the host
  driver).
- libpmix-dev is dropped because the IB image builds slurm against DOCA's
- standalone PMIx 3.2.5 (under /usr/mpi/gcc/openmpi/, see debian/rules
- --with-pmix= patch). Keeping apt libpmix-dev (PMIx 5.x) installed would
- put two PMIx vendors on the image and let the wrong pmix.h leak into
- the build search path.
+ libpmix-dev is dropped because the IB image builds slurm against HPC-X's
+ PMIx 5 (under /opt/mellanox/hpcx/ompi5/, see debian/rules --with-pmix=
+ patch). Keeping apt libpmix-dev installed would put a second pmix.h on
+ the build search path and a second libpmix.so on the loader path, with
+ the wrong vendor/ABI for the HPC-X OMPI 5 stack.
 Author: meshy-dev
 Forwarded: not-needed
 Last-Update: 2026-05-06

schedmd/slurm/25.11/ubuntu24.04/patches/ib/debian-rules-nvml-ib.patch

@@ -1,17 +1,16 @@
-Description: Configure Slurm with NVML and DOCA-bundled PMIx for IB fabric.
+Description: Configure Slurm with NVML and HPC-X PMIx 5 for IB fabric.
  --with-nvml=/usr/local/cuda tells slurm's GPU autodetect plugin to
  link libnvidia-ml at build time, enabling slurmd GPU enumeration via
  NVML (required for GresTypes=gpu + GPUs GRES binding on H200 nodes).
  .
- --with-pmix=/usr/mpi/gcc/openmpi points slurmd's mpi/pmix plugin at
- the standalone PMIx 3.2.5 tree shipped inside DOCA's OpenMPI install
- (DOCA openmpi.deb lays down libpmix.so.2 + pmix.h + pmix.pc under
- /usr/mpi/gcc/openmpi-4.1.9a1/, exposed via the /usr/mpi/gcc/openmpi
- version-agnostic symlink). This keeps slurmd's PMIx server plugin
- ABI-identical to the PMIx client linked into DOCA OpenMPI 4 — single
- vendor, no cross-version dance — which is what NVIDIA HPC-X / DOCA
- OMPI requires (DOCA OMPI was not built --with-pmi=slurm and only
- supports PMIx-based launch).
+ --with-pmix=/opt/mellanox/hpcx/ompi5 points slurmd's mpi/pmix plugin
+ at the PMIx 5.0.10rc3 tree shipped inside NVIDIA HPC-X SDK's OMPI 5
+ subtree. HPC-X's ompi5/lib/libpmix.so.2 has self-contained DT_NEEDED
+ entries (libevent_core, libevent_pthreads, libhwloc) and `$ORIGIN/.`
+ RUNPATH, so it links cleanly without dragging in libopen-pal — unlike
+ DOCA-OFED's openmpi.deb whose libpmix has 21 unresolved opal_libevent2022_*
+ / opal_hwloc201_* symbols. HPC-X is also the modern OMPI 5 + PMIx 5
+ stack NVIDIA documents for HPC + slurm; OMPI 4 / PMIx 3 are not used.
 Author: meshy-dev
 Forwarded: not-needed
 Last-Update: 2026-05-06
@@ -22,7 +21,7 @@ Last-Update: 2026-05-06
  	# Run with --disable-debug to allow optimizations, -g is still provided
  	# by dpkg-buildflags.
 -	dh_auto_configure -- --sysconfdir=/etc/slurm --disable-debug --with-mysql_config --with-slurmrestd --with-pmix --enable-pam --with-pam_dir=/usr/lib/$(DEB_HOST_MULTIARCH)/security --with-systemdsystemunitdir=/lib/systemd/system/ SUCMD=/bin/su SLEEP_CMD=/bin/sleep
-+	dh_auto_configure -- --sysconfdir=/etc/slurm --disable-debug --with-mysql_config --with-slurmrestd --with-pmix=/usr/mpi/gcc/openmpi --with-nvml=/usr/local/cuda --enable-pam --with-pam_dir=/usr/lib/$(DEB_HOST_MULTIARCH)/security --with-systemdsystemunitdir=/lib/systemd/system/ SUCMD=/bin/su SLEEP_CMD=/bin/sleep
++	dh_auto_configure -- --sysconfdir=/etc/slurm --disable-debug --with-mysql_config --with-slurmrestd --with-pmix=/opt/mellanox/hpcx/ompi5 --with-nvml=/usr/local/cuda --enable-pam --with-pam_dir=/usr/lib/$(DEB_HOST_MULTIARCH)/security --with-systemdsystemunitdir=/lib/systemd/system/ SUCMD=/bin/su SLEEP_CMD=/bin/sleep
 
  override_dh_auto_install:
  	dh_auto_install