feat(validators): RoCE NET variant for nccl-all-reduce-bw (AICR_NCCL_FABRIC), default EFA

Adds a ConnectX RoCE path to the nccl-all-reduce-bw-net test, selected by
AICR_NCCL_FABRIC=roce (default efa — every existing EFA recipe is
unchanged). Fabric-keyed and accelerator-agnostic: the RoCE NET template
lives at testdata/roce/{service}/ and is shared across EKS RoCE nodes
rather than per-accelerator, so it does not depend on any specific
accelerator constant.

When fabric=roce, the NET test:
- uses NCCL's built-in IB/verbs transport over the ConnectX RoCE devices
  (NCCL_IB_HCA=rocep, NCCL_NET_PLUGIN=none to bypass the bundled aws-ofi
  EFA plugin), on a CUDA-13 pytorch image (sm_103-capable; the EFA
  hpc-cloud image is CUDA 12 + EFA-only);
- claims RoCE NICs via a DRA ResourceClaimTemplate (roce.networking.k8s.aws)
  instead of the vpc.amazonaws.com/efa extended resource;
- disables HPC-X UCC/HCOLL and forces the ob1 PML over TCP (their team-create
  / UCX PML segfault during MPI_Init on this image — MPI is only bootstrap).

The transport assertion (verifyTransportFromLogs) is unchanged: it already
accepts any non-Socket NET plugin, so 'Using network IB' passes.

AICR_NCCL_FABRIC is forwarded to the NET check pod via buildEnv (the test
runs in-Job), scoped to nccl-all-reduce-bw-net.

Validated end-to-end on a ConnectX-RoCE GB300 cluster: NCCL IB over 8
rocep* RoCE devices (GPUDirect RDMA), ~387 GB/s peak busbw. The env knob is
interim; snapshot-based fabric auto-detection is tracked in #1413.

Refs #1413, #1410.

Author: yuanchen8911

docs/user/validation.md

@@ -48,9 +48,21 @@ ones) that match the target fabric:
 | Check | Transport | When it's selected |
 |---|---|---|
 | `nccl-all-reduce-bw` | Auto-detect (whatever NCCL picks) | H100/H200 on EKS, H100 on GKE, and B200/GB200 on self-managed clusters (`service=any`). Preserves the pre-variant behavior. |
-| `nccl-all-reduce-bw-net` | NET (EFA on EKS) | GB200 + EKS. Asserts EFA actually carried traffic — catches silent fallback to Socket when the NVIDIA driver is missing `NVreg_GrdmaPciTopoCheckOverride=1`. |
+| `nccl-all-reduce-bw-net` | NET (EFA on EKS by default; ConnectX RoCE via `AICR_NCCL_FABRIC=roce`) | GB200 + EKS. Asserts EFA actually carried traffic — catches silent fallback to Socket when the NVIDIA driver is missing `NVreg_GrdmaPciTopoCheckOverride=1`. |
 | `nccl-all-reduce-bw-nvls` | NVLS (MNNVL across an NVL72 IMEX domain) | GB200 + EKS, and GB200 + OKE. Asserts the NVLS communicator actually initialized — catches silent fallback to EFA (EKS) or Socket (OKE) when the IMEX domain is misconfigured. |
 
+The `-net` check defaults to the AWS EFA fabric. On a ConnectX **RoCE** cluster
+(e.g. DGXC GB300 `p6e-gb300r`), set `AICR_NCCL_FABRIC=roce` in the `aicr
+validate` environment to run the NET test over NCCL's built-in IB/verbs
+transport across `roce.networking.k8s.aws` DRA devices instead. The value is
+scoped to the `-net` check only; unset (or `efa`) leaves every existing recipe
+on the EFA path unchanged, and any other value is rejected. The RoCE runtime
+image installs `openssh-server` at startup, so the GPU nodes need apt egress;
+on an air-gapped cluster the RoCE NET test cannot bootstrap. This env override is
+interim — snapshot-based fabric auto-detection (and removing the runtime
+package install once a CUDA-13 image ships sshd) is tracked in
+[NVIDIA/aicr#1413](https://github.com/NVIDIA/aicr/issues/1413).
+
 GB200/EKS recipes (both `training` and `inference` intents) enable `-net` and
 `-nvls` together rather than the auto-detect variant, because those nodes
 expose two inter-node fabrics simultaneously and a single auto-detect test

pkg/validator/v1/job_plan_internal.go

@@ -52,6 +52,20 @@ const (
 
 	requireScopedInferenceGatewayEnv = "AICR_REQUIRE_SCOPED_INFERENCE_GATEWAY"
 
+	// ncclAllReduceBWNetCheckName is the catalog name of the NCCL all-reduce NET
+	// check. Used to scope AICR_NCCL_FABRIC forwarding to that validator only.
+	ncclAllReduceBWNetCheckName = "nccl-all-reduce-bw-net"
+
+	// ncclFabricEnv selects the NET fabric (efa default | roce). Forwarded to
+	// the NET check pod so the in-Job validator can observe it. This is the
+	// orchestrator (forwarding) end; the validator-pod (reading) end defines the
+	// same literal as ncclFabricEnv in
+	// validators/performance/nccl_all_reduce_bw_constraint.go — keep the two in
+	// sync. The split mirrors the other forwarded validator envs (HF_TOKEN,
+	// AICR_REQUIRE_SCOPED_INFERENCE_GATEWAY, AICR_INFERENCE_PERF_NO_CLEANUP): the
+	// pod binary is a separate package that does not import this one.
+	ncclFabricEnv = "AICR_NCCL_FABRIC"
+
 	// inferencePerfNoCleanupEnv, when truthy, makes the inference-perf validator
 	// leave its namespace/DGD/workers/frontend/AIPerf Job in place after the run
 	// for post-mortem inspection. Forwarded only to the inference-perf pod.
@@ -155,6 +169,14 @@ func buildEnv(
 		env = append(env, corev1.EnvVar{Name: requireScopedInferenceGatewayEnv, Value: v})
 	}
 
+	// Forward the NCCL fabric selector to the nccl-all-reduce-bw-net check pod.
+	// The NET test runs inside the Job, so it can't observe the CLI environment
+	// unless forwarded here. Unset (default) leaves the check on EFA; scoped to
+	// the NET check so unrelated validator pods don't carry it.
+	if v, ok := os.LookupEnv(ncclFabricEnv); ok && v != "" && entry.Name == ncclAllReduceBWNetCheckName {
+		env = append(env, corev1.EnvVar{Name: ncclFabricEnv, Value: v})
+	}
+
 	// Forward the inference-perf no-cleanup debug toggle into that validator pod.
 	// Cleanup runs inside the Job, so it can't see the CLI process environment
 	// unless the orchestrator carries the value across. Scoped to the
@@ -175,7 +197,7 @@ func buildEnv(
 	// forwarded value (k8s takes the last duplicate), breaking that trust
 	// boundary.
 	for _, e := range entry.Env {
-		if e.Name == hfTokenEnvVar || e.Name == requireScopedInferenceGatewayEnv || e.Name == inferencePerfNoCleanupEnv {
+		if e.Name == hfTokenEnvVar || e.Name == requireScopedInferenceGatewayEnv || e.Name == inferencePerfNoCleanupEnv || e.Name == ncclFabricEnv {
 			continue
 		}
 		env = append(env, corev1.EnvVar{Name: e.Name, Value: e.Value})

pkg/validator/v1/job_plan_test.go

@@ -522,6 +522,85 @@ func TestBuildJobPlan_ForwardsInferencePerfNoCleanupEnv(t *testing.T) {
 	})
 }
 
+// TestBuildJobPlan_ForwardsNCCLFabricEnv verifies the NET fabric selector is
+// carried from the CLI process into the nccl-all-reduce-bw-net validator Job
+// (where ncclFabric() reads it), only that validator, and that a catalog-pinned
+// value can never shadow or substitute for the forwarded one. Unlike the
+// no-cleanup toggle, the value is forwarded verbatim (the validator validates it).
+func TestBuildJobPlan_ForwardsNCCLFabricEnv(t *testing.T) {
+	build := func(entry ValidatorEntry) map[string]string {
+		plan, err := BuildJobPlan(entry, "run-1", "ns", "1.0.0", "abc123", "sa", nil, nil, nil, "", "", nil)
+		if err != nil {
+			t.Fatalf("BuildJobPlan error: %v", err)
+		}
+		m := make(map[string]string)
+		for _, e := range plan.Env {
+			m[e.Name] = e.Value
+		}
+		return m
+	}
+
+	netEntry := ValidatorEntry{Name: ncclAllReduceBWNetCheckName, Phase: "performance", Image: "img:v1", Timeout: time.Minute}
+
+	t.Run("forwarded verbatim to nccl-all-reduce-bw-net", func(t *testing.T) {
+		t.Setenv(ncclFabricEnv, "roce")
+		if got := build(netEntry)[ncclFabricEnv]; got != "roce" {
+			t.Errorf("%s env = %q, want roce", ncclFabricEnv, got)
+		}
+	})
+	t.Run("empty value omitted", func(t *testing.T) {
+		t.Setenv(ncclFabricEnv, "")
+		if _, present := build(netEntry)[ncclFabricEnv]; present {
+			t.Errorf("%s should not be in Job env when empty", ncclFabricEnv)
+		}
+	})
+	t.Run("not forwarded to other validators", func(t *testing.T) {
+		t.Setenv(ncclFabricEnv, "roce")
+		other := ValidatorEntry{Name: InferencePerfCheckName, Phase: "performance", Image: "img:v1", Timeout: time.Minute}
+		if _, present := build(other)[ncclFabricEnv]; present {
+			t.Errorf("%s must not be forwarded to a non-NET validator", ncclFabricEnv)
+		}
+	})
+
+	// values collects every occurrence of the env var (not just the last) so we
+	// can assert the catalog value is dropped, not merely shadowed.
+	values := func(entry ValidatorEntry) []string {
+		plan, err := BuildJobPlan(entry, "run-1", "ns", "1.0.0", "abc123", "sa", nil, nil, nil, "", "", nil)
+		if err != nil {
+			t.Fatalf("BuildJobPlan error: %v", err)
+		}
+		var got []string
+		for _, e := range plan.Env {
+			if e.Name == ncclFabricEnv {
+				got = append(got, e.Value)
+			}
+		}
+		return got
+	}
+
+	t.Run("catalog value cannot override forwarded value", func(t *testing.T) {
+		t.Setenv(ncclFabricEnv, "roce")
+		entry := ValidatorEntry{
+			Name: ncclAllReduceBWNetCheckName, Phase: "performance", Image: "img:v1", Timeout: time.Minute,
+			Env: []EnvVar{{Name: ncclFabricEnv, Value: "efa"}},
+		}
+		if got := values(entry); len(got) != 1 || got[0] != "roce" {
+			t.Errorf("%s env = %v, want exactly [roce] (catalog value must be dropped)", ncclFabricEnv, got)
+		}
+	})
+
+	t.Run("catalog value alone cannot select fabric", func(t *testing.T) {
+		t.Setenv(ncclFabricEnv, "")
+		entry := ValidatorEntry{
+			Name: ncclAllReduceBWNetCheckName, Phase: "performance", Image: "img:v1", Timeout: time.Minute,
+			Env: []EnvVar{{Name: ncclFabricEnv, Value: "roce"}},
+		}
+		if got := values(entry); len(got) != 0 {
+			t.Errorf("%s env = %v, want none (catalog must not select fabric without shell env)", ncclFabricEnv, got)
+		}
+	})
+}
+
 func TestBuildJobPlanWithDefaults(t *testing.T) {
 	// Test with minimal entry (no custom resources, no tolerations, no node selector)
 	entry := ValidatorEntry{

recipes/validators/README.md

@@ -47,7 +47,7 @@ Applied by `catalog.Load` (`pkg/validator/catalog/catalog.go`) in order:
 | Name | Description | Timeout |
 |------|-------------|---------|
 | `nccl-all-reduce-bw` | Verify NCCL All Reduce Bus Bandwidth meets threshold | 30m |
-| `nccl-all-reduce-bw-net` | Verify NCCL All Reduce Bus Bandwidth on the NET transport (EFA on EKS) | 30m |
+| `nccl-all-reduce-bw-net` | Verify NCCL All Reduce Bus Bandwidth on the NET transport (EFA on EKS; ConnectX RoCE via `AICR_NCCL_FABRIC=roce`) | 30m |
 | `nccl-all-reduce-bw-nvls` | Verify NCCL All Reduce Bus Bandwidth on the NVLS transport (MNNVL across an NVL72 IMEX domain) | 30m |
 
 ### Conformance Phase