Skip to content

test(otel): add GPU, Neuron, and EFA DRA-path integration tests - #753

Open
samehkhalil wants to merge 1 commit into
aws:mainfrom
samehkhalil:test/multi-efa-dra-per-device-correlation
Open

samehkhalil wants to merge 1 commit into
aws:mainfrom
samehkhalil:test/multi-efa-dra-per-device-correlation

Conversation

@samehkhalil

@samehkhalil samehkhalil commented Sep 2, 2026

Copy link
Copy Markdown

Description of the issue

The awsdevicepodcorrelation processor's Dynamic Resource Allocation (DRA) path had no
integration coverage - only the device-plugin GPU/Neuron/EFA correlation tests existed. There
was nothing exercising devices allocated via a ResourceClaimTemplate end to end (driver ->
ResourceSlice/ResourceClaim -> metric correlation), so regressions on the DRA path could ship
undetected.

Depends on aws-observability/helm-charts#356 (DRA correlation config + resource.k8s.io
RBAC) and the processor change in amazon-contributing/opentelemetry-collector-contrib#631.
These go green in upstream CI once those land; until then they run against a chart/agent image
carrying those changes.

Description of changes

Add integration coverage mirroring the device-plugin tests, each exposing devices via a DRA
driver instead of the device-plugin resource and asserting per-device pod correlation:

  • test/otel/multi_efa_dra: EFA via dranet (driver dra.net); efaburn claims one of two
    devices, the other stays unclaimed. Guards the per-device correlation collapse
    (ResourceSlice keying via dra.net/rdmaDevice plus the groupbyattrs split before the
    resource-level promote).
  • test/otel/neuron_dra: Neuron via the AWS Neuron DRA driver (DeviceClass neuron.aws.com).
    Single Trainium device (trn1.2xlarge); the claimed device's two cores attribute to the burn
    pod and to no other pod. The Neuron DRA driver supports Trainium only, so this targets
    trn1.2xlarge.
  • test/otel/gpu_dra: GPU via the NVIDIA DRA driver (DeviceClass gpu.nvidia.com).
    g4dn.12xlarge (4 GPUs); one claimed GPU correlates to the burn pod, the other three stay
    uncorrelated. Asserts device count, consecutive indices, and all DCGM metrics per device.

New terraform modules under terraform/eks/daemon (otel-multi-efa-dra, otel-neuron-dra,
otel-gpu-dra) install the DRA driver in place of the device plugin and apply a
ResourceClaimTemplate burn workload. All run k8s 1.35 (the processor uses the GA
resource.k8s.io/v1 DRA API, available since 1.34), and are wired into the test case
generator.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

go vet -tags integration is clean for all three packages and the generator. Validated
end-to-end on live EKS 1.35 clusters (all tests pass):

  • neuron_dra (trn1.2xlarge): 6/6 pass; the claimed device's 2 cores correlate to the burn pod.
  • gpu_dra (g4dn.12xlarge): 5/5 pass (incl. 6 DCGM per-device subtests); 1 of 4 GPUs claimed
    and correlated, the other 3 uncorrelated.
  • multi_efa_dra: claimed device correlates to the pod; the second device stays unclaimed.

Test output from the live runs (count=N = correlated-series counts each assertion queried
from CloudWatch):

query returned count=2
--- PASS: TestNeuronDRADeviceCount / CoreCount / CoresPerDevice /
          DeviceIndicesAreIntegers / ClaimedDeviceCorrelation / BurnPodLabels
ok  .../test/otel/neuron_dra   (6/6 pass)

query returned count=1   # claimed GPU on the burn pod
query returned count=4   # GPUs on the node
--- PASS: TestGPUDRADeviceCount / ConsecutiveIndices / ClaimedVsUnclaimedCorrelation /
          BurnPodLabels / AllMetricsPerDevice{GPU_UTIL,MEM_COPY_UTIL,FB_USED,FB_FREE,GPU_TEMP,POWER_USAGE}
ok  .../test/otel/gpu_dra   (5/5 pass)

PR checklist

@samehkhalil
samehkhalil requested a review from a team as a code owner September 2, 2026 11:53
@samehkhalil
samehkhalil force-pushed the test/multi-efa-dra-per-device-correlation branch from 077c30e to f803c04 Compare September 8, 2026 16:28
@samehkhalil samehkhalil changed the title test(otel/multi_efa_dra): add integration test for the EFA DRA path test(otel): add GPU, Neuron, and EFA DRA-path integration tests Sep 8, 2026
@samehkhalil
samehkhalil force-pushed the test/multi-efa-dra-per-device-correlation branch from f803c04 to d43ae5d Compare September 8, 2026 16:56
@samehkhalil
samehkhalil force-pushed the test/multi-efa-dra-per-device-correlation branch from d43ae5d to 3073b10 Compare September 22, 2026 19:32
Add integration coverage for the awsdevicepodcorrelation processor's DRA
(Dynamic Resource Allocation) path, mirroring the device-plugin GPU/Neuron/EFA
correlation tests. Each package exposes its devices via a DRA driver (through a
ResourceClaimTemplate) instead of the device-plugin resource, and asserts
per-device pod correlation.

- test/otel/multi_efa_dra: EFA via dranet (driver dra.net); efaburn claims one
  of two devices, the other stays unclaimed. Guards the per-device correlation
  collapse (ResourceSlice keying via dra.net/rdmaDevice plus the groupbyattrs
  split before the resource-level promote).
- test/otel/neuron_dra: Neuron via the AWS Neuron DRA driver (DeviceClass
  neuron.aws.com). Single Trainium device (trn1.2xlarge); the claimed device's
  two cores attribute to the burn pod and to no other pod. The Neuron DRA driver
  supports Trainium only, so this targets trn1.2xlarge.
- test/otel/gpu_dra: GPU via the NVIDIA DRA driver (DeviceClass gpu.nvidia.com).
  g4dn.12xlarge (4 GPUs); one claimed GPU correlates to the burn pod, the other
  three stay uncorrelated. Asserts device count, consecutive indices, and all
  DCGM metrics per device.

New terraform modules under terraform/eks/daemon (otel-multi-efa-dra,
otel-neuron-dra, otel-gpu-dra) install the DRA driver in place of the device
plugin and apply a ResourceClaimTemplate burn workload. The processor uses the
GA resource.k8s.io/v1 DRA API (available since Kubernetes 1.34), so the clusters
run k8s 1.35 like the rest of the suite. Wired into the test case generator.
Requires a chart carrying the DRA correlation config and resource.k8s.io RBAC.
@samehkhalil
samehkhalil force-pushed the test/multi-efa-dra-per-device-correlation branch from 3073b10 to b0171bf Compare September 23, 2026 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant