test(otel): add GPU, Neuron, and EFA DRA-path integration tests - #753
Open
samehkhalil wants to merge 1 commit into
Open
samehkhalil wants to merge 1 commit into
samehkhalil wants to merge 1 commit into
Conversation
samehkhalil
force-pushed
the
test/multi-efa-dra-per-device-correlation
branch
from
September 8, 2026 16:28
077c30e to
f803c04
Compare
samehkhalil
force-pushed
the
test/multi-efa-dra-per-device-correlation
branch
from
September 8, 2026 16:56
f803c04 to
d43ae5d
Compare
samehkhalil
force-pushed
the
test/multi-efa-dra-per-device-correlation
branch
from
September 22, 2026 19:32
d43ae5d to
3073b10
Compare
Add integration coverage for the awsdevicepodcorrelation processor's DRA (Dynamic Resource Allocation) path, mirroring the device-plugin GPU/Neuron/EFA correlation tests. Each package exposes its devices via a DRA driver (through a ResourceClaimTemplate) instead of the device-plugin resource, and asserts per-device pod correlation. - test/otel/multi_efa_dra: EFA via dranet (driver dra.net); efaburn claims one of two devices, the other stays unclaimed. Guards the per-device correlation collapse (ResourceSlice keying via dra.net/rdmaDevice plus the groupbyattrs split before the resource-level promote). - test/otel/neuron_dra: Neuron via the AWS Neuron DRA driver (DeviceClass neuron.aws.com). Single Trainium device (trn1.2xlarge); the claimed device's two cores attribute to the burn pod and to no other pod. The Neuron DRA driver supports Trainium only, so this targets trn1.2xlarge. - test/otel/gpu_dra: GPU via the NVIDIA DRA driver (DeviceClass gpu.nvidia.com). g4dn.12xlarge (4 GPUs); one claimed GPU correlates to the burn pod, the other three stay uncorrelated. Asserts device count, consecutive indices, and all DCGM metrics per device. New terraform modules under terraform/eks/daemon (otel-multi-efa-dra, otel-neuron-dra, otel-gpu-dra) install the DRA driver in place of the device plugin and apply a ResourceClaimTemplate burn workload. The processor uses the GA resource.k8s.io/v1 DRA API (available since Kubernetes 1.34), so the clusters run k8s 1.35 like the rest of the suite. Wired into the test case generator. Requires a chart carrying the DRA correlation config and resource.k8s.io RBAC.
samehkhalil
force-pushed
the
test/multi-efa-dra-per-device-correlation
branch
from
September 23, 2026 11:42
3073b10 to
b0171bf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the issue
The
awsdevicepodcorrelationprocessor's Dynamic Resource Allocation (DRA) path had nointegration coverage - only the device-plugin GPU/Neuron/EFA correlation tests existed. There
was nothing exercising devices allocated via a
ResourceClaimTemplateend to end (driver ->ResourceSlice/ResourceClaim -> metric correlation), so regressions on the DRA path could ship
undetected.
Depends on aws-observability/helm-charts#356 (DRA correlation config +
resource.k8s.ioRBAC) and the processor change in amazon-contributing/opentelemetry-collector-contrib#631.
These go green in upstream CI once those land; until then they run against a chart/agent image
carrying those changes.
Description of changes
Add integration coverage mirroring the device-plugin tests, each exposing devices via a DRA
driver instead of the device-plugin resource and asserting per-device pod correlation:
test/otel/multi_efa_dra: EFA via dranet (driver dra.net);efaburnclaims one of twodevices, the other stays unclaimed. Guards the per-device correlation collapse
(ResourceSlice keying via
dra.net/rdmaDeviceplus the groupbyattrs split before theresource-level promote).
test/otel/neuron_dra: Neuron via the AWS Neuron DRA driver (DeviceClass neuron.aws.com).Single Trainium device (trn1.2xlarge); the claimed device's two cores attribute to the burn
pod and to no other pod. The Neuron DRA driver supports Trainium only, so this targets
trn1.2xlarge.
test/otel/gpu_dra: GPU via the NVIDIA DRA driver (DeviceClass gpu.nvidia.com).g4dn.12xlarge (4 GPUs); one claimed GPU correlates to the burn pod, the other three stay
uncorrelated. Asserts device count, consecutive indices, and all DCGM metrics per device.
New terraform modules under
terraform/eks/daemon(otel-multi-efa-dra, otel-neuron-dra,otel-gpu-dra) install the DRA driver in place of the device plugin and apply a
ResourceClaimTemplate burn workload. All run k8s 1.35 (the processor uses the GA
resource.k8s.io/v1DRA API, available since 1.34), and are wired into the test casegenerator.
License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
go vet -tags integrationis clean for all three packages and the generator. Validatedend-to-end on live EKS 1.35 clusters (all tests pass):
and correlated, the other 3 uncorrelated.
Test output from the live runs (
count=N= correlated-series counts each assertion queriedfrom CloudWatch):
PR checklist
makepasses locally —make simple-lint(checklicense + impi, the Go checks build-check.yml runs) passes on all tracked files;make compilepasses. Note these targets do not pass-tags integration, so the added test code (all//go:build integration) is compile-checked withgo vet -tags integrationon all three DRA packages + the generator (clean).