[core] Avoid FabricManager stall on NVLink systems in GpuProfilingManager

[aschuh-hf]

Fixes #63243

On NVLink/NVSwitch nodes (e.g. A100 SXM4), bare nvidia-smi triggers a blocking RPC to the FabricManager daemon. If another process (e.g. dynolog) holds the NVML lock, this stalls 15–20 s—long enough to exceed the raylet's dashboard agent startup timeout and prevent ray.init() from succeeding.

Changes

• node_has_gpus(): replace bare nvidia-smi with nvidia-smi --query-gpu=name --format=csv,noheader, which queries NVML directly (~0.1 s) without touching the FabricManager.

# Before
subprocess.check_output(["nvidia-smi"], stderr=subprocess.DEVNULL)

# After
subprocess.check_output(
    ["nvidia-smi", "--query-gpu=name", "--format=csv,noheader"],
    stderr=subprocess.DEVNULL,
)

• enabled property: move binary-presence checks before node_has_gpus() so the GPU detection call is skipped entirely on nodes without dynolog installed.

# Before: node_has_gpus() always called first
return self.node_has_gpus() and self._dynolog_bin is not None and self._dyno_bin is not None

# After: short-circuits before the GPU check when dynolog is absent
return self._dynolog_bin is not None and self._dyno_bin is not None and self.node_has_gpus()

• Tests: added test_node_has_gpus_uses_query_gpu_flag (asserts exact nvidia-smi flags) and test_enabled_does_not_call_node_has_gpus_when_dynolog_missing (asserts short-circuit behavior).

[gemini-code-assist]

Code Review

This pull request optimizes GPU detection in the GpuProfilingManager by reordering the enabled property to short-circuit the GPU check when required binaries are missing and updating the nvidia-smi command with specific query flags to avoid stalls. Review feedback suggests adding a timeout to the subprocess.check_output call for better robustness against system hangs. Additionally, it was noted that the class constructor might still trigger the GPU check unconditionally, which could interfere with the intended optimization and cause test failures.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 9df1a13. Configure here.}