Prioritize TensorFlow pip CUDA/cuDNN runtime libs for MLPerf TF GPU runs

[Copilot]

🧾 PR Checklist

• Target branch is dev

📌 Note: PRs must be raised against dev. Do not commit directly to main.

📁 File Hygiene & Output Handling

• No unintended files (e.g., logs, cache, temp files, pycache, output folders) are committed

📝 Comments & Communication

• Proper inline comments are added to explain important or non-obvious changes

• PR title and description clearly state what the PR does and why

• Related issues (if any) are properly referenced (Fixes #, Related to #, etc.)

• Issue summary

  • ResNet50 TensorFlow GPU runs can fail with No DNN in stream executor due to cuDNN runtime mismatch (runtime cuDNN lower than TensorFlow’s compiled requirement).
  • The failure is caused by loading incompatible system CUDA/cuDNN libs ahead of TensorFlow’s pip-provided NVIDIA runtime libs.

• Change set

  • Scoped runtime-path fix in TF reference path (script/app-mlperf-inference-mlcommons-python/customize.py)
    • Added a helper that discovers site-packages/nvidia/*/lib directories from the active Python environment.
    • Prepends those directories to +LD_LIBRARY_PATH for MLC_MLPERF_BACKEND=tf and MLC_MLPERF_DEVICE in {gpu,cuda} on non-Windows.
  • Hardening
    • Uses os.pathsep for portability.
    • Uses subprocess.check_output([...], timeout=20) with targeted exception handling to avoid hangs and broad exception masking.

• Illustrative snippet

if os_info['platform'] != 'windows' and env.get('MLC_MLPERF_BACKEND') == 'tf' and env.get('MLC_MLPERF_DEVICE') in ['gpu', 'cuda']:
    _prepend_tensorflow_nvidia_libs_to_ld_library_path(env, logger)

# inside helper:
env['+LD_LIBRARY_PATH'] = lib_dirs + env.get('+LD_LIBRARY_PATH', [])

🛡️ Safety & Security

• No secrets or credentials are committed
• Paths, shell commands, and environment handling are safe and portable