add latency benchmark mode

[RongLei-intel]

This pull request introduces a new latency benchmark mode to the HCCL demo tool, allowing users to measure communication latency per iteration rather than overall throughput. The changes include new CLI options, updates to documentation, and logic to select the appropriate benchmarking method based on user input.

Latency Benchmark Feature:

• Added a new CLI flag --latency_benchmark to run_hccl_demo.py and documented its usage in README.md, enabling users to run tests that measure latency per iteration. [1] [2]
• Updated the command construction in run_hccl_demo.py to set the HCCL_DEMO_LATENCY_BENCHMARK environment variable when latency benchmarking is enabled.

Benchmark Logic and Reporting:

• Implemented the benchmark_latency function in hccl_demo.cpp to measure and report average latency per iteration, including warmup and correctness validation.
• Modified the collective test driver and test runner in hccl_demo.cpp to select between throughput and latency benchmarking based on the new environment variable. [1] [2]
• Updated reporting in hccl_demo.cpp to distinguish between latency and throughput benchmarks in the summary output. [1] [2]

Documentation:

• Added a usage example and explanation for latency benchmarking in the README.md, clarifying its purpose and how to enable it.

[RongLei-intel]

@lviner Could you help review this PR? We need this feature to measure the latency for collectives. Thanks!

[gad-arbel]

Hi @RongLei-intel,

Thank you for your contribution. Please note that @lviner is no longer with Intel.

We maintain two copies of this repository: an internal one (where CI runs) and this public repo. Updates are typically made internally first, then merged here. For internal PRs, we require each to be linked to a JIRA ticket.

Could you please open a JIRA ticket and provide more details about the use case for this feature? If you have access to the internal repo, you can copy your commits there. If not, let me know and I can assist with copying them over.

Thank you!