Tests - Add basic utilities for module-level numerical tests#75
Open
yzygitzh wants to merge 2 commits into
Open
Tests - Add basic utilities for module-level numerical tests#75yzygitzh wants to merge 2 commits into
yzygitzh wants to merge 2 commits into
Conversation
yzygitzh
changed the title
There was a problem hiding this comment.
Pull Request Overview
This PR adds basic utilities for module-level numerical tests to enable consistent testing and comparison of module behavior across different configurations. The utilities provide infrastructure for running numerical tests, capturing module statistics, and comparing results between different test runs.
Key changes include:
- Test infrastructure with base classes and configuration utilities
- Statistical analysis tools for computing means, standard deviations, and similarity metrics
- Environment variable management to preserve user NVTE settings during testing
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
tests/numerical_tests/modules/conftest.py |
Pytest configuration with result directory option and cleanup fixtures |
tests/numerical_tests/modules/test_module.py |
Base test class for module-level numerical tests with distributed setup and result saving |
tests/numerical_tests/modules/test_utilities.py |
Enhanced Utils class that preserves NVTE environment variables during testing |
tests/numerical_tests/utils/module_mean_and_std.py |
Statistical computation utility using Welford's algorithm for streaming mean/std calculation |
tests/numerical_tests/utils/module_similarity.py |
Cosine similarity calculation tool for comparing tensor statistics between test runs |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
cp5555
reviewed
Jul 28, 2025
| with open(args.output_file, 'w') as f: | ||
| json.dump(comparison_result, f, indent=2) | ||
|
|
||
| if __name__ == '__main__': |
cp5555
reviewed
Jul 28, 2025
| torch.save(mean_result, args.output_mean_file) | ||
| torch.save(std_result, args.output_std_file) | ||
|
|
||
| if __name__ == '__main__': |
cp5555
reviewed
Jul 28, 2025
|
|
||
|
|
||
| def pytest_addoption(parser): | ||
| parser.addoption( |
There was a problem hiding this comment.
Why do we need to add required option to run the tests
cp5555
reviewed
Jul 28, 2025
| bf16=config.bf16, | ||
| use_distributed_optimizer=True, | ||
| lr=1e-3, | ||
| clip_grad=0.0 |
There was a problem hiding this comment.
Why is there hard code for adam, lr, etc.
cp5555
approved these changes
Jul 28, 2025
abuccts
reviewed
Jul 28, 2025
Comment on lines
+28
to
+29
| seed = 42 | ||
| torch.manual_seed(seed) |
Member
There was a problem hiding this comment.
do you need to set torch.cuda.manual_seed_all? and seed in python/numpy as well
| model_parallel_cuda_manual_seed(seed) | ||
|
|
||
| def teardown_method(self, method): | ||
| Utils.destroy_model_parallel() |
Comment on lines
+65
to
+66
| torch.save(mean_result, args.output_mean_file) | ||
| torch.save(std_result, args.output_std_file) |
Member
There was a problem hiding this comment.
why not return the value directly?
|
Marking as stale. No activity in 60 days. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds basic utilities for module-level numerical tests, in tests/numerical_tests folder. Including: