Benchmarks: Add Mixture of Experts Model #679
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #679 +/- ##
==========================================
+ Coverage 86.44% 86.47% +0.03%
==========================================
Files 100 102 +2
Lines 7406 7541 +135
==========================================
+ Hits 6402 6521 +119
- Misses 1004 1020 +16
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a new Mixture of Experts (MoE) model variant using MixtralConfig with two parameter sets (8x7b and 8x22b), along with associated tests and documentation updates. It includes adding version checks for Python, conditional imports, benchmark registrations, and exporting support for the new model.
Reviewed Changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/helper/decorator.py | Added a Python version check decorator for tests. |
| tests/benchmarks/model_benchmarks/test_pytorch_mixtral.py | Introduced tests for the new Mixtral MoE benchmark (8x7b variant). |
| superbench/benchmarks/model_benchmarks/pytorch_mixtral.py | Implemented the Mixtral benchmark model and registered two variants. |
| superbench/benchmarks/model_benchmarks/init.py | Updated module imports and all to conditionally include MoE model. |
| superbench/benchmarks/micro_benchmarks/_export_torch_to_onnx.py | Extended ONNX export support to include Mixtral models. |
| docs/user-tutorial/benchmarks/model-benchmarks.md | Added documentation for MoE models. |
Files not reviewed (1)
- docs/superbench-config.mdx: Language not supported
Comments suppressed due to low confidence (1)
superbench/benchmarks/micro_benchmarks/_export_torch_to_onnx.py:28
- [nitpick] Consider renaming this class to Torch2ONNXExporter to adhere to standard Python CamelCase naming conventions.
class torch2onnxExporter():
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Description Add release note for v0.12.0 # Main Features ## SuperBench Improvement 1. - [x] Update Image Build Pipeline (#659) 2. - [x] Add support for arm64 build (#660) 3. - [x] Upgrade dependency versions in pipeline (#671) 4. - [x] Fix installation and lint issues (#684) 5. - [x] Update Flake8 repo (#683) 6. - [x] Init latest python support. (#687) 7. - [x] Add image build on arm64 arch (#690) 8. - [x] Enhancement of ignoring errors for import pkg_resources (#692) 9. - [x] Update label in the ROCm image build (#693) 10. - [x] Support cuda12.8 for Blackwell arch (#682) 11. - [x] Merge multi-arch image (#696) 12. - [x] Update OS of runner to the latest. (#702) 13. - [x] cuda arch flag for cublaslt (#701) ## Micro-benchmark Improvement 1. - [x] Bug Fix - Fix numa error on grace cpu in gpu-copy (#658) 2. - [x] Dependency - Bump onnxruntime-gpu version from 1.10.0 to 1.12.0 (#663) 3. - [x] Benchmarks: micro benchmarks - add general CPU bandwidth and latency benchmark (#662) 4. - [x] Benchmarks: micro benchmarks - add nvbandwidth build and benchmark (#665 and #669) 5. - [x] Fix stderr message in gpu-copy benchmark (#673) 6. - [x] Add arch support for 10.0 in gemm-flops (#680) 7. - [x] Fix tensorrt-inference parsing (#674) 8. - [x] nvbandwidth benchmark need to handle N/A value (#675) 9. - [x] Avoid Unintended nvbandwidth Function Calls in All Benchmarks (#685) 10. - [x] Add GPU Stream Micro Benchmark (#697) 11. - [x] Cuda arch flag for cublaslt (#701) 12. - [x] Support autotuning in cublaslt gemm (#706) 14. - [x] Add FP4 GEMM FLOPS support for cublaslt_gemm benchmark (#711) 15. - [x] CPU Stream Benchmark Revise (#712) 16. - [x] Add cuda12.9 docker image (#716) 17. - [x] Add Grace CPU support for CPU Stream (#719) ## Model Benchmark Improvement 1. - [x] Add LLaMA-2 Models (#668) 2. - [x] Fix typos in documentation and code files (#686) 3. - [x] Add Mixture of Experts Model (#679) 4. - [ ] Add DeepSeek Training Benchmark 5. - [x] Add DeepSeek Inference Benchmark (AMD GPU) (#713) ## Documentation 1. - [x] Update CODEOWNERS (#670) 2. - [x] Update CODEOWNERS (#718) ## Result Analysis 1. - [x] Enhance logging information for diagnosis rule op baseline errors. (#689)
Description Add release note for v0.12.0 # Main Features ## SuperBench Improvement 1. - [x] Update Image Build Pipeline (#659) 2. - [x] Add support for arm64 build (#660) 3. - [x] Upgrade dependency versions in pipeline (#671) 4. - [x] Fix installation and lint issues (#684) 5. - [x] Update Flake8 repo (#683) 6. - [x] Init latest python support. (#687) 7. - [x] Add image build on arm64 arch (#690) 8. - [x] Enhancement of ignoring errors for import pkg_resources (#692) 9. - [x] Update label in the ROCm image build (#693) 10. - [x] Support cuda12.8 for Blackwell arch (#682) 11. - [x] Merge multi-arch image (#696) 12. - [x] Update OS of runner to the latest. (#702) 13. - [x] cuda arch flag for cublaslt (#701) ## Micro-benchmark Improvement 1. - [x] Bug Fix - Fix numa error on grace cpu in gpu-copy (#658) 2. - [x] Dependency - Bump onnxruntime-gpu version from 1.10.0 to 1.12.0 (#663) 3. - [x] Benchmarks: micro benchmarks - add general CPU bandwidth and latency benchmark (#662) 4. - [x] Benchmarks: micro benchmarks - add nvbandwidth build and benchmark (#665 and #669) 5. - [x] Fix stderr message in gpu-copy benchmark (#673) 6. - [x] Add arch support for 10.0 in gemm-flops (#680) 7. - [x] Fix tensorrt-inference parsing (#674) 8. - [x] nvbandwidth benchmark need to handle N/A value (#675) 9. - [x] Avoid Unintended nvbandwidth Function Calls in All Benchmarks (#685) 10. - [x] Add GPU Stream Micro Benchmark (#697) 11. - [x] Cuda arch flag for cublaslt (#701) 12. - [x] Support autotuning in cublaslt gemm (#706) 14. - [x] Add FP4 GEMM FLOPS support for cublaslt_gemm benchmark (#711) 15. - [x] CPU Stream Benchmark Revise (#712) 16. - [x] Add cuda12.9 docker image (#716) 17. - [x] Add Grace CPU support for CPU Stream (#719) ## Model Benchmark Improvement 1. - [x] Add LLaMA-2 Models (#668) 2. - [x] Fix typos in documentation and code files (#686) 3. - [x] Add Mixture of Experts Model (#679) 4. - [ ] Add DeepSeek Training Benchmark 5. - [x] Add DeepSeek Inference Benchmark (AMD GPU) (#713) ## Documentation 1. - [x] Update CODEOWNERS (#670) 2. - [x] Update CODEOWNERS (#718) ## Result Analysis 1. - [x] Enhance logging information for diagnosis rule op baseline errors. (#689)
Added MoE model using MixtralConfig.