Support endpoints results for submission generation#1029
Open
anandhu-eng wants to merge 11 commits into
Open
Conversation
- generate-mlperf-inference-submission: detect endpoint runs (run_metadata.json present, no mlperf_log files), copy and rename result files to match checker expectations (results_summary.json → result_summary.json), build result strings from endpoint metadata (QPS, TTFT P99) - run-mlperf-inference-submission-checker: prepend tools/submission/ to PYTHONPATH when using the modular checker so relative imports in submission_checker/main.py resolve correctly without requiring -m invocation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…n detection - Remove ENDPOINTS_PERF_RENAME workaround; checker and harness now agree on results_summary.json (inference repo PR #2622) - Update ENDPOINTS_ACC_FILES to use results_summary.json - Replace fragile negative detection (run_metadata.json present + no mlperf_log_summary.txt) with positive check for all three files the endpoints harness always writes: config.yaml, results_summary.json, results.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Merge ENDPOINTS_PERF_FILES + ENDPOINTS_ACC_FILES into one ENDPOINTS_FILES list (run_metadata.json is optional, other files are the same for both modes) - Simplify is_endpoints_run to check only for config.yaml, which is written by the endpoints harness and absent from all standard loadgen runs - Remove stale ENDPOINTS_PERF_RENAME copy block (no rename needed since harness and checker agree on results_summary.json) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolved conflict in generate-mlperf-inference-submission/customize.py, preserving unified ENDPOINTS_FILES list, config.yaml-only run detection, and removal of ENDPOINTS_PERF_RENAME copy workaround. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ata.json run_metadata.json is optional; use the two files that are always written by the endpoints harness to validate a perf/run_1 directory is complete. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
Avoids ModuleNotFoundError when a stale submission_checker package exists in site-packages by cd-ing into tools/submission/ and running via -m, which sets sys.path[0] to the correct package root before any other entry can shadow it. Also handles the SHORT_RUN modified copy (submission_checker1) via relpath-derived module name. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…aults - generate-mlperf-inference-submission/customize.py: read accuracy scores from the nested 'accuracy_scores' key in endpoints results.json instead of iterating the top-level dict (which yielded N/A for all scores) - generate-mlperf-inference-submission/meta.yaml: bump wg-inference default MLC_MLPERF_SUBMISSION_CHECKER_VERSION from v5.1 to v6.1 so --run_checker=yes validates against the correct round without requiring an explicit --version flag - run-mlperf-inference-submission-checker/customize.py: remove cd-based module invocation for modularised checker; relative --input paths broke after cd shifted the working directory Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
Author
|
Modifications based on the requirement for supporting endpoints folder structure detailed here: https://github.com/mlcommons/inference/blob/master/tools/submission/submission_structure.md |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🧾 PR Checklist
dev📌 Note: PRs must be raised against
dev. Do not commit directly tomain.📁 File Hygiene & Output Handling
📝 Comments & Communication
Fixes #,Related to #, etc.)🛡️ Safety & Security