fix: skip packages without C benchmarks in benchmark-random-c#13138
Draft
Planeshifter wants to merge 1 commit into
Draft
fix: skip packages without C benchmarks in benchmark-random-c#13138Planeshifter wants to merge 1 commit into
Planeshifter wants to merge 1 commit into
Conversation
Contributor
|
Hello! Thank you for your contribution to stdlib. We noticed that the contributing guidelines acknowledgment is missing from your pull request. Here's what you need to do:
This acknowledgment confirms that you've read the guidelines, which include:
We can't review or accept contributions without this acknowledgment. Thank you for your understanding and cooperation. We look forward to reviewing your contribution! |
`benchmark-random-c` uses `PACKAGES_PATTERN='manifest.json'` to select random packages. WASM packages (e.g. `@stdlib/blas/base/wasm/*`) carry a `manifest.json` listing their C dependencies but ship no `benchmark/c/*.c` file and require Emscripten to compile — unavailable in CI. When one of these packages was selected, `benchmark-c` failed, causing the `random_benchmarks` CI workflow to fail on ~97% of runs (29/30 in 30 days). Add a pre-flight guard that checks for `*.c` files inside the package's `benchmark/` directory before invoking `benchmark-c`. Packages without C benchmarks are silently skipped, leaving the `manifest.json` selection pool intact while eliminating the false-positive failures.
54b523d to
ecd0070
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes persistent
random_benchmarksCI failures (~97% failure rate, 29/30 runs in the last 30 days).Root cause
benchmark-random-cusesPACKAGES_PATTERN='manifest.json'to select random packages. WASM packages (e.g.@stdlib/blas/base/wasm/dasum,@stdlib/blas/base/wasm/dswap) carry amanifest.jsonlisting their C dependencies but ship nobenchmark/c/*.cfile and require Emscripten to compile — which is not available in the CI runner. When one of these 47 WASM packages was selected from the pool of 1845,benchmark-cfailed and the workflow aborted.The very high failure rate (97%) reflects that at least one WASM package is statistically nearly certain to appear in 10 randomly drawn packages from 1845 when 47 are problematic.
Fix
Add a pre-flight
findguard before callingbenchmark-c. If a selected package has no*.cfiles under itsbenchmark/directory, it is silently skipped.This keeps the original
PACKAGES_PATTERN='manifest.json'selection pool (changing it tobenchmark*.cwould be excluded byFIND_LIB_PACKAGES_EXCLUDE_FLAGSwhich strips**/benchmark/*paths, silently returning 0 packages and running nothing).Validation
*.cinbenchmark/(confirmed), math/blas packages doFIND_LIB_PACKAGES_EXCLUDE_FLAGSexcludesbenchmark/subdirs, ruling out the pattern-change approach