Skip to content

fix: skip packages without C benchmarks in benchmark-random-c#13138

Draft
Planeshifter wants to merge 1 commit into
developfrom
philipp/ci-fix-random-benchmarks-c-2026-06-25
Draft

fix: skip packages without C benchmarks in benchmark-random-c#13138
Planeshifter wants to merge 1 commit into
developfrom
philipp/ci-fix-random-benchmarks-c-2026-06-25

Conversation

@Planeshifter

@Planeshifter Planeshifter commented Jun 25, 2026

Copy link
Copy Markdown
Member

Summary

Fixes persistent random_benchmarks CI failures (~97% failure rate, 29/30 runs in the last 30 days).

Root cause

benchmark-random-c uses PACKAGES_PATTERN='manifest.json' to select random packages. WASM packages (e.g. @stdlib/blas/base/wasm/dasum, @stdlib/blas/base/wasm/dswap) carry a manifest.json listing their C dependencies but ship no benchmark/c/*.c file and require Emscripten to compile — which is not available in the CI runner. When one of these 47 WASM packages was selected from the pool of 1845, benchmark-c failed and the workflow aborted.

The very high failure rate (97%) reflects that at least one WASM package is statistically nearly certain to appear in 10 randomly drawn packages from 1845 when 47 are problematic.

Fix

Add a pre-flight find guard before calling benchmark-c. If a selected package has no *.c files under its benchmark/ directory, it is silently skipped.

 benchmark-random-c: $(NODE_MODULES)
 	$(QUIET) $(MAKE) -f $(this_file) -s list-random-lib-pkgs PACKAGES_PATTERN='manifest.json' | while read -r pkg; do \
+		if find "$$pkg/benchmark" -name "*.c" 2>/dev/null | grep -q .; then \
 			echo ""; \
 			echo "Running benchmark: $$pkg"; \
 			NODE_ENV="$(NODE_ENV_BENCHMARK)" \
 			NODE_PATH="$(NODE_PATH_BENCHMARK)" \
 			$(MAKE) -f $(this_file) benchmark-c BENCHMARKS_FILTER="$$pkg/.*" || exit 1; \
+		fi; \
 	done

This keeps the original PACKAGES_PATTERN='manifest.json' selection pool (changing it to benchmark*.c would be excluded by FIND_LIB_PACKAGES_EXCLUDE_FLAGS which strips **/benchmark/* paths, silently returning 0 packages and running nothing).

Validation

  • Guard tested: WASM packages have no *.c in benchmark/ (confirmed), math/blas packages do
  • 3-agent review caught that FIND_LIB_PACKAGES_EXCLUDE_FLAGS excludes benchmark/ subdirs, ruling out the pattern-change approach

@stdlib-bot

Copy link
Copy Markdown
Contributor

Hello! Thank you for your contribution to stdlib.

We noticed that the contributing guidelines acknowledgment is missing from your pull request. Here's what you need to do:

  1. Please read our contributing guidelines.

  2. Update your pull request description to include this checked box:

    - [x] Read, understood, and followed the [contributing guidelines](https://github.com/stdlib-js/stdlib/blob/develop/CONTRIBUTING.md)

This acknowledgment confirms that you've read the guidelines, which include:

  • The developer's certificate of origin
  • Your agreement to license your contributions under the project's terms

We can't review or accept contributions without this acknowledgment.

Thank you for your understanding and cooperation. We look forward to reviewing your contribution!

`benchmark-random-c` uses `PACKAGES_PATTERN='manifest.json'` to select
random packages. WASM packages (e.g. `@stdlib/blas/base/wasm/*`) carry a
`manifest.json` listing their C dependencies but ship no `benchmark/c/*.c`
file and require Emscripten to compile — unavailable in CI. When one of
these packages was selected, `benchmark-c` failed, causing the
`random_benchmarks` CI workflow to fail on ~97% of runs (29/30 in 30 days).

Add a pre-flight guard that checks for `*.c` files inside the package's
`benchmark/` directory before invoking `benchmark-c`. Packages without C
benchmarks are silently skipped, leaving the `manifest.json` selection
pool intact while eliminating the false-positive failures.
@Planeshifter Planeshifter force-pushed the philipp/ci-fix-random-benchmarks-c-2026-06-25 branch from 54b523d to ecd0070 Compare June 25, 2026 14:54
@Planeshifter Planeshifter changed the title fix: select C benchmark packages by benchmark*.c rather than manifest.json fix: skip packages without C benchmarks in benchmark-random-c Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants