Repository navigation
feat: create_scoops for batched block verification, build fixes for current gcc, GitHub Actions - #1
Merged
Merged
Conversation
- Compile the C code with -fno-strict-aliasing. The SIMD code reads values through pointers of other types (e.g. a uint64_t nonce through a uint32_t pointer). gcc 15 reorders such reads before the writes at -O3, and create_plots then returns wrong plots for every SIMD batch. The binaries shipped so far were built with gcc 8 and are not affected. - Make the per-file Shabal contexts static. They were defined in several files and only linked because older gcc versions merged them (-fcommon); since gcc 10 the link fails with "multiple definition". Every file initializes and uses its own copy. - Declare the exported functions as extern "C" explicitly (same ABI, removes the deprecation warnings). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lanes Block verification needs one scoop per block, and the blocks of a batch have unrelated generators and nonces. create_plots only generates consecutive nonces of one account, and create_scoop computes one nonce at a time with scalar code. create_scoops fills every SIMD lane with its own account id and nonce, runs the same hash chain as noncegen_<simd>, and extracts only the requested scoop per lane (including the PoC2 mirror), skipping the scatter of all 4096 scoops. One template (noncegen_scoops_impl.h) serves SSE2, AVX, AVX2 and AVX-512. Remainders of two or more are padded into one more SIMD batch, a single remainder uses the scalar code. Count and offset are u32 to match Java int under JNA. Single core on an i7-7700HQ: 65 ms per scoop scalar, 15.8 ms with AVX2. Tests compare every SIMD level the CPU supports and the scalar path byte for byte with the single nonce implementation, cover the C export and its error codes, and guard create_plots and findBestDeadline against their scalar references. Tests run optimized, the PoC code is unusably slow otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…al macOS Travis CI no longer runs. The workflow tests and builds linux-x86-64, linux- aarch64 (native ARM runner, NEON) and win32-x86-64, and builds one universal macOS library for Intel and Apple Silicon. Tagged builds attach all native libraries to the GitHub release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- mshabal_hash_fast_avx512f loaded the message with aligned loads, which need 64 byte alignment. The nonce buffers come from malloc and the scoops for findBestDeadline from Java arrays, so create_plots, create_scoops and findBestDeadline crashed (SIGSEGV) on CPUs with AVX-512. Use unaligned loads; on aligned data they cost nothing. The AVX2 code already loads unaligned. - Build the SIMD/NEON libraries before the base library, so write_seed and write_term from common.c resolve: static archives are searched in link order. The NEON build failed to link because of this. - CI prints the CPU extensions of the Linux runners. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cc prints rerun-if-env-changed lines, and once a build script prints any rerun-if line, cargo reruns it only for the listed variables and files. Changed C sources were therefore not recompiled, and new Rust code was linked against stale C archives. Watch build.rs and src explicitly. CI also fails the Linux builds if the shared library has unresolved symbols, which otherwise only show when Java loads the library. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Built on the runner, the libraries required glibc 2.34 (pthread_key_create, stat64 from Rust's std), so they would not load on Ubuntu 20.04, Debian 11 or Raspberry Pi OS bullseye, and Java would fall back to the slow pure Java Shabal. manylinux_2_28 (glibc 2.28) covers Debian 10+, Ubuntu 18.10+, RHEL 8+ and Raspberry Pi OS buster+. CI fails if a newer glibc is required. Version 1.5.0 for the new create_scoops function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review findings before the 1.5.0 release: - create_scoops had no NEON path, so on aarch64 (Raspberry Pi, ARM servers, Apple Silicon) batches ran the scalar code and gained nothing. noncegen_scoops_neon runs the shared template on mshabal_hash_fast_neon, as shabal_neon.c does for the deadline search. The tests compare it with the single nonce reference on the aarch64 CI runners. - The SIMD implementations copy a global Shabal context that only shabal_init() sets up. Called without it, create_scoops returned wrong scoops and no error. create_scoops now calls shabal_init() itself (a std::sync::Once, so a cheap check after the first call). - Documented that -3 also means inconsistent lengths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
Review before the 1.5.0 releaseFixed in 1a6e2ed:
Checked, no change needed:
Known gap: the x86_64 slice of the macOS universal library is built but not tested, because the runner is arm64. It is the same C code as on Linux x86-64, which is tested with SSE2 to AVX-512. CI: green on all four platforms. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Syncing a Signum node is limited by PoC verification: every block needs one scoop, and computing it means generating the whole nonce (8192 chained Shabal256 hashes). The node computes it with
create_scoop, one nonce at a time in scalar code. On an i7-7700HQ, a testnet sync spends about 4–5 cores on this and reaches ~44 blocks/s.The SIMD nonce generators in this library are about 4–5× faster per core, but
create_plotsonly generates consecutive nonces of one account. The blocks of a batch have unrelated generators and nonces.What
Three commits:
fix(build): builds correctly with current gcc-fno-strict-aliasingfor all C code. The SIMD code reads values through pointers of other types, e.g.*(mshabal_u32 *)((char *)&nonce1 + o). gcc 15 reorders these reads before the writes at-O3, andcreate_plotsreturns wrong plots for every SIMD batch. The binaries shipped in signumj (gcc 8, 2020) are correct: I checked them via JNA.static. They only linked because older gcc merged duplicate definitions (-fcommon); since gcc 10 the link fails.extern "C"on all exports (same ABI).feat:create_scoopsnoncegen_scoops_impl.h, serves SSE2/AVX (4 lanes), AVX2 (8) and AVX-512 (16).countand the offset areu32, matching Javaintunder JNA.ci: GitHub Actions instead of TravisBenchmark (single core, i7-7700HQ,
cargo test --release -- --ignored --nocapture bench_create_scoops)create_scoopcode)create_plotsAVX2, same build (comparison)The old gcc-8 binary runs
create_plotsat 12.4 ms per nonce. The difference comes from gcc 15's code generation, not from-fno-strict-aliasing(measured without it: same speed). A follow-up could try clang in CI.Tests
The crate had one test before; it has 14 now, plus a benchmark.
create_scoopsagainst the single-nonce reference, byte for byte. Every SIMD level the CPU supports plus the scalar path; counts 1, 2, lanes−1, lanes, lanes+1, 2·lanes+3; PoC1 and PoC2; scoops 0 and 4095. The same with automatic dispatch.count = 0.create_plotsagainstcreate_plot, andfindBestDeadlineagainst the scalar C implementation.cargo llvm-cov):scoops.rs87.7% lines. What's missing are dispatch arms for SIMD levels this CPU doesn't use, and the ignored benchmark. In total 71% lines (previously one test).Not tested locally: AVX-512 (this CPU doesn't have it; the test runs automatically where the CPU supports it). The CI runs below are the first build of the aarch64, Windows and macOS targets.
Next
curve25519_*.🤖 Generated with Claude Code