Skip to content

feat: create_scoops for batched block verification, build fixes for current gcc, GitHub Actions - #1

Merged
ohager merged 8 commits into
mainfrom
feat/create-scoops
Nov 14, 2026
Merged

ohager merged 8 commits into
mainfrom
feat/create-scoops

Conversation

@ohager

@ohager ohager commented Nov 13, 2026

Copy link
Copy Markdown
Member

Why

Syncing a Signum node is limited by PoC verification: every block needs one scoop, and computing it means generating the whole nonce (8192 chained Shabal256 hashes). The node computes it with create_scoop, one nonce at a time in scalar code. On an i7-7700HQ, a testnet sync spends about 4–5 cores on this and reaches ~44 blocks/s.

The SIMD nonce generators in this library are about 4–5× faster per core, but create_plots only generates consecutive nonces of one account. The blocks of a batch have unrelated generators and nonces.

What

Three commits:

  1. fix(build): builds correctly with current gcc
    • -fno-strict-aliasing for all C code. The SIMD code reads values through pointers of other types, e.g. *(mshabal_u32 *)((char *)&nonce1 + o). gcc 15 reorders these reads before the writes at -O3, and create_plots returns wrong plots for every SIMD batch. The binaries shipped in signumj (gcc 8, 2020) are correct: I checked them via JNA.
    • The per-file Shabal contexts are static. They only linked because older gcc merged duplicate definitions (-fcommon); since gcc 10 the link fails.
    • Explicit extern "C" on all exports (same ABI).
  2. feat: create_scoops
    int32_t create_scoops(const uint64_t* account_ids, const uint64_t* nonces, const uint32_t* scoops,
                          uint32_t count, uint8_t poc_version, uint8_t* scoop_buffer, uint32_t scoop_buffer_offset);
    • Every SIMD lane gets its own account id and nonce, and only the requested scoop is extracted, PoC2 mirror included. That skips the scatter of all 4096 scoops that plotting does.
    • One template, noncegen_scoops_impl.h, serves SSE2/AVX (4 lanes), AVX2 (8) and AVX-512 (16).
    • A remainder of two or more entries is padded into one more SIMD batch; a single remainder uses the scalar code.
    • Returns 0 on success, −1 for a scoop ≥ 4096, −2 if the SIMD code fails (allocation), −3 for null pointers.
    • count and the offset are u32, matching Java int under JNA.
  3. ci: GitHub Actions instead of Travis
    • Test and build on linux-x86-64, linux-aarch64 (native ARM runner, NEON) and win32-x86-64, plus a universal macOS library (x86-64 + arm64).
    • Tags attach all libraries to a release.
    • Until now there was no ARM build at all, so Raspberry Pi nodes fall back to the pure-Java Shabal.

Benchmark (single core, i7-7700HQ, cargo test --release -- --ignored --nocapture bench_create_scoops)

Path ms per scoop
scalar (create_scoop code) 65.1
SSE2 25.4
AVX 23.8
AVX2 15.8
create_plots AVX2, same build (comparison) 15.8

The old gcc-8 binary runs create_plots at 12.4 ms per nonce. The difference comes from gcc 15's code generation, not from -fno-strict-aliasing (measured without it: same speed). A follow-up could try clang in CI.

Tests

The crate had one test before; it has 14 now, plus a benchmark.

  • create_scoops against the single-nonce reference, byte for byte. Every SIMD level the CPU supports plus the scalar path; counts 1, 2, lanes−1, lanes, lanes+1, 2·lanes+3; PoC1 and PoC2; scoops 0 and 4095. The same with automatic dispatch.
  • The C export: offset handling, untouched surrounding bytes, every error code, count = 0.
  • Regression guards: create_plots against create_plot, and findBestDeadline against the scalar C implementation.
  • Coverage (Rust only, cargo llvm-cov): scoops.rs 87.7% lines. What's missing are dispatch arms for SIMD levels this CPU doesn't use, and the ignored benchmark. In total 71% lines (previously one test).

Not tested locally: AVX-512 (this CPU doesn't have it; the test runs automatically where the CPU supports it). The CI runs below are the first build of the aarch64, Windows and macOS targets.

Next

  • signumj: JNA binding, a batch method with Java fallback, the new native libraries (including linux-aarch64), known-answer tests for curve25519_*.
  • signum-node: the CPU verifier threads process blocks in batches.

🤖 Generated with Claude Code

ohager and others added 7 commits November 13, 2026 18:31
- Compile the C code with -fno-strict-aliasing. The SIMD code reads values
  through pointers of other types (e.g. a uint64_t nonce through a uint32_t
  pointer). gcc 15 reorders such reads before the writes at -O3, and
  create_plots then returns wrong plots for every SIMD batch. The binaries
  shipped so far were built with gcc 8 and are not affected.
- Make the per-file Shabal contexts static. They were defined in several files
  and only linked because older gcc versions merged them (-fcommon); since
  gcc 10 the link fails with "multiple definition". Every file initializes and
  uses its own copy.
- Declare the exported functions as extern "C" explicitly (same ABI, removes
  the deprecation warnings).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lanes

Block verification needs one scoop per block, and the blocks of a batch have
unrelated generators and nonces. create_plots only generates consecutive
nonces of one account, and create_scoop computes one nonce at a time with
scalar code.

create_scoops fills every SIMD lane with its own account id and nonce, runs
the same hash chain as noncegen_<simd>, and extracts only the requested scoop
per lane (including the PoC2 mirror), skipping the scatter of all 4096 scoops.
One template (noncegen_scoops_impl.h) serves SSE2, AVX, AVX2 and AVX-512.
Remainders of two or more are padded into one more SIMD batch, a single
remainder uses the scalar code. Count and offset are u32 to match Java int
under JNA.

Single core on an i7-7700HQ: 65 ms per scoop scalar, 15.8 ms with AVX2.

Tests compare every SIMD level the CPU supports and the scalar path byte for
byte with the single nonce implementation, cover the C export and its error
codes, and guard create_plots and findBestDeadline against their scalar
references. Tests run optimized, the PoC code is unusably slow otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…al macOS

Travis CI no longer runs. The workflow tests and builds linux-x86-64, linux-
aarch64 (native ARM runner, NEON) and win32-x86-64, and builds one universal
macOS library for Intel and Apple Silicon. Tagged builds attach all native
libraries to the GitHub release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- mshabal_hash_fast_avx512f loaded the message with aligned loads, which need
  64 byte alignment. The nonce buffers come from malloc and the scoops for
  findBestDeadline from Java arrays, so create_plots, create_scoops and
  findBestDeadline crashed (SIGSEGV) on CPUs with AVX-512. Use unaligned loads;
  on aligned data they cost nothing. The AVX2 code already loads unaligned.
- Build the SIMD/NEON libraries before the base library, so write_seed and
  write_term from common.c resolve: static archives are searched in link order.
  The NEON build failed to link because of this.
- CI prints the CPU extensions of the Linux runners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cc prints rerun-if-env-changed lines, and once a build script prints any
rerun-if line, cargo reruns it only for the listed variables and files.
Changed C sources were therefore not recompiled, and new Rust code was linked
against stale C archives. Watch build.rs and src explicitly.

CI also fails the Linux builds if the shared library has unresolved symbols,
which otherwise only show when Java loads the library.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Built on the runner, the libraries required glibc 2.34 (pthread_key_create,
stat64 from Rust's std), so they would not load on Ubuntu 20.04, Debian 11 or
Raspberry Pi OS bullseye, and Java would fall back to the slow pure Java
Shabal. manylinux_2_28 (glibc 2.28) covers Debian 10+, Ubuntu 18.10+, RHEL 8+
and Raspberry Pi OS buster+. CI fails if a newer glibc is required.

Version 1.5.0 for the new create_scoops function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review findings before the 1.5.0 release:

- create_scoops had no NEON path, so on aarch64 (Raspberry Pi, ARM
  servers, Apple Silicon) batches ran the scalar code and gained nothing.
  noncegen_scoops_neon runs the shared template on mshabal_hash_fast_neon,
  as shabal_neon.c does for the deadline search. The tests compare it
  with the single nonce reference on the aarch64 CI runners.
- The SIMD implementations copy a global Shabal context that only
  shabal_init() sets up. Called without it, create_scoops returned wrong
  scoops and no error. create_scoops now calls shabal_init() itself
  (a std::sync::Once, so a cheap check after the first call).
- Documented that -3 also means inconsistent lengths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ohager

ohager commented Nov 14, 2026

Copy link
Copy Markdown
Member Author

Review before the 1.5.0 release

Fixed in 1a6e2ed:

  1. Wrong scoops without shabal_init(). The SIMD implementations copy a global Shabal context that only shabal_init() sets up. A caller that skipped it got wrong scoops and status 0. signumj initializes first, so the node was not affected, but the C ABI must not depend on that. create_scoops now calls shabal_init() itself (a Once).
  2. No NEON path. On aarch64 (Raspberry Pi, ARM servers, Apple Silicon), batches ran the scalar code and gained nothing. noncegen_scoops_neon runs the shared template on mshabal_hash_fast_neon, as shabal_neon.c does for the deadline search. On the Linux aarch64 and macOS arm64 runners it matches the single nonce reference for 1, 2, 3, 4, 5 and 11 entries, PoC1 and PoC2.
  3. Documented that -3 also means inconsistent lengths.

Checked, no change needed:

  • Buffer bounds: create_scoops writes count * 64 bytes from the offset and trusts the caller for the buffer size, as create_plot/create_scoop do. This is documented. No u32 overflow on 64-bit targets.
  • Alignment: termination strings are vector-typed unions (aligned loads are fine). Messages come from malloc and are read with loadu (AVX2, AVX-512), or are 16-byte aligned (SSE2, AVX, NEON through SSE2NEON).
  • Threads: per call, the context is copied to the stack and the buffers are allocated. Globals are only written in shabal_init (Once). The node calls this from several verifier threads.
  • PoC1/PoC2 scoop halves and the padded last batch: covered by the reference tests for every implementation and remainder size.
  • The new template type-puns only through memcpy and unions, so it does not rely on -fno-strict-aliasing (which is still needed for the old noncegen code).

Known gap: the x86_64 slice of the macOS universal library is built but not tested, because the runner is arm64. It is the same C code as on Linux x86-64, which is tested with SSE2 to AVX-512.

CI: green on all four platforms.

@ohager
ohager merged commit 47839e7 into main Nov 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant