Skip to content

RELEASE_EVIDENCE.md

Latest commit

 

History

History
584 lines (499 loc) · 26.2 KB

RELEASE_EVIDENCE.md

File metadata and controls

584 lines (499 loc) · 26.2 KB
Raw

Release Evidence

base64-ng treats release evidence as part of the artifact, not as an informal local habit. The release gate generates and verifies evidence that downstream users can inspect before adopting a version.

Run the gate with:

scripts/stable_release_gate.sh release

release mode rejects pre-release Cargo versions. Use scripts/stable_release_gate.sh check for development snapshots.

The published crate package includes the core local gate scripts, Rust toolchain pin, and cargo-deny policy referenced by this document, so downstream reviewers can inspect the release checks alongside the source and documentation.

Install the optional targets and Cargo tools that make the release gate exercise the deepest local paths:

rustup target add x86_64-unknown-linux-gnu aarch64-unknown-linux-gnu x86_64-unknown-freebsd wasm32-unknown-unknown thumbv7em-none-eabihf
cargo install --locked cargo-nextest --version 0.9.140
cargo install --locked cargo-fuzz --version 0.13.2
cargo install --locked kani-verifier --version 0.67.0

cargo-fuzz and Miri use nightly components:

rustup toolchain install nightly --component miri
cargo +nightly miri setup

Required Checks

The release gate runs:

  • formatting checks
  • release metadata validation
  • documentation version consistency checks for README, changelog, and SIMD release-status docs
  • MSRV/toolchain policy validation for Cargo.toml, rust-toolchain.toml, docs.rs metadata, CI install paths, target matrices, and release-evidence tooling
  • public API audit validation; stable releases fail if public API rows remain marked as review pending
  • packaged release script presence, executable-bit, and shebang validation
  • zero-dependency policy check for the published crate
  • packaged dependency admission policy for future external-crate review
  • dependency admission review keeping the core crate dependency-free while treating base64-ng-serde, base64-ng-bytes, base64-ng-subtle, and base64-ng-tokio as separately reviewed optional companion crates; zeroize, property-testing, and benchmark frameworks remain out of the core package unless separately admitted
  • reserved feature placeholder checks for tokio, kani, and fuzzing, including inert-feature and per-feature dependency graph validation
  • fail-closed wasm wipe policy check proving default wasm32 builds reject compiler-fence-only cleanup and the explicit allow-wasm32-best-effort-wipe opt-in build succeeds
  • wasm SIMD posture validation proving the narrow wasm simd128 runtime dispatch profile, its feature gates, its runtime smoke evidence, and its remaining JIT/zeroization caveats
  • wasm SIMD codegen evidence through scripts/generate_wasm_simd_evidence.sh, which emits release test-harness LLVM IR with target-feature=+simd128 when the wasm target is installed and checks for vector shuffle, 128-bit byte-vector, and wasm bitselect markers while leaving runtime/JIT behavior to the separate runtime smoke gate
  • wasm simd128 runtime smoke evidence through scripts/check_wasm_runtime_dispatch.sh, which builds a wasm32 smoke module with target-feature=+simd128 and executes it under Node/V8 and Wasmtime when installed, requiring wasm-simd128 active encode/decode reporting and Standard plus URL-safe deterministic length sweeps, independent scalar reference encode checks, malformed-input rejection, and round trips
  • strict decode public-surface evidence through standard_family_decode_surfaces_cover_tails_and_padding and standard_family_decode_error_surfaces_match_scalar, proving Standard and URL-safe padded/unpadded slice, clear-tail, stack-buffer, vec, and secret helpers match the scalar reference for accepted input and rejected input while clear-tail APIs wipe caller buffers on errors
  • wasm simd128 browser smoke evidence through scripts/check_wasm_browser_dispatch.sh, which executes the same wasm32 smoke module in a Chromium-family browser when installed or when BASE64_NG_BROWSER points to a compatible browser binary; the gate first proves its exact success attribute is absent from static HTML and then requires that runtime-created attribute in the browser's dumped DOM
  • wasm simd128 Firefox/SpiderMonkey smoke evidence through scripts/check_wasm_browser_firefox_dispatch.sh, which executes the same wasm32 smoke module through geckodriver when Firefox is installed
  • Safari/WebKit WebDriver smoke evidence through scripts/check_wasm_browser_safari_dispatch.sh; the 1.3.3 release evidence includes a macOS pass with /System/Cryptexes/App/usr/bin/safaridriver
  • fail-closed unsupported-native wipe policy documented through allow-compiler-fence-only-wipe for architectures without an implemented hardware wipe barrier
  • clippy with warnings denied
  • default, all-features, and no-default-features tests
  • CI platform tests on Linux, Windows, pinned macOS ARM images (macos-15, macos-26), pinned Intel macOS (macos-15-intel), and macos-latest as a moving-label migration signal
  • local macOS host verification through scripts/check_macos.sh, which runs the full host test/clippy set and compile-checks both Apple Darwin triples
  • local AArch64 Linux host verification through scripts/check_aarch64_linux.sh, which runs the full host test/clippy set, NEON encode block evidence, backend evidence, SIMD feature-bundle checks, and SIMD admission validators on real ARM Linux hardware
  • big-endian QEMU user-mode verification through scripts/check_big_endian_qemu.sh, which runs s390x-unknown-linux-gnu no-std portability checks and QEMU-executed RFC4648, buffer, backend fallback, malformed-input, clear-tail, in-place, wrapped/legacy, and stream tests. This is functional correctness and scalar/fallback evidence under emulation only; it is not real hardware performance, timing, microarchitectural, or side-channel evidence. The script records that community reports from real s390x, powerpc64, and big-endian AArch64 systems are still requested before any backend is upgraded from QEMU-tested to hardware-attested.
  • RISC-V QEMU user-mode verification through scripts/check_riscv_qemu.sh, which runs riscv64gc-unknown-linux-gnu no-std portability checks and QEMU-executed RFC4648, buffer, backend fallback, malformed-input, clear-tail, in-place, wrapped/legacy, strict decode dispatch surface, and stream tests. This is functional correctness and scalar/fallback evidence under emulation only; it is not RVV hardware performance, timing, microarchitectural, register-retention, or side-channel evidence. The script records that community reports from real RVV 1.0 systems are still requested before any RISC-V backend is upgraded from QEMU-tested to hardware-attested. The stronger RVV proof and backend-admission review is scheduled for 1.3.10; 1.3.9 preserves scalar active dispatch.
  • moved-code review for the src/alphabet.rs extraction, preserving root public exports for built-in alphabets, custom alphabet validation, and the define_alphabet! macro
  • moved-code review for the src/profiles.rs extraction, preserving root public exports for Profile and the named MIME/PEM/bcrypt/crypt profiles
  • moved-code review for the src/cleanup.rs extraction, preserving internal cleanup call paths and updating the unsafe-boundary gate for the new audited unsafe location
  • moved-code review for the src/buffers/ extraction, preserving root public exports for stack-backed buffers, exposed ownership wrappers, and SecretBuffer
  • all-features and no-default-features doctests plus documentation builds
  • cargo deny check
  • cargo audit
  • daily and manually dispatchable RustSec plus cargo-deny advisory monitoring for the workspace and isolated fuzz, dudect, and performance lockfiles
  • cargo license --json
  • async admission documentation packaged while the tokio feature remains inert and dependency-free
  • Miri through scripts/check_miri.sh when nightly Miri is installed, covering no-default-features scalar APIs and all-features alloc/stream APIs and writing a release evidence manifest
  • fuzz target compile check when cargo-fuzz is installed
  • fuzz corpus policy validation for target-specific reviewed corpus inputs and release-blocking artifact cleanup
  • isolated dudect, fuzz, and performance harness dependency checks as part of the standard gate
  • installed-target no_std checks for the reserved simd feature
  • no-alloc portability smoke crate checks for stack-backed encode/decode, wrapped output, URL-safe no-padding, and constant-time-oriented decode with default features disabled, plus validate-only, legacy decode, in-place encode/decode, scalar and constant-time clear-tail cleanup, constant-time-oriented in-place decode, named MIME/PEM/bcrypt/crypt profiles, custom alphabet/profile, recoverable length, stack-buffer state surfaces, and native byte-array and FromStr interop surfaces; the harness also runs host-side unit tests before cross-target compile checks
  • Standard-family encode surface tests covering encode_slice, encode_slice_clear_tail, stack buffers, and alloc helpers for every input length from 0 through 193 bytes, including fixed-block thresholds, all tail lengths, and padded or unpadded output
  • Local and CI target-matrix no-alloc portability smoke checks so installed Linux, FreeBSD, wasm32, ARM, and Cortex-M targets compile the same stack-backed dependency-free harness
  • migration-guide smoke tests for strict standard, URL-safe no-pad, MIME/PEM, legacy whitespace, custom alphabet, stack-buffer, secret-buffer, and stream migration examples
  • reserved SIMD feature-bundle compile checks for AVX2, AVX-512 VBMI, SSSE3/SSE4.1, NEON, and wasm simd128 under no_std when the corresponding Rust targets are installed
  • backend evidence capture for runtime backend reporting, admitted AVX-512 VBMI, AVX2, SSSE3/SSE4.1, or NEON encode dispatch when supported, and admitted strict decode dispatch when supported
  • Standard-family strict decode surface tests covering decode_slice, decode_slice_clear_tail, stack buffers, and alloc helpers for every input length from 0 through 193 bytes after scalar-reference encode, including fixed-block thresholds, short inputs, non-block tails, and padded or unpadded input
  • SIMD admission policy for the current release series, with AVX-512 VBMI, AVX2, SSSE3/SSE4.1, and NEON encode admitted only for std x86/x86_64 or little-endian std aarch64 Standard and URL-safe alphabets, AVX-512 VBMI, AVX2, SSSE3/SSE4.1, and NEON strict decode admitted only for std x86/x86_64 or little-endian std aarch64 Standard and URL-safe alphabets, and no SIMD performance claims without complete local benchmark evidence
  • unsafe-boundary validation that confines allow(unsafe_code) to the audited cleanup helpers in src/cleanup.rs, CT barrier/comparison helpers in src/ct/, and the SIMD boundary in src/simd/
  • unsafe-boundary validation that confines inline assembly to the cleanup and CT barriers and confines CPU feature detection and target_feature gates to src/simd/
  • unsafe-boundary validation that requires inventory documentation for every SIMD-boundary unsafe function and a nearby SAFETY: explanation for every unsafe block
  • panic-policy validation that fails on unreviewed non-test panic!, unreachable!, .unwrap(), or .expect() sites
  • constant-time policy validation that keeps non-claim wording and generated-code review requirements in the documented release bar
  • dudect-style timing harness compile and dependency checks, with timing runs opt-in for local release evidence
  • constant-time assembly evidence generation for no-default-features and all-features release builds
  • runtime backend report tests proving the public active backend remains scalar until an accelerated backend is explicitly admitted
  • runtime backend policy tests for scalar execution and no-SIMD deployment assertions
  • high-assurance scalar-only backend policy tests
  • stable runtime enum string identifier tests for audit-friendly evidence
  • stable key/value runtime report and policy-failure formatting tests
  • constant-time-oriented clear-tail decode tests for success, malformed input, undersized output, and in-place cleanup
  • constant-time-oriented validate/decode agreement tests for valid and malformed inputs across supported alphabets and padding modes
  • stream encoder and decoder tests proving policy accessors, state accessors, finish(), try_finish(), into_inner(), and adjacent-payload behavior remain intact after cleanup hardening
  • stream encoder and decoder retry tests proving pending input survives wrapped writer failures, and finalization flush retries do not re-emit terminal encoded or decoded bytes
  • stream encoder and decoder short-write tests proving buffered writer output is retained until the wrapped writer reports bytes accepted
  • stream reader output queues drain into caller buffers in bounded slices while consumed queue slots are cleared
  • stream decoder fail-closed tests proving malformed Base64 input poisons the adapter while preserving explicit unchecked inner recovery
  • stream fuzz coverage for chunked writers, fragmented reader sources, and adjacent framed payload boundaries, including fail-closed decoder state invariants after malformed input
  • profile and custom-alphabet fuzz coverage for MIME, PEM, bcrypt-style, crypt(3)-style, and caller-defined alphabets
  • opt-in bounded fuzz smoke evidence through BASE64_NG_RUN_FUZZ_SMOKE=1 scripts/check_fuzz.sh
  • generated constant-time assembly artifacts through scripts/generate_ct_asm_evidence.sh
  • manual generated-code review checklist in CT_ASM_REVIEW.md
  • LTO symbol-presence checks for non-inlined wipe boundaries and the constant_time_eq_public_len equal-length comparison helper; the parser accepts both legacy Rust symbols and the v0 symbols enabled by default in Rust 1.97.1
  • Kani proofs through scripts/check_kani.sh; current local evidence is the full no-default-features harness set on the Rust 1.90.0 Kani toolchain with cargo-kani 0.67.0
  • bounded Kani coverage for constant-time-oriented decode result bounds, clear-tail cleanup on error, and validate/decode agreement
  • bounded-index invariant documentation in INVARIANTS.md
  • explicit Kani compatibility or verifier-exception documentation in KANI.md if a future installed Kani compiler cannot run the proofs
  • the historical initial 1.0.0 Kani verifier exception is superseded for the current bounded harness set by the clean scripts/check_kani.sh run above; future verifier incompatibility must be documented explicitly rather than treated as proof
  • SBOM generation
  • reproducible package/build check

Generated Artifacts

Evidence is written under:

target/release-evidence/

Expected files:

  • base64-ng.spdx.json
  • base64-ng.cyclonedx.json
  • sbom-MANIFEST.txt
  • backend/MANIFEST.txt
  • backend/runtime-backend-report.txt
  • backend/simd-prototype-equivalence.txt
  • asm/MANIFEST.txt
  • asm/base64_ng-no-default-features.s
  • asm/base64_ng-all-features.s

The SBOMs and sbom-MANIFEST.txt describe the published crate dependency graph and record tool versions, commands, and checksums. The normal published crate is zero-dependency; fuzz-only dependencies live under fuzz/ and are reviewed separately.

Fuzz-Only Dependency Evidence

The fuzz harness is intentionally isolated from the published crate. Review it with:

scripts/check_fuzz.sh

fuzz/deny.toml allows the NCSA license only for libfuzzer-sys. The root deny.toml remains stricter for the published crate.

scripts/check_fuzz.sh explicitly runs:

cargo audit --file fuzz/Cargo.lock
scripts/cargo-deny-check.sh fuzz/Cargo.toml fuzz/deny.toml

The differential fuzz target includes static RFC 4648 ground-truth vectors in addition to comparison against the established base64 crate oracle.

The stream_chunks fuzz target covers:

  • chunked streaming encoders and decoders
  • fragmented EncoderReader sources compared with slice encoding
  • fragmented DecoderReader sources compared with slice decoding when payload boundary semantics match
  • padded DecoderReader payloads followed by adjacent framed bytes, proving the reader leaves those bytes unread
  • stream state-helper invariants for pending quanta, buffered output capacity, recovery readiness, and terminal input state

Run a bounded local smoke test with:

cargo +nightly fuzz run stream_chunks -- -runs=1000

LibFuzzer may generate local corpus files under fuzz/corpus/; review them before committing and discard accidental local corpus churn.

SIMD Feature-Bundle Evidence

Reserved SIMD code must compile under the feature bundles that future admitted backends will rely on. Check installed SIMD feature bundles with:

scripts/check_simd_feature_bundles.sh

This currently proves no_std reserved builds for AVX2, SSSE3/SSE4.1, the AVX-512 Base64 candidate bundle (avx512f, avx512bw, avx512vl, and avx512vbmi), NEON, and wasm simd128 when the corresponding Rust targets are installed. For wasm simd128, the script also builds the wasm test binaries with target-feature=+simd128 so the admitted fixed-block wasm code remains typechecked and codegen-ready.

Capture local runtime backend and prototype evidence with:

scripts/check_backend_evidence.sh

The script runs the runtime backend-report test and the gated SIMD scalar-equivalence tests with --nocapture. The runtime report records candidate_detection_mode, which distinguishes x86/x86_64 std runtime CPU probing from compile-time target-feature reporting used by no_std and other compile-time-only targets. On CPUs with AVX-512 VBMI, AVX2, SSSE3/SSE4.1, or little-endian AArch64 NEON, or wasm simd128, an admitted encode path may be active for Standard and URL-safe alphabets. Big-endian AArch64 stays scalar, and 32-bit ARM NEON remains scaffold evidence. Wasm simd128 evidence is kept in scripts/check_simd_feature_bundles.sh as compile/test-binary evidence and in scripts/check_wasm_runtime_dispatch.sh as Node/V8 and Wasmtime runtime smoke evidence, in scripts/check_wasm_browser_dispatch.sh as Chromium-family browser smoke evidence, in scripts/check_wasm_browser_firefox_dispatch.sh as Firefox/SpiderMonkey smoke evidence, and in scripts/check_wasm_browser_safari_dispatch.sh as Safari/WebKit smoke evidence. Runtime/JIT timing behavior remains outside the crate's formal claim. The script writes target/release-evidence/backend/MANIFEST.txt, runtime-backend-report.txt, and simd-prototype-equivalence.txt so local CPU evidence can be archived. The manifest labels prototype-only evidence as real-non-dispatchable and separately records active_backend_admitted=avx512-vbmi-or-avx2-or-ssse3-sse4.1-or-neon-or-wasm-simd128-encode, so audit logs do not confuse remaining fixed-block prototype execution with active dispatch admission.

The release gate also runs:

scripts/validate-simd-admission.sh
scripts/validate-simd-encode-admission-draft.sh

That validator keeps active SIMD dispatch limited to admitted backends until a release includes the required scalar differential tests, fuzz evidence, unsafe inventory updates, architecture evidence, benchmark evidence, release-note wording, and an updated docs/SIMD_ADMISSION.md manifest.

For a future encode-dispatch release, use SIMD_ENCODE_ADMISSION_DRAFT.md as the working package. It defines the runtime-report expectations, benchmark record, release-note wording, and decision checklist required before any encode backend can move from real non-dispatchable prototype evidence to active dispatch. scripts/validate-simd-encode-admission-draft.sh keeps that draft packaged and checks that the future admission contract still names the required runtime, fallback, benchmark, release-note, and architecture-specific evidence.

Miri Evidence

Run Miri coverage with:

scripts/check_miri.sh

When nightly Miri is installed, the script runs no-default-features and all-features test surfaces and writes target/release-evidence/miri/MANIFEST.txt, no-default-features.txt, and all-features.txt. This evidence is useful for release review of the dependency-free scalar core, alloc helpers, stream wrappers, and cleanup helpers. It remains tool-backed undefined-behavior evidence, not a formal proof.

Constant-Time Timing Evidence

The standard local gate, normal CI gate, and release gate compile the isolated dudect-style harness and check its dependency policy:

scripts/check_dudect.sh

Timing measurements are opt-in because shared CI runners are not stable enough for reliable side-channel statistics:

BASE64_NG_RUN_DUDECT=1 scripts/check_dudect.sh

Archive the raw output with CPU, OS, Rust version, sample count, and command line when using dudect-style evidence for a security review. Opt-in timing runs write target/release-evidence/dudect/dudect-output.txt and target/release-evidence/dudect/MANIFEST.txt for this purpose. This evidence is empirical and does not replace generated-code review or Kani proofs.

The release gate also generates assembly artifacts for reviewer inspection with:

scripts/generate_ct_asm_evidence.sh

The script writes target/release-evidence/asm/base64_ng-no-default-features.s, target/release-evidence/asm/base64_ng-all-features.s, and target/release-evidence/asm/base64_ng-all-features-lto.s, plus target/release-evidence/asm/MANIFEST.txt with rustc metadata, commands, review focus, and artifact checksums. The LTO artifact exists so reviewers can check that cleanup primitives such as wipe_bytes and wipe_barrier remain visible call boundaries under aggressive optimization.

Capture generated assembly evidence for x86 SIMD encode paths with:

scripts/generate_simd_asm_evidence.sh

On x86/x86_64 hosts, the script emits release test-harness assembly for the admitted AVX-512 VBMI, AVX2, and SSSE3/SSE4.1 encode paths, then checks for the expected byte-shuffle, byte-permute, vector-register, and cleanup instructions. When the aarch64-unknown-linux-gnu target is installed, it also emits AArch64 NEON release assembly and checks for table lookup, bit-select, and register-cleanup instructions. Cross-host runs record NEON library assembly and compile evidence; real AArch64 host runs provide the matching test-harness execution evidence. On non-x86 hosts it records a skip manifest. The generated files are written to target/release-evidence/simd-asm/.

Performance Evidence

The performance harness is intentionally isolated from the published crate. The standard local gate compiles and reviews its dependencies. Run the same check directly while iterating on benchmark code with:

scripts/check_perf.sh

Run local default measurements with:

cargo run --release --manifest-path perf/Cargo.toml

The default perf build enables the base64-ng simd feature. Run the scalar baseline explicitly with:

cargo run --release --manifest-path perf/Cargo.toml --no-default-features

Capture benchmark output and a manifest with:

BASE64_NG_RUN_PERF=1 scripts/check_perf.sh

This writes target/release-evidence/perf/perf-output.csv and target/release-evidence/perf/perf-scalar-output.csv, plus target/release-evidence/perf/MANIFEST.txt.

Every CSV row records effective_backend, active_backend, active_decode_backend, candidate_backend, detection_mode, target_arch, and target_os so release notes can tie throughput numbers to the exact selected backend and the backend that actually executed for that row. active_backend is the primary encode backend reported by runtime::backend_report(). active_decode_backend is the normal strict decode backend reported by BackendReport::active_decode_backend(). effective_backend is scalar or a smaller fallback backend for encode or decode inputs too small to fill the selected SIMD block; rows from the comparison base64 crate use external.

Performance numbers are release notes evidence only when paired with hardware, OS, Rust version, CPU governor, and the exact command output.

Reproducibility

The reproducible package/build check packages and verifies the crate twice and compares the generated package file list. This catches accidental metadata, include-list, or generated-file drift before release.

Publishing

Before tagging:

scripts/stable_release_gate.sh release

The stable release gate is the expensive pre-tag gate and includes Kani, generated assembly evidence, SBOM generation, and reproducibility checks. Run it before creating the immutable GitHub tag.

After the gate passes, push the release commit, wait for GitHub CI, then create and push the v<version> tag. Publish only from that tagged commit:

scripts/release_crates.py --check
scripts/release_crates.py --dry-run

Publish with:

scripts/release_crates.py

The helper reads release-crates.toml, refuses real publishing unless HEAD matches a verified signed release tag, runs the standard local gate and cargo publish --dry-run for selected crates, publishes base64-ng first, waits for crates.io visibility, and then publishes dependent companion crates. The default publish preflight does not rerun Kani because Kani is already part of the pre-tag stable gate. Use scripts/release_crates.py --full-gate only when the release manager deliberately wants to rerun the expensive gate immediately before upload.

Manual fallback for companion releases:

cargo publish -p base64-ng
cargo package -p base64-ng-sanitization
cargo publish -p base64-ng-sanitization --dry-run
cargo publish -p base64-ng-sanitization
cargo package -p base64-ng-derive
cargo publish -p base64-ng-derive --dry-run
cargo publish -p base64-ng-derive
cargo package -p base64-ng-serde
cargo publish -p base64-ng-serde --dry-run
cargo publish -p base64-ng-serde
cargo package -p base64-ng-bytes
cargo publish -p base64-ng-bytes --dry-run
cargo publish -p base64-ng-bytes
cargo package -p base64-ng-subtle
cargo publish -p base64-ng-subtle --dry-run
cargo publish -p base64-ng-subtle
cargo package -p base64-ng-tokio
cargo publish -p base64-ng-tokio --dry-run
cargo publish -p base64-ng-tokio

After cargo publish, verify crates.io metadata with:

cargo info base64-ng

Do not move an existing release tag. If the tagged source is wrong, cut a new patch release.