Tolerance-aware evaluation of silent data corruption (SDC) checking in matrix multiplication.
sdc-bench is the experimental framework behind the paper How Much Detection Is Enough? Tolerance-Aware Evaluation of Silent Data Corruption Checking in Numerical Kernels (R. Singh). It injects single-bit faults into GEMM in four floating-point precisions and evaluates SDC checkers against the error an application can tolerate. The checkers are duplication, fixed-tolerance checksums, rounding-bound checksums, calibrated thresholds, and tolerance-aware checksums. For each one it reports three quantities:
- coverage of harmful corruptions;
- false alarms on fault-free runs, per check and per GEMM;
- wasted recovery: harmless corruptions that would still trigger recomputation.
The campaign is fully deterministic: the reference run reproduces bit for bit on different x86-64 machines and compilers.
- Bitwise change is not harm. In FP64, 95.65% of injections change the output bitwise, but only 21.48% exceed a relative tolerance of 10⁻³.
- Fixed tolerances do not port across precision. A 10⁻⁸ tolerance raises no false alarm in FP64 and fires on 92.85% of fault-free FP32 checks.
- Calibrated thresholds do not port across inputs. Scaled on four input distributions, they raise an alarm on every fault-free GEMM of the held-out non-negative distribution.
- Tolerance-aware checking. At τ = 10⁻³, a checksum with threshold τ/(1+τ)·max|C′| misses 2 of 424,743 harmful corruptions across the four precisions. It raises no false alarm and wastes 1.30–8.09% of recoveries; duplication wastes 86.76–96.86%.
docs/results.md has the full headline tables.
git clone <this repository> && cd sdc-bench
pip install -r analysis/requirements.txt
make test # builds, runs a reduced campaign (~20 s), checks bit-identical output
make full # full campaign: 1,080,000 injections, 2,688,000 fault-free checksThe results land in results/:
| File | Contents |
|---|---|
report.md |
Every table, in readable form |
summary.json |
Every number, with intervals |
fig/ |
Data for every figure |
To re-analyse the archived records without rerunning anything, see docs/reproducing.md.
sdc-bench/
├── src/
│ ├── common.hpp shared input generation: RNG, matrices, fault schedule, FP16/BF16, record layouts
│ ├── campaign.cpp Algorithm 1: fault-injection campaign
│ ├── verify.cpp Algorithm 3: independent bit-level verifier
│ └── bench_timing.cpp detector and recovery cost
├── analysis/
│ ├── analyze.py Algorithm 2: metrics, calibration, statistics, cost model, report
│ └── requirements.txt
├── scripts/
│ ├── run_all.sh full pipeline: build → campaign → verify → timing → analysis
│ └── make_release.sh packages the records of a run for archiving
├── tests/
│ ├── check_quick.sh determinism regression test (make test)
│ └── quick_reference.sha256
├── docs/
│ ├── methodology.md what is measured and how
│ ├── reproducing.md step-by-step reproduction
│ ├── data-format.md binary record layout
│ └── results.md headline results of the reference run
├── data/README.md where the archived records are and what they contain
├── .github/workflows/ci.yml
├── CITATION.cff LICENSE LICENSE-DATA CHANGELOG.md CONTRIBUTING.md
└── Makefile
| Stage | Program | Input | Output |
|---|---|---|---|
| M1 Fault-injection campaign | bin/campaign |
Matrices, precisions, fault models | Injection and fault-free check records |
| M2 Harm and checker evaluation | bin/campaign, bin/verify |
Records | Harmful/benign label and every checker's decision, verified bit for bit |
| M3 Metrics, calibration, statistics | analysis/analyze.py |
Records | Coverage, false alarms and wasted recovery vs. τ; held-out calibration; intervals; cost |
docs/methodology.md covers the checker definitions, the fault models, the calibration protocols and the statistical treatment.
The records of the reference run (about 170 MB) are archived on Zenodo: https://doi.org/10.5281/zenodo.23911926 (data/README.md lists the contents). Records produced by a fresh run should match these SHA-256 digests:
6eaedfc4c1d7759139637dc9621e03b26ae58f71b3fd4133f66a9876b9a8b342 injections.bin
d18c4e1c10c58fec611bd98fc021b6b9dfd0af0a8d907ff6dc5cd1d1e77bf26c clean_checks.bin
If you use sdc-bench or its records, please cite the software using CITATION.cff; GitHub's "Cite this repository" button reads it.
Ramanjeet Singh, Department of Computer Science and Engineering, Chandigarh University, Gharuan, Mohali, India. Email: ramanjeetsingh310@gmail.com