Repository navigation
Nothing verifies that a sanitized lane actually compiled with CBM_SANITIZED_BUILD (UBSan and trap-UBSan have no probe) #1648
Description
Summary
Nothing verifies that a sanitized lane actually compiled with CBM_SANITIZED_BUILD defined. #1615 gave
CBM_SANITIZED one spelling and a compiler-probe backstop, but the backstop only covers the three
sanitizers that announce themselves. For UBSan and trap-UBSan there is no probe and never can be, so the
build system is the sole source of truth — and a silent regression there is invisible: the suite still
passes, just against native timing budgets on an instrumented binary.
That is not hypothetical. It is exactly what happened twice:
- trap-UBSan, recorded in the comment above
SANITIZED_DEFINE(Makefile.cbm:88-96): the leg "once
ran NATIVE timing budgets on an instrumented binary". - TSan, fixed in
0a163d4f:CFLAGS_TSANnever includedSANITIZED_DEFINE, so every sanitized
budget compiled to its native value. It surfaced assubprocess_run_spawn_failurefailing on a PR
whose diff could not possibly have caused it — twice on one SHA — and cost two more PRs (fix(config): self-heal an orphan managed marker on removal (#1558) #1590, fix(subprocess+tsan): widen the spawn-retry window where it was silently disabled #1595)
to chase down.
Opening this per CONTRIBUTING.md ("Open an issue first — always"; build-system changes need prior
design discussion). Happy to implement once there is a decision on shape — nothing is written yet.
Why the existing mechanisms do not cover it
| mechanism | covers | does not cover |
|---|---|---|
SANITIZED_DEFINE (Makefile.cbm:93-96) |
any lane keyed off $(SANITIZE) |
a lane with its own sanitize variable — precisely the TSan gap |
probes in sanitized.h (#1615) |
ASan, TSan, MSan | UBSan, trap-UBSan — no macro, no __has_feature bit exists |
| the test suite passing | nothing here | a native budget on an instrumented binary is looser to violate, not tighter; it mostly still passes |
The third row is the important one. The failure mode is a test that passes for months and then flakes on
a loaded runner, attributed to whatever PR happened to be in flight.
It is not only about budgets any more
scripts/msan.sh:120-129 reasons from this macro while ruling out a cause:
CBM_THREAD_STACK_MBat 256 MiB and at 1024 MiB (the cap). The knob is verified compiled in
(CBM_SANITIZED_BUILDis defined for this lane) and the floor is applied incbm_thread_create[...]
The fault address did not move by a single byte between 256 MiB and 1024 MiB — a 4x stack increase
changing nothing is what rules out "the stack is merely too small".
If the define were ever absent on that lane, cbm_thread_stack_floor would compile to return requested
(src/foundation/compat_thread.c:33-48), the 4x increase would have changed nothing because the knob was
inert, and a documented "do not repeat" conclusion would be wrong. The premise is asserted in a comment;
nothing checks it.
Proposed shape
Three small pieces, and no changes under .github/workflows/ at all — which is the point of this shape.
1. The binary reports what it was compiled with. A --build-config early-exit in
tests/test_main.c, next to the existing --version (line ~796) and --list-suites (line ~861),
printing one line of key=value:
$ build/c/test-runner --build-config
sanitized=1 test_seams=1
sanitized comes straight from CBM_SANITIZED, so it answers the only question that matters: what did
the compiler actually receive, not what the Makefile intended to send.
2. scripts/test.sh asserts it. This is the single choke point — every sanitized lane in
_test.yml already goes through it:
| leg | _test.yml |
invocation | expected |
|---|---|---|---|
test-unix |
:93 | scripts/test.sh CC=… CXX=… (default ASan+UBSan) |
sanitized=1 |
test-diag |
:221 | scripts/test.sh … SANITIZE="-fsanitize=address,undefined …" |
sanitized=1 |
test-lsan-macos |
:308 | scripts/test.sh … (ASan+LSan) |
sanitized=1 |
test-tsan |
:340 | scripts/test.sh --tsan … |
sanitized=1 |
| Windows CLANGARM64 | test.sh:173-177 |
trap-UBSan injected by the script itself | sanitized=1 |
| explicit plain build | — | scripts/test.sh SANITIZE= |
sanitized=0 |
test.sh already knows the answer: it tracks SANITIZE_GIVEN (:147), the --tsan mode (:98, :179-185)
and the ARM64 trap-UBSan default (:173-177). So it can compute the expectation from the flags it is about
to pass to make, run --build-config on the binary it just built, and fail loudly on mismatch, before the
suite runs:
ERROR: build config mismatch — this lane builds instrumented but the binary reports sanitized=0.
Every sanitized-budget branch compiled to its NATIVE value.
Check that this lane's flag set carries $(SANITIZED_DEFINE) or -DCBM_SANITIZED_BUILD=1.
Note the TSan mode must check $BUILD_DIR/test-runner-tsan, not test-runner.
3. scripts/msan.sh gets the same one-liner. It is the only lane that bypasses test.sh
(_test.yml:284 → Docker → msan.sh, running build/msan/test-runner). One check next to the build at
msan.sh:66-68 — and it would turn the comment quoted above into something enforced.
What this would have caught, and how fast
The TSan gap would have failed test-tsan in seconds, at the build step, with a message naming the cause
— instead of surfacing as an unrelated-looking flaky spawn test across three PRs.
Limits, stated up front
- It proves the define reached the compiler, not that the sanitizer is instrumenting. That was never
the failure mode, and-fsanitize=not working would fail far louder. - It cannot cover a future lane that routes around both scripts. Partial mitigation: the fix(build): give sanitized-build detection one spelling and a backstop #1615 probes
still self-heal ASan/TSan/MSan, so a new lane would have to be UBSan-only to slip through. - It adds one flag to a test-only binary. No production surface.
test-windows(_test.yml:398) runs unsanitized, so it assertssanitized=0— which is also worth
having: it catches the inverse mistake of leaking the define into a native lane.
Alternatives considered
#errorinsanitized.hwhen a probe fires without the define. Rejected in fix(build): give sanitized-build detection one spelling and a backstop #1615, for reasons
that still hold: it breaks the binary instead of correcting it, and breaks an out-of-tree
make CFLAGS_EXTRA=-fsanitize=address. It is also blind to UBSan, the case this issue is really about.- Env var (
CBM_EXPECT_SANITIZED=1) plus an in-suite test. A lane that forgets the variable skips
the check silently — the same silence that caused the problem. Inverting the default (fail when unset)
breaks every ad-hoc local run. - Grep
Makefile.cbmin CI for the define on each flag set. Verifies the intent, not the binary, and
would have missed nothing only by accident — a flag set can also be assembled in a script. - A
make print-flagstarget compared in CI. Same weakness: it reports what make would pass, not
what the compiler received, and it needs the CI workflow edits this shape avoids.
Scope
Three files: tests/test_main.c, scripts/test.sh, scripts/msan.sh. Roughly 40-60 lines including the
error text. One issue, one PR, per CONTRIBUTING.md.
Activity
DeusData commentedon Aug 14, 2026
This is the right generalisation, and it comes from a real miss on our side rather than a hypothetical.
#1595 widened the spawn-retry budget for sanitized builds and it did nothing on the TSan leg, because
CBM_SANITIZED_BUILDwas never defined there —SANITIZED_DEFINEkeys off$(SANITIZE), and TSan uses its ownTSAN_SANITIZE. 169 thread-instrumented compile lines, not one carrying the define. The fix was a one-line Makefile change; the expensive part was that nothing could tell us the budget was inert, so the leg kept failing on a PR whose diff made causation impossible.Your #1615 then found a fourth spelling I had missed entirely (
tests/test_cs_lsp_bench.cchecking ASan only, so TSan and MSan measured instrumented parses against a native 200 ms budget). Two independent instances of the same class inside a week is the argument for asserting it rather than remembering it.A runtime or build-time assertion that a lane which is instrumented also knows it is instrumented would have caught both at the moment they were introduced instead of after a release cycle. The Makefile comment already warns about exactly this for trap-UBSan — the lesson was written down and then not enforced, which is precisely the gap a check closes.
Fair warning on timing: our CI is badly backed up tonight behind a v0.10.5 that fixes a batch of install and startup blockers, so this will get proper attention after that lands. Thank you for filing it as an issue with the analysis rather than just patching around it — and for the three accurate reports on my code this week.
astandrik commentedon Aug 17, 2026
I’d like to take this one. I’ll keep the implementation to the agreed shape: a
--build-configreport from the test runner, fail-closed verification inscripts/test.sh, and the same verification inscripts/msan.sh, with no workflow or production-code changes. I’ll validate the default sanitized, explicitSANITIZE=, TSan, and MSan paths locally first and follow up here before opening a PR.astandrik commentedon Aug 18, 2026
The implementation is now in #1716.
It changes only
tests/test_main.c,scripts/test.sh, andscripts/msan.sh: 76 insertions and one deletion. The Makefile, workflows, dependencies, and production code are unchanged. The commit is signed and passes the DCO check.I checked the exact LF-terminated
--build-configoutput and the early exit before test-home or cache setup. A plain runner reportssanitized=0 test_seams=1; the default sanitized and explicit UBSan runners reportsanitized=1 test_seams=1. Empty and whitespace-onlySANITIZEvalues take the plain path. A reused plain build also stops at the new check before a sanitized iteration can start a suite. The fullscripts/test.shrun passed on Linux x86-64.Some platform checks still need CI. On Apple Silicon, ASan and TSan fail in the sanitizer runtime before
main. The remote MSan path cannot reach its build command because the Compose service is pinned tolinux/arm64and the x86-64 builder has no binfmt/QEMU registration. Windows CLANG64/CLANGARM64 and the authoritative sanitizer lanes will run in PR CI.DeusData commentedon Sep 27, 2026
Fixed on
mainin 11b662f (PR #1716, thanks to @astandrik): the test binary now reports its sanitizer build define, and the sanitizer lanes refuse to run when the compiled configuration doesn't match the lane. Thank you for raising it!