Skip to content

fix(tests): publish a readiness marker atomically, and place tests by cost - #37

Merged
benjipeng merged 1 commit into
mainfrom
fix/test-evidence-lanes
Sep 19, 2026
Merged

benjipeng merged 1 commit into
mainfrom
fix/test-evidence-lanes

Conversation

@benjipeng

Copy link
Copy Markdown
Contributor

Stage 31. One test-fixture fix, and the standard corrections it taught.

Why

provisioning_process_death_recovers_one_exact_passive_child failed three runs in four. The fixture
created its readiness marker and then filled it, while the parent took existence for completeness
and parsed the contents — so it read an empty file and reported a wrong marker. Writing beside the
destination and renaming makes existence imply completeness. The suite has now run clean twenty
times.

Measuring refuted two of the three things stage 31 claimed

Stage 31 was written in this repo by an agent, and it was wrong twice.

Claimed Measured
A Tier 4 test to relegate to its own lane A broken handshake. Fixed. It costs 50 ms of a 15.77 s workspace suite
46 script citations across 23 documents say nothing about what kind they are 36 of 40 table citations already do. 4 did not
A process-spawning test charges the whole loop Its permission-store sibling costs 4.16 s spawning processes the same way, and nobody had mentioned it

The rule is not "don't spawn processes". It is: a marker whose existence is the signal must not
also carry data the reader parses
— unless it is published atomically. Two sibling tests use the
same spawn-a-child pattern and are safe for structural reasons: one waits for full exit, the other's
marker is a fixed literal nobody parses.

What changed

standards/testing.md — two corrections and one addition:

  • Cost and determinism place a test, never mechanism. Judged by mechanism, the old rule condemned a
    50 ms deterministic process test and overlooked a 4.16 s one.
  • Both the Tier 4 rule and the flaky rule stopped pointing at a lane CI does not have: verify runs
    cargo test --workspace and all seven smoke scripts in one job, so there was never a slower
    lane to move anything into. A flaky test is fixed, or deleted with its claim marked unproven.
  • The handshake rule is stated once, beside the readiness-signal rule that already governed the
    pattern.

Four evidence rows now match the convention the other 36 use: named tests first, then ; and a
clause saying what a script or a gate proves — composer-menu.md SKP-1, selection-and-copy.md
SEL-5, transcript-entry.md ENT-1, session-journal.md JRN-5. CTL-1, COL-4, COL-5, CHB-2 and CHB-3
have real proof again across three evidence documents.

Stage 31, the phase status and the roadmap row record what happened rather than what was
predicted: fixed, not moved.

Not done here, deliberately

plexmaton-command/src/prefix.rs carries MAX_PARSE_TIME = 20 ms as a wall-clock bail-out on every
command admission. When it trips, the parse is abandoned and a prefix Deny stops matching, so
evaluation falls through to grants where an exact grant can allow the call — a Deny skipped because
the machine was busy. Observed as 1 failure in 5 cargo test --workspace runs.

That belongs to spike/permission-boundary, whose head commit is already "fence a command's writes
instead of inspecting it"
— fixing it here would be thrown away and would conflict.

Blast zone verified disjoint: none of the 8 files here appears among that branch's 14
(git diff --stat main...spike/permission-boundary).

Verification

All run outside the sandbox — 17 runtime tests bind loopback fixture servers, which a sandbox
refuses, and that reads as 17 failures.

  • runtime suite 20 consecutive clean runs, 247 passed, ~1.09 s
  • cargo test --workspace --locked exit 0
  • cargo clippy --workspace --all-targets --locked -- -D warnings exit 0
  • citations gate: 1349 named proofs all resolve with unique owners; frames gate passes
  • 6 pre-commit gates pass

standards/testing.md is 274 bytes larger than at branch start, not smaller as planned: it gained a
rule it did not have. Budgets are advisory, and the paragraphs explaining why the policy changed are
what gets deleted once the corpus has settled into it.

… cost

The process-death fixture created its readiness marker and then filled it, while the parent took
existence for completeness and parsed the contents, so it read an empty file and reported a wrong
marker. It failed three runs in four. Writing beside the destination and renaming makes existence
imply completeness, and the suite has now run clean twenty times.

Stage 31 had recorded this as a Tier 4 test to relegate to its own lane. Measuring first refuted
that: the test costs 50 ms of a 15.77 s workspace suite, and its permission-store sibling costs
4.16 s spawning processes the same way without anyone mentioning it. Nothing needed relegating,
and CTL-1, COL-4, COL-5, CHB-2 and CHB-3 have real proof again across three evidence documents.

So the standard places a test by cost and determinism rather than by mechanism, which had condemned
the cheap deterministic one and overlooked the expensive one. It also states the handshake rule this
bug taught, beside the readiness-signal rule that already governed the pattern.

Both the Tier 4 rule and the flaky rule stopped pointing at a lane CI does not have: verify runs
cargo test and all seven smoke scripts in one job, so there was never a slower lane to move anything
into. A flaky test is fixed or deleted with its claim marked unproven.

Of 40 script citations in evidence tables, 36 already named what the script proves. The four that
listed a script or a gate among test function names now say which kind of evidence it is, so a
reader can tell how firmly the invariant stands.

The standard remains over its advisory byte budget. The paragraphs explaining why the policy changed
are the part that gets deleted once the corpus has settled into it.
@benjipeng
benjipeng merged commit 43d9346 into main Sep 19, 2026
2 of 6 checks passed
@benjipeng
benjipeng deleted the fix/test-evidence-lanes branch September 19, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant