docs: weight the test tiers for the one user who runs this - #36
Merged
Merged
Conversation
The corpus described six tiers without saying what they are worth here. Plexmaton has one user, who runs it daily and reports what breaks, so Tiers 1-3 are the instrument worth investing in and a Tier 4-5 test earns its place only by reaching what no assertion can. A settled boundary is a precondition for Tiers 4-5 and not for Tier 1: much of this project has boundaries nobody has drawn, where a unit test is welcome and mocking the edges is how the boundary gets drawn at all, while an end-to-end test has nothing to assert against a system nobody has decided yet. A flaky test proved nothing and was worse than none while a document still cited it. Fix it, move it to a reporting lane and mark what it claimed unproven, or leave the corpus reading as settled when it is not. An evidence row now says what kind of proof backs an invariant. A fast deterministic test, a rendered frame the user has read, and a real-terminal script all prove something, and they do not prove it equally; a row that lists them alike leaves a reader unable to tell how firmly an invariant stands. Phase 04 records what the code does not yet follow: stage 30 for automatic compaction reporting nothing while a requested one does, and stage 31 for the evidence kinds, the process-spawning test sitting in the fast suite, and this standard outgrowing its budget while it carries the change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents only. No code changes.
Why
The testing standard described six tiers without saying what they are worth here. Plexmaton has
one user, who runs it daily and reports what breaks — so the usual reason to invest in end-to-end
automation is already covered, at a price the corpus was paying without saying so.
That price became visible while fixing compaction:
provisioning_process_death_recovers_one_exact_passive_childspawns and kills a real OS process from inside
cargo test, and fails three runs in four onunmodified
main. Four documents cite it as proof of five invariants. A test that mostly fails isworse than no test, because the corpus reads as settled when it is not.
What changed
standards/testing.mdonly by reaching what no assertion can: terminal lifecycle, escape sequences, a real process
boundary. Anything else buys detection the user already gives.
boundaries nobody has drawn. A unit test is welcome there, and mocking the edges is often how
the boundary gets drawn at all; an end-to-end test has nothing to assert against a system nobody
has decided yet, and repairing one defends a guess.
fast suite charges the whole development loop for evidence only a wrap-up ever needed.
still cites it. Fix it, or move it to a reporting lane and mark what it claimed unproven.
.agents/README.md— an evidence row says what kind of proof it is. A fast deterministic test,a rendered frame the user has read, and a real-terminal script all prove something and do not prove
it equally; listing them alike leaves a reader unable to tell how firmly an invariant stands.
phases/phase-04-product-polish.md— stage 29 recorded complete (the compaction decline, #35),and two unstarted stages record what the code does not yet follow:
nothing, while the same operation asked for by name reports a line. Needs a rendered frame
first: an automatic checkpoint lands mid-turn, where a requested one lands after the last entry.
citations across 23 evidence documents, and this standard outgrowing its 8 KB budget while it
carries the change.
Note on the budget warning
standards/testing.mdwas 8186 of 8192 bytes before this change — already full. It is now 931over, and
.agents/README.md389 over. Both warnings are advisory. Part of the excess istransitional: the paragraphs explaining why the policy is changing become unnecessary once the
code follows it, and the stage-31 notes are deleted when consumed. The rest is genuinely new
policy, and stage 31 records that the standard's
Ownslists four things while performanceevidence is a separate trigger.
Citations and frames gates pass; six pre-commit gates pass.