Skip to content

docs: weight the test tiers for the one user who runs this - #36

Merged
benjipeng merged 1 commit into
mainfrom
docs/testing-strategy
Sep 19, 2026
Merged

benjipeng merged 1 commit into
mainfrom
docs/testing-strategy

Conversation

@benjipeng

Copy link
Copy Markdown
Contributor

Documents only. No code changes.

Why

The testing standard described six tiers without saying what they are worth here. Plexmaton has
one user, who runs it daily and reports what breaks — so the usual reason to invest in end-to-end
automation is already covered, at a price the corpus was paying without saying so.

That price became visible while fixing compaction: provisioning_process_death_recovers_one_exact_passive_child
spawns and kills a real OS process from inside cargo test, and fails three runs in four on
unmodified main. Four documents cite it as proof of five invariants. A test that mostly fails is
worse than no test, because the corpus reads as settled when it is not.

What changed

standards/testing.md

  • Tiers 1–3 are the instrument, because they shape behaviour. A Tier 4–5 test earns its place
    only by reaching what no assertion can: terminal lifecycle, escape sequences, a real process
    boundary. Anything else buys detection the user already gives.
  • A settled boundary is a precondition for Tiers 4–5, not for Tier 1. Much of this project has
    boundaries nobody has drawn. A unit test is welcome there, and mocking the edges is often how
    the boundary gets drawn at all; an end-to-end test has nothing to assert against a system nobody
    has decided yet, and repairing one defends a guess.
  • A Tier 4 test belongs in its own lane. One that spawns or kills a process while sitting in the
    fast suite charges the whole development loop for evidence only a wrap-up ever needed.
  • The flaky rule is rewritten: a flaky test proves nothing and is worse than none while a document
    still cites it. Fix it, or move it to a reporting lane and mark what it claimed unproven.

.agents/README.md — an evidence row says what kind of proof it is. A fast deterministic test,
a rendered frame the user has read, and a real-terminal script all prove something and do not prove
it equally; listing them alike leaves a reader unable to tell how firmly an invariant stands.

phases/phase-04-product-polish.md — stage 29 recorded complete (the compaction decline, #35),
and two unstarted stages record what the code does not yet follow:

  • 30, automatic compaction is invisible. A checkpoint the runtime takes on its own reports
    nothing, while the same operation asked for by name reports a line. Needs a rendered frame
    first: an automatic checkpoint lands mid-turn, where a requested one lands after the last entry.
  • 31, test evidence says what kind it is. The process-spawning test's lane, the 46 smoke-script
    citations across 23 evidence documents, and this standard outgrowing its 8 KB budget while it
    carries the change.

Note on the budget warning

standards/testing.md was 8186 of 8192 bytes before this change — already full. It is now 931
over, and .agents/README.md 389 over. Both warnings are advisory. Part of the excess is
transitional: the paragraphs explaining why the policy is changing become unnecessary once the
code follows it, and the stage-31 notes are deleted when consumed. The rest is genuinely new
policy, and stage 31 records that the standard's Owns lists four things while performance
evidence is a separate trigger.

Citations and frames gates pass; six pre-commit gates pass.

The corpus described six tiers without saying what they are worth here. Plexmaton has one user,
who runs it daily and reports what breaks, so Tiers 1-3 are the instrument worth investing in and
a Tier 4-5 test earns its place only by reaching what no assertion can.

A settled boundary is a precondition for Tiers 4-5 and not for Tier 1: much of this project has
boundaries nobody has drawn, where a unit test is welcome and mocking the edges is how the
boundary gets drawn at all, while an end-to-end test has nothing to assert against a system nobody
has decided yet.

A flaky test proved nothing and was worse than none while a document still cited it. Fix it, move
it to a reporting lane and mark what it claimed unproven, or leave the corpus reading as settled
when it is not.

An evidence row now says what kind of proof backs an invariant. A fast deterministic test, a
rendered frame the user has read, and a real-terminal script all prove something, and they do not
prove it equally; a row that lists them alike leaves a reader unable to tell how firmly an
invariant stands.

Phase 04 records what the code does not yet follow: stage 30 for automatic compaction reporting
nothing while a requested one does, and stage 31 for the evidence kinds, the process-spawning test
sitting in the fast suite, and this standard outgrowing its budget while it carries the change.
@benjipeng
benjipeng merged commit 689a6ea into main Sep 19, 2026
3 checks passed
@benjipeng
benjipeng deleted the docs/testing-strategy branch September 19, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant