Repository navigation
The end to end paths driven by hand or by nothing, played: 23 scenarios with written answers, kills and prepared states - #115
Merged
Conversation
…tten answers The five scenarios tests/e2e/README.md gave a developer to drive are pytest scenarios now: planning from the specs killed in the interview, an amendment killed once recorded, an amendment in the conversation, a break on a commit of the developer refused or accepted, and the three ways on after the ceiling. The developer's answers are written in the tests and given one session each, with --resume. Every question of the chain carries a recommendation, so one reply that takes it answers any question of an interview; no model plays the developer, which is what the evaluations do. The toy gains the prepared states the paths of issue 9 start from. Those named after a death hold what a session leaves in the files when it dies there. A kill now also takes what a session left at work in its project, since a gate runs in a process group of its own and would record its run after the kill.
`/surface-status abandon` records nothing before the developer's yes, then abandons, commits the journal alone, pushes and refreshes the description; the conformity check of the host's CI then fails on a plan abandoned after its approval. On a branch that carries two plans, each of the three commands asks which one and records nothing in either.
…rst push, played The toy had no CI, so only one branch of the first push had ever run. A variant of it holds a workflow that runs on every push and on every pull request: `/surface-plan` drafts the plan, commits it, and waits for the developer's agreement before anything leaves the machine, then pushes and opens the draft. Declined, it pushes nothing and says the plan is committed locally. Without a CI, a plan committed and never pushed is pushed at the relaunch, and its draft opened, without a question.
…aunched, played A session is killed while a fixer has changed code, while the gate of the plan runs, while the reviewer works, while the extractor draws the blueprint and while the checker reads it. Each relaunch keeps the journal of the dead session as it was and takes the plan where its path leads. The two moments a kill cannot hit, since an agent records and commits in one command, start from a state that holds what a death there leaves: a slice recorded and not committed, which the relaunch commits as it stands, and a review written and not recorded, which it records with no second reviewer.
… prepared states Planning that passed its ceiling, two cross-checks having counted the same omission, records the block, launches nothing, and takes the developer's instruction in the conversation: `resumed`, then the blueprint drawn and cross-checked again. A break an executor suspects waits in the journal for a reviewer, whatever became of the session. Two states hold what it leaves: one on a command the feature would replace without the blueprint saying so, which the reviewer confirms and the developer then refuses, and one on a test helper the plan does not name, which the reviewer dismisses with a note for the next executor.
…relaunch finds no plan The last step of the loop commits `conformant`, then pushes and refreshes the description. A state holds what a session leaves when it dies between the two, and the scenario holds the relaunch to the push that stop still owes. It fails as the chain stands, and is kept as it is: `/surface-execute` finds no plan once the only one of the branch is conformant, says so and stops, so the branch stays unpushed and the pull request keeps the description of a plan in progress. Whether a prompt or the resolver takes that plan up again is the chain's to decide.
…an agent cuts its commands The scenario that kills a session in a slice waits for code an executor wrote and has not committed. An executor that writes, tests, records and commits in one command leaves that code uncommitted for a fraction of a second: the session then runs to its end, and the scenario failed that way once. Any state of the toy can now be built slow, with a test on the main branch that lasts 20 seconds, so that the code stays uncommitted for as long as the agent's tests run. The kill in a slice and the kill in a fix start from a project built slow, as the kill in a gate run did under a state of its own, which the option replaces. The kill in a slice now also takes what its session left at work: the tests of the executor would outlive it, then record and commit the slice under the session that takes over.
… until the chain takes that push up A session that dies between the commit of a conformant plan and its push is taken up by no relaunch: /surface-execute finds no plan once the only one of the branch is over, and stops before any push (#114). The scenario that shows it fails by construction. Left red, it would fail the whole set each time it is played, for a fault that is known and has its issue. It is marked as expected to fail, strictly: the day the chain takes that push up, the scenario passes, the mark fails, and the mark goes. Played once so: reported as an expected failure.
…nd their cost kept The scenarios had last run on several versions of the prompts, and no single version had run them all (#9). Each was played on the prompts of commit 4220424 as it was written, the four that existed included: 33 runs, 13.61 USD at list price. docs/e2e/2026-10-07.md keeps the verdict and the cost of each, what the runs found and what they do not say. tests/e2e/README.md says where that record is, and that the one scenario that fails as the chain stands is marked as expected to fail.
… leaves no session behind it An independent review of the scenarios found no failure that repeats, and a harness that could disturb the session it watches, or lose it. - `dirty` asked for a plain `git status`, which takes the lock of the index. Asked every half second while a session commits, it can make a `git add` of that session fail on a lock the chain never took: the reviewer counted 24 failures in 400 against a status asked without a pause, and none with `--no-optional-locks`, which the look now passes. - `kill_when` killed its session at the moment only. A watch that raised left the session running to its end, billed, behind a scenario that had failed: it is killed whatever the watch raises. `stream` reads a log as it grows, and no longer raises on a character the read cut in two. - The scenario of a kill in a slice kept a loop of its own, without the guard that the moment came: `kill_when` plays it like the others. - `launches` reads an agent launched under either name of the tool, `Agent` or `Task`, as the evaluations do. Under the other name it found none, and a scenario that asserts that no agent was launched held nothing. - A reply that follows a refusal goes to the session the refusal gave, where it went to the one before it. - The plan folder is read in the project where a scenario named it from the date a second time, after its session: a run that crossed midnight UTC looked for the folder of the next day. - `tests/e2e/README.md`: the row of the conformity never pushed is back in its table, the neighbours of the prepared states are three, and the kill by hand says what the kill of a process group misses. The tests that start no session pass in the container, 27 of them, and the scenario of a kill in a slice was played once more as it now stands, on the prompts of commit 505848a: killed after the approval and before any slice was done, then passed, 0.43 USD for the session that took over. `docs/e2e/2026-10-07.md` says so, under what its runs do not say. Left as they are, and said in the pull request: a kill that takes more than a real death does, since it takes a gate under way with its session; the strict mark of #114, which covers its whole test; a conversation that takes the recommendation without reading that a question was asked; and the kill in a review, which does not assert that a reviewer was launched again.
This was referenced Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The end to end tests held four scenarios, all of
/surface-executefrom a prepared state. Five more were described intests/e2e/README.mdto be driven by hand, and #9 lists the paths no scenario covered: a session killed anywhere but in a slice, two plans on one branch, a break an executor suspects, the ceiling in planning, an abandon. #8 asked for a toy whose CI reacts to a push.This adds 23 scenarios, 26 tests with those played once per command or per agent, to the four. The first seven commits are the work of an agent given #9 as its task, each stage its own commit. The next two are the mark on the one scenario that fails and the record of the runs, and the last mends what a review found.
What is played now
/surface-planwarns and waits before the first push, pushes and opens the draft on a yes, and pushes nothing on a no.How each is played and what it must hold is in
tests/e2e/README.md, rewritten: nothing is left to drive by hand.How
gh, and of what a session said one word at most.tests/e2e/toy/toy.pygains the prepared states those paths start from, each documented in the README and checked without a session bytest_a_prepared_state_is_the_one_named. A state that holds work not committed or not pushed is what a death there leaves: an agent chains a record and its commit in one command, and leaves no moment to kill it at.ARCHITECTURE.mdand ADR 0025 say that the answers are written in the tests, where they said that those scenarios were played by hand.Nothing of
skills/,agents/, the stand-inghor the image changes.Played
Every scenario was played on the prompts of commit
4220424: 33 runs, 13.61 USD at list price.docs/e2e/2026-10-07.mdkeeps the verdicts and the cost of each. 21 of the 23 new scenarios passed their only run, one passed on its second, and one fails as the chain stands.They were not played in one command, and every verdict but five rests on one run: how often a scenario fails for a path a model takes is not known.
What they found
conformantand its push is taken up by no relaunch. The scenario is marked as expected to fail, strictly: the day the chain takes that push up, the mark fails and goes.Read by an independent reviewer
Before its merge, on 2026-10-08: fifteen findings, none of them a failure that repeats. The last commit mends those that cost little and that a run can check:
git status, which takes the lock of the index, and could make agit addof that session fail: the reviewer counted 24 failures in 400 against a status asked without a pause. The look passes--no-optional-locks.Task, is read. A reply after a refusal goes to the session the refusal gave. The plan folder is read in the project, where a scenario named it from the date after its session. Three slips of the README.The tests that start no session pass in the container, and the kill in a slice was played once more as it stands, on the prompts of commit
505848a: killed after the approval and before any slice was done, then passed, 0.43 USD.Left as they are:
kill -9ofclaudeleaves on a developer's machine: a gate that outlives its session and records its run under the next one. No run shows it as a fault of the chain. The README says it where it describes a kill by hand.toy.py, the README and the list of a test, and nothing checks that they agree.Not done
actions/checkoutin the guide, which Exercise the pull request steps and the host CI snippet against GitHub #8 keeps.mainas it stands: two prompts changed since, the extractor (A blueprint whose long sections hold a title in bold on each thing the developer comes back for #111) and the hand over (A hand over that tells none of the steps that led to it, and gives the address of the pull request #113), and no scenario asserts what either says.