Repository navigation
A hand over that tells none of the steps that led to it, and gives the address of the pull request - #113
Merged
Conversation
…e address of the pull request What is left of #67 after its three choices were measured: every one of the 19 hand overs of the fifth campaign tells a step of the chain, "I redrew and cross-checked the blueprint, which found no omissions, then committed and pushed. The pull request description is refreshed." The judge has counted it as noise for five campaigns, `following.no-noise` 2 to 3, under a criterion the developer left as it is when they drew its one exception, the launch that approves. Part of it was asked for. Since #105 the command was told, where it says in which words it speaks: "Say that the blueprint was drawn and cross-checked, not which agent did it nor in which file its report stands." It was written as an example of the words to use, and reads as an order to say it: nine hand overs of nineteen repeat it word for word. - `skills/surface-plan/SKILL.md`, the hand over: none of the steps that led there is told, the drafting, the cross-check and what it found, the commit, the push, the pull request opened or refreshed. A hand over means they were done, and only a step that was not done is told, as the section on the commit and the push has it. - The same step asks for the address of the pull request when there is one, whose description links the blueprint. It asked for "its link in the pull request", which sessions answered without an address: 17 hand overs of 19 give one, 16 of them in the sentence that tells the opening. - Where the command says in which words it speaks, the sentence no longer hands it one to say: a step is named, by what it does, when it has to be told, a failed one for instance. That the cross-check found no omission is always true at a hand over, since the state script refuses `plan-drafted` otherwise: it tells the developer nothing, and where it is false there is no hand over, but the stop at the ceiling, which says so. Read by an independent reviewer, which found the sentence of #105 and the address that only the telling carried: both were blocking, and neither was in the first draft. Measured on two runs of `01-overdue-list` stopped at the hand over, four hand overs, against the four of the same case in the fifth campaign: none tells the cross-check, the commit or the push, against four; two still say that the pull request is refreshed; three give its address, against two; 83 to 135 words, against 112 to 209. The judge was not asked.
PierreMardon
added a commit
that referenced
this pull request
Oct 8, 2026
…os with written answers, kills and prepared states (#115) The end to end tests held four scenarios, all of `/surface-execute` from a prepared state. Five more were described in `tests/e2e/README.md` to be driven by hand, and #9 lists the paths no scenario covered: a session killed anywhere but in a slice, two plans on one branch, a break an executor suspects, the ceiling in planning, an abandon. #8 asked for a toy whose CI reacts to a push. This adds 23 scenarios to the four, 26 tests with those played once per command or per agent: - The scenarios driven by hand, with their answers written in the test, one session per answer: planning from the specs killed during the interview, an amendment killed once recorded, an amendment in the conversation and a sentence that agrees, a break on a commit of the developer refused, and accepted, the ceiling then its three ways on. - The status command and a branch with two plans: an abandon that waits for a yes, then the conformity check that fails on it; each of the three commands puts two plans to the developer and records nothing. - A CI that runs on a push (#8): `/surface-plan` warns and waits before the first push, pushes and opens the draft on a yes, and pushes nothing on a no. - Sessions that died: in a fix, in a gate run, in a review, in the extraction, in the cross-check, and, from prepared states, between a record and its commit and between a commit and its push. - The ceiling in planning, and a break suspected in a slice, confirmed then refused, and dismissed, from prepared states. No model plays the developer here, which is what the evaluations do: every question of the chain carries a recommendation, so one written reply that takes it answers a question that changes from run to run. A scenario asserts what the project holds once its sessions ended, the journal, the plan folder, git and the record of the stand-in `gh`, and of what a session said one word at most. `tests/e2e/toy/toy.py` gains the prepared states those paths start from, each documented in the README and checked without a session by `test_a_prepared_state_is_the_one_named`. A state that holds work not committed or not pushed is what a death there leaves: an agent chains a record and its commit in one command, and leaves no moment to kill it at. A kill now also takes what the session left at work in the project, since a gate runs in a process group of its own and would record its run while the next session works. Any state can be built slow, with a test that lasts 20 seconds, so that a kill in a slice or in a fix no longer depends on how an agent cuts its commands: the scenario that existed passed by luck. `tests/e2e/README.md` is rewritten: nothing is left to drive by hand. `ARCHITECTURE.md` and ADR 0025 say that the answers are written in the tests, where they said that those scenarios were played by hand. Nothing of `skills/`, `agents/`, the stand-in `gh` or the image changes. Every scenario was played on the prompts of commit 4220424: 33 runs, 13.61 USD at list price. `docs/e2e/2026-10-07.md` keeps the verdicts and the cost of each. 21 of the 23 new scenarios passed their only run, one passed on its second, and one fails as the chain stands: a session that dies between the commit of `conformant` and its push is taken up by no relaunch (#114). It is marked as expected to fail, strictly: the day the chain takes that push up, the mark fails and goes. Read by an independent reviewer before its merge: fifteen findings, none of them a failure that repeats. Mended: the look at the repository of a running session passes `--no-optional-locks`, where a plain `git status` took the lock of the index and could make a `git add` of that session fail; a session is killed whatever the watch on it raises, where a watch that raised left it running, billed; the kill in a slice is played by the helper of the other kills. That scenario was played once more as it stands, on the prompts of commit 505848a: passed, 0.43 USD. What the review found and was left as it is stands in the pull request: a kill that takes a gate under way with its session, which a real death does not, and the strict mark of #114, which covers its whole test. Every verdict but five rests on one run, and the set was not played again on `main` as it stands: two prompts changed since, the extractor (#111) and the hand over (#113), and no scenario asserts what either says.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is left of #67 after its three choices were measured in the fifth campaign: every one of its 19 hand overs tells a step of the chain.
The judge has counted it as noise for five campaigns,
following.no-noise2 to 3 out of 5, under a criterion the developer left as it is when they drew its one exception, the launch that approves (#104).Part of it was asked for. Since #105, where the command says in which words it speaks: "Say that the blueprint was drawn and cross-checked, not which agent did it nor in which file its report stands." It was written as an example of the words to use and reads as an order to say it: nine hand overs of nineteen repeat it word for word.
What changes
In
skills/surface-plan/SKILL.md:gh, no remote.tests/prompts/test_surface_plan.pyholds the three.Why it is the chain's to say
That the cross-check found no omission is always true at a hand over: the state script refuses
plan-draftedotherwise. It tells the developer nothing, and where it is false there is no hand over but the stop at the ceiling, which says so. The guide lists what the command says at its end, where to read the blueprint and which gates the loop will run, and the cross-check is not among it.Read by an independent reviewer
Two findings were blocking, and neither was in the first draft: the sentence of #105 that asks for what the new one forbids, and the address of the pull request, which only the telling carried. It also asked for "a step that was not done", where the draft said "could not be done": a push the developer declined could have been.
What it says not to expect: seven hand overs of nineteen reassure on what did not happen, "No code is written", "Nothing is approved yet". They are not steps that led there, and the sentence does not aim at them.
Measured
Two runs of
01-overdue-listplayed to the hand over from this branch (14 sessions, 3.01 USD,.evals/hand-over-67/, out of git), four hand overs, against the four of the same case in the fifth campaign:For instance: "The blueprint is at
docs/plans/2026-10-07-overdue-command/blueprint.md. The draft pull request is https://github.com/stand-in/lending/pull/1, and its description links to the blueprint."Not measured: the judge was not asked, one case was played, and no run met a step that was not done, a declined push for instance.