Skip to content

The end to end paths driven by hand or by nothing, played: 23 scenarios with written answers, kills and prepared states - #115

Merged
PierreMardon merged 10 commits into
mainfrom
test/e2e-remaining-paths
Oct 8, 2026
Merged

PierreMardon merged 10 commits into
mainfrom
test/e2e-remaining-paths

Conversation

@PierreMardon

@PierreMardon PierreMardon commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

The end to end tests held four scenarios, all of /surface-execute from a prepared state. Five more were described in tests/e2e/README.md to be driven by hand, and #9 lists the paths no scenario covered: a session killed anywhere but in a slice, two plans on one branch, a break an executor suspects, the ceiling in planning, an abandon. #8 asked for a toy whose CI reacts to a push.

This adds 23 scenarios, 26 tests with those played once per command or per agent, to the four. The first seven commits are the work of an agent given #9 as its task, each stage its own commit. The next two are the mark on the one scenario that fails and the record of the runs, and the last mends what a review found.

What is played now

  • The scenarios driven by hand, with their answers written in the test, one session per answer: planning from the specs killed during the interview, an amendment killed once recorded, an amendment in the conversation and a sentence that agrees, a break on a commit of the developer refused, and accepted, the ceiling then its three ways on.
  • The status command and a branch with two plans: an abandon that waits for a yes, then the conformity check that fails on it; each of the three commands puts two plans to the developer and records nothing.
  • A CI that runs on a push (Exercise the pull request steps and the host CI snippet against GitHub #8): /surface-plan warns and waits before the first push, pushes and opens the draft on a yes, and pushes nothing on a no.
  • Sessions that died: in a fix, in a gate run, in a review, in the extraction, in the cross-check, and, from prepared states, between a record and its commit and between a commit and its push.
  • The ceiling in planning, and a break suspected in a slice, confirmed then refused, and dismissed, from prepared states.

How each is played and what it must hold is in tests/e2e/README.md, rewritten: nothing is left to drive by hand.

How

  • No model plays the developer here, which is what the evaluations do: every question of the chain carries a recommendation, so one written reply that takes it answers a question that changes from run to run.
  • A scenario asserts what the project holds once its sessions ended, the journal, the plan folder, git and the record of the stand-in gh, and of what a session said one word at most.
  • tests/e2e/toy/toy.py gains the prepared states those paths start from, each documented in the README and checked without a session by test_a_prepared_state_is_the_one_named. A state that holds work not committed or not pushed is what a death there leaves: an agent chains a record and its commit in one command, and leaves no moment to kill it at.
  • A kill now also takes what the session left at work in the project, since a gate runs in a process group of its own and would record its run while the next session works.
  • ARCHITECTURE.md and ADR 0025 say that the answers are written in the tests, where they said that those scenarios were played by hand.

Nothing of skills/, agents/, the stand-in gh or the image changes.

Played

Every scenario was played on the prompts of commit 4220424: 33 runs, 13.61 USD at list price. docs/e2e/2026-10-07.md keeps the verdicts and the cost of each. 21 of the 23 new scenarios passed their only run, one passed on its second, and one fails as the chain stands.

They were not played in one command, and every verdict but five rests on one run: how often a scenario fails for a path a model takes is not known.

What they found

Read by an independent reviewer

Before its merge, on 2026-10-08: fifteen findings, none of them a failure that repeats. The last commit mends those that cost little and that a run can check:

  • A look that took a lock. The scenarios looked at the repository of a running session with a plain git status, which takes the lock of the index, and could make a git add of that session fail: the reviewer counted 24 failures in 400 against a status asked without a pause. The look passes --no-optional-locks.
  • A session left running. A watch that raised left its session running to its end, billed, behind a scenario that had failed. The session is killed whatever the watch raises, and a stream read as it grows no longer raises on a character cut in two.
  • A kill in a slice with a loop of its own, without the guard that the moment came: the helper of the other kills plays it.
  • Smaller. An agent launched under the other name of the tool, Task, is read. A reply after a refusal goes to the session the refusal gave. The plan folder is read in the project, where a scenario named it from the date after its session. Three slips of the README.

The tests that start no session pass in the container, and the kill in a slice was played once more as it stands, on the prompts of commit 505848a: killed after the approval and before any slice was done, then passed, 0.43 USD.

Left as they are:

  • A kill that takes more than a real death does. It takes what the session left at work in its project, a gate under way among it. So no scenario plays what a kill -9 of claude leaves on a developer's machine: a gate that outlives its session and records its run under the next one. No run shows it as a fault of the chain. The README says it where it describes a kill by hand.
  • The strict mark of A session that dies between the commit of a conformant plan and its push leaves the push to nobody #114 covers its whole test: any failure of that scenario reads as the expected one until the mark goes.
  • A conversation takes the recommendation after every session that did not reach the state it waits for, without reading that a question was asked, twelve times at most.
  • The kill in a review does not assert that the relaunch launched a reviewer: a report the killed one left is recorded as it stands, which is the chain's rule, and the scenario passes on either path.
  • What a prepared state is made of is spread over several tables of toy.py, the README and the list of a test, and nothing checks that they agree.

Not done

…tten answers

The five scenarios tests/e2e/README.md gave a developer to drive are pytest
scenarios now: planning from the specs killed in the interview, an amendment
killed once recorded, an amendment in the conversation, a break on a commit of
the developer refused or accepted, and the three ways on after the ceiling.

The developer's answers are written in the tests and given one session each,
with --resume. Every question of the chain carries a recommendation, so one
reply that takes it answers any question of an interview; no model plays the
developer, which is what the evaluations do.

The toy gains the prepared states the paths of issue 9 start from. Those named
after a death hold what a session leaves in the files when it dies there. A
kill now also takes what a session left at work in its project, since a gate
runs in a process group of its own and would record its run after the kill.
`/surface-status abandon` records nothing before the developer's yes, then
abandons, commits the journal alone, pushes and refreshes the description; the
conformity check of the host's CI then fails on a plan abandoned after its
approval. On a branch that carries two plans, each of the three commands asks
which one and records nothing in either.
…rst push, played

The toy had no CI, so only one branch of the first push had ever run. A variant
of it holds a workflow that runs on every push and on every pull request:
`/surface-plan` drafts the plan, commits it, and waits for the developer's
agreement before anything leaves the machine, then pushes and opens the draft.
Declined, it pushes nothing and says the plan is committed locally. Without a
CI, a plan committed and never pushed is pushed at the relaunch, and its draft
opened, without a question.
…aunched, played

A session is killed while a fixer has changed code, while the gate of the plan
runs, while the reviewer works, while the extractor draws the blueprint and
while the checker reads it. Each relaunch keeps the journal of the dead session
as it was and takes the plan where its path leads.

The two moments a kill cannot hit, since an agent records and commits in one
command, start from a state that holds what a death there leaves: a slice
recorded and not committed, which the relaunch commits as it stands, and a
review written and not recorded, which it records with no second reviewer.
… prepared states

Planning that passed its ceiling, two cross-checks having counted the same
omission, records the block, launches nothing, and takes the developer's
instruction in the conversation: `resumed`, then the blueprint drawn and
cross-checked again.

A break an executor suspects waits in the journal for a reviewer, whatever
became of the session. Two states hold what it leaves: one on a command the
feature would replace without the blueprint saying so, which the reviewer
confirms and the developer then refuses, and one on a test helper the plan
does not name, which the reviewer dismisses with a note for the next executor.
…relaunch finds no plan

The last step of the loop commits `conformant`, then pushes and refreshes the
description. A state holds what a session leaves when it dies between the two,
and the scenario holds the relaunch to the push that stop still owes.

It fails as the chain stands, and is kept as it is: `/surface-execute` finds no
plan once the only one of the branch is conformant, says so and stops, so the
branch stays unpushed and the pull request keeps the description of a plan in
progress. Whether a prompt or the resolver takes that plan up again is the
chain's to decide.
…an agent cuts its commands

The scenario that kills a session in a slice waits for code an executor wrote
and has not committed. An executor that writes, tests, records and commits in
one command leaves that code uncommitted for a fraction of a second: the
session then runs to its end, and the scenario failed that way once.

Any state of the toy can now be built slow, with a test on the main branch
that lasts 20 seconds, so that the code stays uncommitted for as long as the
agent's tests run. The kill in a slice and the kill in a fix start from a
project built slow, as the kill in a gate run did under a state of its own,
which the option replaces. The kill in a slice now also takes what its session
left at work: the tests of the executor would outlive it, then record and
commit the slice under the session that takes over.
… until the chain takes that push up

A session that dies between the commit of a conformant plan and its push is taken up by
no relaunch: /surface-execute finds no plan once the only one of the branch is over, and
stops before any push (#114). The scenario that shows it fails by construction. Left red,
it would fail the whole set each time it is played, for a fault that is known and has its
issue.

It is marked as expected to fail, strictly: the day the chain takes that push up, the
scenario passes, the mark fails, and the mark goes. Played once so: reported as an
expected failure.
…nd their cost kept

The scenarios had last run on several versions of the prompts, and no single version had
run them all (#9). Each was played on the prompts of commit 4220424 as it was written,
the four that existed included: 33 runs, 13.61 USD at list price.

docs/e2e/2026-10-07.md keeps the verdict and the cost of each, what the runs found and
what they do not say. tests/e2e/README.md says where that record is, and that the one
scenario that fails as the chain stands is marked as expected to fail.
… leaves no session behind it

An independent review of the scenarios found no failure that repeats, and a harness that
could disturb the session it watches, or lose it.

- `dirty` asked for a plain `git status`, which takes the lock of the index. Asked every
  half second while a session commits, it can make a `git add` of that session fail on a
  lock the chain never took: the reviewer counted 24 failures in 400 against a status
  asked without a pause, and none with `--no-optional-locks`, which the look now passes.
- `kill_when` killed its session at the moment only. A watch that raised left the session
  running to its end, billed, behind a scenario that had failed: it is killed whatever
  the watch raises. `stream` reads a log as it grows, and no longer raises on a character
  the read cut in two.
- The scenario of a kill in a slice kept a loop of its own, without the guard that the
  moment came: `kill_when` plays it like the others.
- `launches` reads an agent launched under either name of the tool, `Agent` or `Task`, as
  the evaluations do. Under the other name it found none, and a scenario that asserts
  that no agent was launched held nothing.
- A reply that follows a refusal goes to the session the refusal gave, where it went to
  the one before it.
- The plan folder is read in the project where a scenario named it from the date a second
  time, after its session: a run that crossed midnight UTC looked for the folder of the
  next day.
- `tests/e2e/README.md`: the row of the conformity never pushed is back in its table, the
  neighbours of the prepared states are three, and the kill by hand says what the kill
  of a process group misses.

The tests that start no session pass in the container, 27 of them, and the scenario of a
kill in a slice was played once more as it now stands, on the prompts of commit 505848a:
killed after the approval and before any slice was done, then passed, 0.43 USD for the
session that took over. `docs/e2e/2026-10-07.md` says so, under what its runs do not say.

Left as they are, and said in the pull request: a kill that takes more than a real death
does, since it takes a gate under way with its session; the strict mark of #114, which
covers its whole test; a conversation that takes the recommendation without reading
that a question was asked; and the kill in a review, which does not assert that a
reviewer was launched again.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant