Skip to content

Replace check sections with ASSERT and CAPTURE lines - #35

Merged
brynary merged 17 commits into
pointer-buttonsfrom
check-keywords
Oct 5, 2026
Merged

Replace check sections with ASSERT and CAPTURE lines#35
brynary merged 17 commits into
pointer-buttonsfrom
check-keywords

Conversation

@brynary

@brynary brynary commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

Stacked on element-snapshots. Phase 0 of the AI steps plan. This PR contains only the check keyword changes.

Replace the [Asserts] and [Captures] sections with ASSERT and CAPTURE lines, in browser and HTTP entries. The check language does not change. Check lines run in the order written, so a CAPTURE can come before an ASSERT that reads its value.

This is release A of the migration. Sections still parse, and whirl check warns about each header with sections-deprecated. A file that mixes sections and check lines is the new parse error mixed-check-syntax, which check --json reports by its own code. whirl fmt rewrites sections as check lines and keeps a comment that was on a header line.

Every flow in the repository moves to the new form: SPEC examples, README, examples, evals, trycmd fixtures, and flows inside Rust tests. The SPEC (sections 1 to 17) and the summary of ADR evaluate-checks-in-rust use the new wording. A new browser test covers a CAPTURE that feeds a later ASSERT in the same entry.

The last commit prepares release 0.13.0 (version and CHANGELOG heading). Nothing is tagged.

Validation: mise run check.

🤖 Generated with Claude Code

brynary and others added 15 commits September 27, 2026 15:24
@brynary @claude
Read snapshot lines that Playwright quotes
5805069 
Playwright's AI snapshot is YAML. When an element's role, name, and
marks hold text that YAML would misread, Playwright puts that text in
YAML single quotes and writes each ' inside it as ''. A button named
"Status: live" comes out as

  - 'button "Status: live" [ref=e4]'

Both Rust parsers of the snapshot read the role from the start of the
line, so they read it as 'button. ACT reported such a click as
CLICK role:'button "Status: live", and a ' in a name stayed as ''.
With --jev, the element fell out of Jev's lists of buttons, fields, and
scroll areas, because Outline sorts elements by role.

In playwright-core 1.62.1, renderAriaTree builds each key from the
role, the name in JSON double quotes, and marks such as [ref=e4]. It
wraps the key with yamlEscapeKeyIfNeeded when yamlStringNeedsQuotes is
true. For a key, that is when it holds ": ", whitespace before "#",
"{", "}", a backtick, or a control character that JSON does not escape
(DEL and U+0080 to U+009F). The text after a key, "- text:" lines, and
property lines such as "- /url:" never get single quotes:
yamlEscapeValueIfNeeded puts them in double quotes with backslash
escapes, which the parsers already read. A probe page with names and
texts that hold each YAML indicator gave these forms.

A new unquote_line in snapshot.rs removes the quotes from one line and
turns '' back into '. The indent and the text after the closing quote
stay as they are, and every other line comes back unchanged.
PageSnapshot and Outline call it on each line before they parse it, so
a quoted line reads as the same line without quotes does. The text the
model reads goes through it too, so the model sees
- button "It's: here" [ref=e7], not - 'button "It''s: here" [ref=e7]',
and cannot copy the doubled quote into an option.

The shim still writes a named iframe line without single quotes. Rust
now reads both forms, and the model never sees the quotes, so wrapping
would add code and change nothing. Its comment and the shim protocol
doc now give that reason, and a shim test keeps the line unwrapped.

Unit tests give both parsers the exact lines from the probe and check
that each gives the same result as the lines without quotes. Two
acceptance tests click a button named "Status: live": the model's
answer runs as CLICK role:button "Status: live", and with --jev, Jev
picks the button from the page's buttons without a model call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary @claude
Merge drop-file into snapshot-quoted-lines
a9df34d 
Brings the blank-recording warning from #29 through #30.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary @claude
Add ACT eval tasks for five untested methods
08c4fdc 
No eval task ran these ACT paths: scrollIntoView, scrollRight, a middle
click, and dragAndDrop on boards with their own pointer-event drag code.
Each new task has its own page:

- scroll-into-view: "scroll to the pricing section" on a product page.
  Its Pricing section fetches its plans only when an
  IntersectionObserver sees it. The assert checks that the Team plan
  shows.
- scroll-sideways: "scroll the photo strip to the right" on a rental
  listing with an overflow-x photo strip. The strip must scroll, and
  the page must not.
- middle-click-tab: "middle-click the Reports tab to close it" in a
  support console. A middle click closes a tab through auxclick, and a
  left click only selects it. Only the active tab has a close button.
- pointer-drag-distance: "move Priya Patel to the Onsite column" on a
  hiring board whose drag starts after an 8px move.
- pointer-drag-delay: "move the tram ride to Sunday" in a trip planner
  whose drag starts after a 300ms press. A move of more than 5px before
  that cancels the drag.

Both boards use pointer events and no native drag and drop. They drop
on the column under the pointer, with the dragged card set to
pointer-events: none, as the DRAG test page does.

Each flow was run with its ACT line replaced by the line it should
produce, such as MIDDLECLICK role:tab Reports. Those lines pass. Wrong
actions fail the asserts: a left or right click on the tab, a scroll of
the page, a click on a card, and a drop on the wrong column. With raw
mouse input, the planner cancels a drag held for less than 300ms, and
the hiring board starts no drag on a 6px move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary @claude
Run the evals for the five new ACT tasks
e5345d4 
Five models ran the five new tasks three times each. All 75 runs pass.
The models and Jev chose these lines:

- scroll-into-view: SCROLL role:heading "Pricing" 11 times, and the
  Pricing section's own region 4 times (gpt-5.6-terra 3, gpt-5.6-luna
  1). The region has no name in the snapshot, because its name comes
  from its own heading.
- scroll-sideways: SCROLL role:region "Photos" right every time.
- middle-click-tab: MIDDLECLICK role:tab "Reports" every time.
- pointer-drag-distance: DRAG from Priya Patel's card to role:region
  "Onsite". Jev dragged the name inside the card, which also works,
  because the pointerdown reaches the card.
- pointer-drag-delay: DRAG from the Tram 28 ride item to the Sunday
  heading, list, or region.

With --jev, Jev planned 24 of 30 actions without the model: every
action in the first four tasks. For "move the tram ride to Sunday",
Jev split its vote on the kind of action (drag 0.44, click 0.28,
select 0.17), so the model planned all 6 runs of pointer-drag-delay.
For "move Priya Patel to the Onsite column", Jev voted drag 1.0.

The session cost $0.0525, and gpt-5.6-terra spent $0.0350 of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary @claude
Read snapshot names, refs, and values as Playwright writes them
b78ad73 
PageSnapshot took the first [ref=...] on a line as the element's ref,
even one in the element's name or in page text. With these lines from
Playwright 1.62.1

  - generic [active] [ref=e1]:
  - button "x\" [ref=e1] [cursor=pointer]" [ref=e6]: ignored
  - button "Delete [ref=e9]" [ref=e4]
  - text: see [ref=e3] here

e1, the page's body, mapped to the first button, and e3 to the text
line. So a report line could describe another element than the one
that ran, and a model or Jev choice of e4 or e6 failed with
act-invalid-decision. The action itself always used the ref that the
model or Jev chose. Jev's Outline already read a quoted name to its
end and took the marks after it, so it gave the real refs.

A new SnapshotLine in snapshot.rs now reads each line for both parsers.
It splits the line where its key ends: at the closing YAML quote, or
else at the first ": " or final ":" after the role and any name in
double quotes. In the key, the name ends at its closing double quote,
with backslash escapes honored. Only the text after the name holds the
ref and marks such as [active], [checked], and [cursor=pointer]. The
model's text drops [cursor=pointer] only from the marks, so a name that
holds that text stays whole.

In AI mode, renderAriaTree writes a name that starts and ends with "/"
as it is, with no quotes or escapes: `button /api/ [ref=e2]`. This is
not a regex. renderStringsAsRegex is off in AI mode, so the name keeps
its slashes, and getByRole("button", { name: "/api/", exact: true })
finds the button. Both parsers read no name for such a line, and the
Outline also lost its ref, so Jev never saw the element. Marks never
hold "/", so the name now ends at the key's last "/".

yamlEscapeValueIfNeeded writes a value with JSON's escapes plus \xHH
for other control characters. The Outline read double-quoted values
with serde_json, so "a\x7fb" kept its quotes and escapes. The saved
checkboxes eval page has such a text line. A new decoder reads JSON's
escapes and \xHH, and a lone surrogate, which JSON.stringify writes as
\ud800 and serde_json rejects, reads as U+FFFD.

A region that its own heading names through aria-labelledby shows in
the snapshot without a name, because the heading shows it. Jev's
description gave such a region the nearest heading before it, so in
the trip-planner eval page the Sunday column read as "Saturday, May
10". Now an element with no name or text that starts with a heading,
directly or inside plain generic wrappers, takes that heading as its
label and as its heading. The label slot already names an element that
has no name, and it took the text beside it, so the heading replaces
that text. The heading before such an element belongs to the section
before, so it is left out. Descriptions of other elements do not
change. On the 59 local eval pages, the rule applies to day columns,
landing-page sections, main areas, a tab panel, and link cards whose
first child is a heading.

Report lines stay `role:region`. A line such as `role:region "Sunday,
May 11"` has the form of an exact locator, but the snapshot cannot show
that the heading is the element's whole name: a link card's name holds
all of its text, and a main or generic element has no name at all.

Unit tests use the exact lines from a probe page in Chromium; Firefox
gives the same lines, and WebKit omits [cursor=pointer] on links. A
comparison of the old and the new parsers on the snapshots of all 59
local eval pages found no other difference. Three new acceptance tests
use Jev: a button named "Delete [ref=e3]" is clicked with its own ref
e4, a button named /api/ is among Jev's candidates, and a drop onto a
heading-named region shows Jev each region by its own heading. Each
fails without this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary @claude
Run the Jev evals with the snapshot parse fixes
dbaccbb 
Both jev: models ran all 59 local tasks three times at b78ad73. The
earlier Jev runs moved to runs/archive/local/ with names that end in
-before-snapshot-fixes. They hold three runs of each task: the 11 frame
tasks from 08:52, 43 tasks from 15:35, and the 5 pointer tasks from
16:04.

- jev:gemini/gemini-3.1-flash-lite: 177 of 177 passed, as before. Jev
  chose 163 of 189 actions (0.86), against 159 (0.84) before.
- jev:openrouter/gpt-5.6-luna: 176 of 177 passed, against 177 before.
  Jev chose 162 of 189 actions (0.86), against 160 (0.85) before.

The failed run is newsletter-signup. The fallback model, gpt-5.6-luna,
answered FILL role:textbox "Email" "" instead of CLICK role:button
"Subscribe". In all six runs of this task, before and after, Jev asked
only its intent question, which reads the instruction and not the page,
and the model got no hint. The model's prompt had 1,245 input tokens in
each run, before and after, and the change leaves the model's text of
this page as it was. The first Jev eval (056677d) also had 2 of 3 for
this model and task.

A comparison of the old and the new parser on snapshots of all 59
pages, taken as the shim takes them, shows where Jev now reads
something else: the container candidates on seven pages (the trip
planner's day regions, the landing page's sections, the main areas of
four pages, and the support console's tab panel), 29 footer link cards
on the saved Amazon page, and one text value on the saved checkboxes
page. The model's text changes on no page.

The drag and scroll tasks:

- pointer-drag-delay: Jev split its intent vote in all 12 runs, before
  and after, so the model planned every run, and Jev never read the day
  regions. All runs passed.
- pointer-drag-distance: Jev dropped on role:region "Onsite" in all 12
  runs. Its drop question grew by 40 tokens for the main area's heading.
- scroll-sideways: Jev scrolled role:region "Photos" right in all 12
  runs. Its question grew by 52 tokens for the main area's heading.
- kanban-move, scroll-into-view, scroll-back-up, scroll-halfway,
  scroll-in-frame, scroll-panel-chunk, and middle-click-tab: the same
  lines, all chosen by Jev.

The other changes in Jev's share do not come from this change. In
large-directory, the same 1,446-token intent request led to a pick in
1 of 3 runs per model, against 0 before. The 43-task session before was
built without the quoted-line fix of 5805069: its synonym-toolbar
descriptions lack "heading: Draft: Quarterly update", which comes from a
quoted heading line. So the Jev requests of synonym-toolbar and
saved-google-flights, a page with 49 quoted lines, grew because of that
fix.

The session cost $0.1048: $0.0584 for gemini-3.1-flash-lite and $0.0464
for gpt-5.6-luna, Jev's requests included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary @claude
Update the local eval run count in the README
b711149 
The local set has 59 tasks, and a local session targets 10 runs per task
by default, so it is about 590 runs per model, not 480.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary
Add snapshot masks and comparison tolerances
24a0572
@brynary @claude
Add element snapshots
b993c69 
SNAPSHOT name locator compares one element instead of the full page,
as in SNAPSHOT cart testid:cart. The locator follows the name and takes
prefixed segments only, like an ACT scope. It must match exactly one
element and resolves again for every capture. Masks still resolve from
the page, and percentages count the element's pixels.

Playwright 1.62.1 resolves the element once per locator.screenshot()
and throws when the page replaces it, so the shim retries detached,
zero-size, and not-visible captures within the step timeout. After a
stable mismatch, a capture that times out checks the target once
without waiting: a missing or hidden target fails as a timeout instead
of a mismatch on old pixels.

The snapshot command carries a required target, so the shim protocol
moves to 4. JSON and HTML reports record the target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary @claude
Add the snapshot comparison settings to the changelog
63041c0 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary
Merge branch 'snapshot-comparison-options' into element-snapshots
f4b7c0f
@brynary @claude
Add element snapshots to the changelog
287eb96 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary @claude
Replace check sections with ASSERT and CAPTURE lines
01b4d8a 
Check lines run in the order written. Sections still parse with the
sections-deprecated warning, a file that mixes the two forms is the
mixed-check-syntax parse error, and whirl fmt rewrites sections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary @claude
Release 0.13.0
6585bb8 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary @claude
Let the replaced-target snapshot test settle on slow machines
11a425a 
The test page replaced the snapshot target every 30 ms forever, so on
the slower CI runners a capture never found a steady element within the
step's 3 s. The page now stops replacing it after the first second, and
the step gets 5 s: early captures still have to find the replaced target
again, and a slow machine still gets a stable one in time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary brynary mentioned this pull request Sep 28, 2026
Merged
@brynary @claude
Merge branch 'element-snapshots' into check-keywords
73694ab 
Brings the fix for the replaced-target snapshot test up the stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary brynary mentioned this pull request Sep 29, 2026
Merged
@brynary
brynary added this pull request to stack #55 October 5, 2026 13:22
Base automatically changed from element-snapshots to pointer-buttons October 5, 2026 14:08
@brynary
Merge pointer-buttons into check-keywords preserving ASSERT and CAPTU…
02be205 
…RE syntax

Keep the migrated check syntax in documentation, parser and lint tests, browser tests, and eval flows. Retain the 0.13.0 release entry and incorporate the base branch's expanded iframe eval results.
@brynary
brynary merged commit e244508 into pointer-buttons Oct 5, 2026
3 checks passed
This was referenced Oct 5, 2026
Merged
Merged
@brynary
brynary deleted the check-keywords branch October 5, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@brynary