Repository navigation
Read snapshot names, refs, and values as Playwright writes them - #33
Merged
Merged
Conversation
brynary
added this pull request to stack #55
October 5, 2026 13:05
brynary
force-pushed
the
snapshot-parse-fixes
branch
from
October 5, 2026 13:21
b711149 to
a84ae0b
Compare
PageSnapshot took the first [ref=...] on a line as the element's ref,
even one in the element's name or in page text. With these lines from
Playwright 1.62.1
- generic [active] [ref=e1]:
- button "x\" [ref=e1] [cursor=pointer]" [ref=e6]: ignored
- button "Delete [ref=e9]" [ref=e4]
- text: see [ref=e3] here
e1, the page's body, mapped to the first button, and e3 to the text
line. So a report line could describe another element than the one
that ran, and a model or Jev choice of e4 or e6 failed with
act-invalid-decision. The action itself always used the ref that the
model or Jev chose. Jev's Outline already read a quoted name to its
end and took the marks after it, so it gave the real refs.
A new SnapshotLine in snapshot.rs now reads each line for both parsers.
It splits the line where its key ends: at the closing YAML quote, or
else at the first ": " or final ":" after the role and any name in
double quotes. In the key, the name ends at its closing double quote,
with backslash escapes honored. Only the text after the name holds the
ref and marks such as [active], [checked], and [cursor=pointer]. The
model's text drops [cursor=pointer] only from the marks, so a name that
holds that text stays whole.
In AI mode, renderAriaTree writes a name that starts and ends with "/"
as it is, with no quotes or escapes: `button /api/ [ref=e2]`. This is
not a regex. renderStringsAsRegex is off in AI mode, so the name keeps
its slashes, and getByRole("button", { name: "/api/", exact: true })
finds the button. Both parsers read no name for such a line, and the
Outline also lost its ref, so Jev never saw the element. Marks never
hold "/", so the name now ends at the key's last "/".
yamlEscapeValueIfNeeded writes a value with JSON's escapes plus \xHH
for other control characters. The Outline read double-quoted values
with serde_json, so "a\x7fb" kept its quotes and escapes. The saved
checkboxes eval page has such a text line. A new decoder reads JSON's
escapes and \xHH, and a lone surrogate, which JSON.stringify writes as
\ud800 and serde_json rejects, reads as U+FFFD.
A region that its own heading names through aria-labelledby shows in
the snapshot without a name, because the heading shows it. Jev's
description gave such a region the nearest heading before it, so in
the trip-planner eval page the Sunday column read as "Saturday, May
10". Now an element with no name or text that starts with a heading,
directly or inside plain generic wrappers, takes that heading as its
label and as its heading. The label slot already names an element that
has no name, and it took the text beside it, so the heading replaces
that text. The heading before such an element belongs to the section
before, so it is left out. Descriptions of other elements do not
change. On the 59 local eval pages, the rule applies to day columns,
landing-page sections, main areas, a tab panel, and link cards whose
first child is a heading.
Report lines stay `role:region`. A line such as `role:region "Sunday,
May 11"` has the form of an exact locator, but the snapshot cannot show
that the heading is the element's whole name: a link card's name holds
all of its text, and a main or generic element has no name at all.
Unit tests use the exact lines from a probe page in Chromium; Firefox
gives the same lines, and WebKit omits [cursor=pointer] on links. A
comparison of the old and the new parsers on the snapshots of all 59
local eval pages found no other difference. Three new acceptance tests
use Jev: a button named "Delete [ref=e3]" is clicked with its own ref
e4, a button named /api/ is among Jev's candidates, and a drop onto a
heading-named region shows Jev each region by its own heading. Each
fails without this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both jev: models ran all 59 local tasks three times at b78ad73. The earlier Jev runs moved to runs/archive/local/ with names that end in -before-snapshot-fixes. They hold three runs of each task: the 11 frame tasks from 08:52, 43 tasks from 15:35, and the 5 pointer tasks from 16:04. - jev:gemini/gemini-3.1-flash-lite: 177 of 177 passed, as before. Jev chose 163 of 189 actions (0.86), against 159 (0.84) before. - jev:openrouter/gpt-5.6-luna: 176 of 177 passed, against 177 before. Jev chose 162 of 189 actions (0.86), against 160 (0.85) before. The failed run is newsletter-signup. The fallback model, gpt-5.6-luna, answered FILL role:textbox "Email" "" instead of CLICK role:button "Subscribe". In all six runs of this task, before and after, Jev asked only its intent question, which reads the instruction and not the page, and the model got no hint. The model's prompt had 1,245 input tokens in each run, before and after, and the change leaves the model's text of this page as it was. The first Jev eval (056677d) also had 2 of 3 for this model and task. A comparison of the old and the new parser on snapshots of all 59 pages, taken as the shim takes them, shows where Jev now reads something else: the container candidates on seven pages (the trip planner's day regions, the landing page's sections, the main areas of four pages, and the support console's tab panel), 29 footer link cards on the saved Amazon page, and one text value on the saved checkboxes page. The model's text changes on no page. The drag and scroll tasks: - pointer-drag-delay: Jev split its intent vote in all 12 runs, before and after, so the model planned every run, and Jev never read the day regions. All runs passed. - pointer-drag-distance: Jev dropped on role:region "Onsite" in all 12 runs. Its drop question grew by 40 tokens for the main area's heading. - scroll-sideways: Jev scrolled role:region "Photos" right in all 12 runs. Its question grew by 52 tokens for the main area's heading. - kanban-move, scroll-into-view, scroll-back-up, scroll-halfway, scroll-in-frame, scroll-panel-chunk, and middle-click-tab: the same lines, all chosen by Jev. The other changes in Jev's share do not come from this change. In large-directory, the same 1,446-token intent request led to a pick in 1 of 3 runs per model, against 0 before. The 43-task session before was built without the quoted-line fix of 5805069: its synonym-toolbar descriptions lack "heading: Draft: Quarterly update", which comes from a quoted heading line. So the Jev requests of synonym-toolbar and saved-google-flights, a page with 49 quoted lines, grew because of that fix. The session cost $0.1048: $0.0584 for gemini-3.1-flash-lite and $0.0464 for gpt-5.6-luna, Jev's requests included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The local set has 59 tasks, and a local session targets 10 runs per task by default, so it is about 590 runs per model, not 480. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brynary
force-pushed
the
snapshot-parse-fixes
branch
from
October 5, 2026 14:02
a84ae0b to
b28e2b8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #32. Four older snapshot parsing bugs, found while building #28, #31, and #32. Both Rust parsers,
PageSnapshotand Jev'sOutline, now read each line through one sharedSnapshotLine. The tests use exact lines from Playwright 1.62.1, which are the same in Chromium and Firefox; WebKit only leaves out[cursor=pointer]on links.Fixes
[ref=inside a name or text- button "Delete [ref=e9]" [ref=e4];- text: see [ref=e3] herePageSnapshottook the first[ref=on a line, so it mappede3to a text line ande1to a button. Report lines then described the wrong element, and choosing the real ref failed withact-invalid-decision. The action always used the ref the planner chose, so nothing ever ran on the wrong element. Jev'sOutlinewas already right. Now the ref and marks come only from the text after the name, which honors escapes.- region [ref=e32]:whose first child is- heading "Sunday, May 11"aria-labelledbyto its own heading, Playwright shows the region with no name. Jev's description then took the heading before it ("Saturday, May 10"). Now an element with no name or text that starts with a heading, looking through plain wrappers, takes that heading as itslabelandheading. Report lines stayrole:region: a true but vague line is better than a precise-looking guess.- button /api/ [ref=e2];- link /docs/ [ref=e11] [cursor=pointer]:getByRole(..., { name: "/api/" })finds it. Both parsers read no name, and Jev lost the element entirely. Now the name keeps its slashes:role:button "/api/".\xHHin values- paragraph [ref=e2]: "a\x7fb"; the saved checkboxes page has `- text: "\x1cOn snapshots of all 59 local eval pages, the old and new parsers give the same model text and the same ref-to-element maps. The only node difference is the
\x1cvalue. The description change in fix 2 touches only unnamed elements that open with a heading: day columns, landing-page sections,mainareas, one tab panel, and Amazon's footer link cards, which before took the previous card's heading.Tests
ACT:a_ref_in_a_name_cannot_redirect_jevs_pick: the button namedDelete [ref=e3]keeps its real ref, and the click lands on it. On the old code this test failed with "e4 … is not in the page snapshot".jev_reads_a_region_that_its_own_heading_names_by_that_heading: a drop onto such a region.jev_picks_a_button_whose_name_starts_and_ends_with_a_slash.mise run checkpasses. SPEC §7.4 (what Jev reads) and the CHANGELOG describe the change.Regression eval
Both
jev:variants, all 59 local tasks, three runs each, atb78ad73. Summary:evals/act/results/2026-09-27-local-jev-snapshot-fixes.md. Cost $0.10.The one failure is
newsletter-signup, where the fallback model answeredFILL role:textbox "Email" ""instead of clicking Subscribe. This PR does not cause it: Jev asked only its intent question, which reads no page, and the model got the same 1,245-token prompt in every run before and after. The first Jev eval also scored 2/3 for this model on this task. The drag, scroll, and middle-click tasks produce the same lines as before.Also
b711149: the eval README now says a local session is about 590 runs per model: 59 tasks at the default 10 runs each.🤖 Generated with Claude Code