Repository navigation
Fix AI snapshot parsing and add visual snapshot controls - #57
Merged
Merged
Conversation
Playwright's AI snapshot is YAML. When an element's role, name, and
marks hold text that YAML would misread, Playwright puts that text in
YAML single quotes and writes each ' inside it as ''. A button named
"Status: live" comes out as
- 'button "Status: live" [ref=e4]'
Both Rust parsers of the snapshot read the role from the start of the
line, so they read it as 'button. ACT reported such a click as
CLICK role:'button "Status: live", and a ' in a name stayed as ''.
With --jev, the element fell out of Jev's lists of buttons, fields, and
scroll areas, because Outline sorts elements by role.
In playwright-core 1.62.1, renderAriaTree builds each key from the
role, the name in JSON double quotes, and marks such as [ref=e4]. It
wraps the key with yamlEscapeKeyIfNeeded when yamlStringNeedsQuotes is
true. For a key, that is when it holds ": ", whitespace before "#",
"{", "}", a backtick, or a control character that JSON does not escape
(DEL and U+0080 to U+009F). The text after a key, "- text:" lines, and
property lines such as "- /url:" never get single quotes:
yamlEscapeValueIfNeeded puts them in double quotes with backslash
escapes, which the parsers already read. A probe page with names and
texts that hold each YAML indicator gave these forms.
A new unquote_line in snapshot.rs removes the quotes from one line and
turns '' back into '. The indent and the text after the closing quote
stay as they are, and every other line comes back unchanged.
PageSnapshot and Outline call it on each line before they parse it, so
a quoted line reads as the same line without quotes does. The text the
model reads goes through it too, so the model sees
- button "It's: here" [ref=e7], not - 'button "It''s: here" [ref=e7]',
and cannot copy the doubled quote into an option.
The shim still writes a named iframe line without single quotes. Rust
now reads both forms, and the model never sees the quotes, so wrapping
would add code and change nothing. Its comment and the shim protocol
doc now give that reason, and a shim test keeps the line unwrapped.
Unit tests give both parsers the exact lines from the probe and check
that each gives the same result as the lines without quotes. Two
acceptance tests click a button named "Status: live": the model's
answer runs as CLICK role:button "Status: live", and with --jev, Jev
picks the button from the page's buttons without a model call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read snapshot lines that Playwright quotes
PageSnapshot took the first [ref=...] on a line as the element's ref,
even one in the element's name or in page text. With these lines from
Playwright 1.62.1
- generic [active] [ref=e1]:
- button "x\" [ref=e1] [cursor=pointer]" [ref=e6]: ignored
- button "Delete [ref=e9]" [ref=e4]
- text: see [ref=e3] here
e1, the page's body, mapped to the first button, and e3 to the text
line. So a report line could describe another element than the one
that ran, and a model or Jev choice of e4 or e6 failed with
act-invalid-decision. The action itself always used the ref that the
model or Jev chose. Jev's Outline already read a quoted name to its
end and took the marks after it, so it gave the real refs.
A new SnapshotLine in snapshot.rs now reads each line for both parsers.
It splits the line where its key ends: at the closing YAML quote, or
else at the first ": " or final ":" after the role and any name in
double quotes. In the key, the name ends at its closing double quote,
with backslash escapes honored. Only the text after the name holds the
ref and marks such as [active], [checked], and [cursor=pointer]. The
model's text drops [cursor=pointer] only from the marks, so a name that
holds that text stays whole.
In AI mode, renderAriaTree writes a name that starts and ends with "/"
as it is, with no quotes or escapes: `button /api/ [ref=e2]`. This is
not a regex. renderStringsAsRegex is off in AI mode, so the name keeps
its slashes, and getByRole("button", { name: "/api/", exact: true })
finds the button. Both parsers read no name for such a line, and the
Outline also lost its ref, so Jev never saw the element. Marks never
hold "/", so the name now ends at the key's last "/".
yamlEscapeValueIfNeeded writes a value with JSON's escapes plus \xHH
for other control characters. The Outline read double-quoted values
with serde_json, so "a\x7fb" kept its quotes and escapes. The saved
checkboxes eval page has such a text line. A new decoder reads JSON's
escapes and \xHH, and a lone surrogate, which JSON.stringify writes as
\ud800 and serde_json rejects, reads as U+FFFD.
A region that its own heading names through aria-labelledby shows in
the snapshot without a name, because the heading shows it. Jev's
description gave such a region the nearest heading before it, so in
the trip-planner eval page the Sunday column read as "Saturday, May
10". Now an element with no name or text that starts with a heading,
directly or inside plain generic wrappers, takes that heading as its
label and as its heading. The label slot already names an element that
has no name, and it took the text beside it, so the heading replaces
that text. The heading before such an element belongs to the section
before, so it is left out. Descriptions of other elements do not
change. On the 59 local eval pages, the rule applies to day columns,
landing-page sections, main areas, a tab panel, and link cards whose
first child is a heading.
Report lines stay `role:region`. A line such as `role:region "Sunday,
May 11"` has the form of an exact locator, but the snapshot cannot show
that the heading is the element's whole name: a link card's name holds
all of its text, and a main or generic element has no name at all.
Unit tests use the exact lines from a probe page in Chromium; Firefox
gives the same lines, and WebKit omits [cursor=pointer] on links. A
comparison of the old and the new parsers on the snapshots of all 59
local eval pages found no other difference. Three new acceptance tests
use Jev: a button named "Delete [ref=e3]" is clicked with its own ref
e4, a button named /api/ is among Jev's candidates, and a drop onto a
heading-named region shows Jev each region by its own heading. Each
fails without this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both jev: models ran all 59 local tasks three times at b78ad73. The earlier Jev runs moved to runs/archive/local/ with names that end in -before-snapshot-fixes. They hold three runs of each task: the 11 frame tasks from 08:52, 43 tasks from 15:35, and the 5 pointer tasks from 16:04. - jev:gemini/gemini-3.1-flash-lite: 177 of 177 passed, as before. Jev chose 163 of 189 actions (0.86), against 159 (0.84) before. - jev:openrouter/gpt-5.6-luna: 176 of 177 passed, against 177 before. Jev chose 162 of 189 actions (0.86), against 160 (0.85) before. The failed run is newsletter-signup. The fallback model, gpt-5.6-luna, answered FILL role:textbox "Email" "" instead of CLICK role:button "Subscribe". In all six runs of this task, before and after, Jev asked only its intent question, which reads the instruction and not the page, and the model got no hint. The model's prompt had 1,245 input tokens in each run, before and after, and the change leaves the model's text of this page as it was. The first Jev eval (056677d) also had 2 of 3 for this model and task. A comparison of the old and the new parser on snapshots of all 59 pages, taken as the shim takes them, shows where Jev now reads something else: the container candidates on seven pages (the trip planner's day regions, the landing page's sections, the main areas of four pages, and the support console's tab panel), 29 footer link cards on the saved Amazon page, and one text value on the saved checkboxes page. The model's text changes on no page. The drag and scroll tasks: - pointer-drag-delay: Jev split its intent vote in all 12 runs, before and after, so the model planned every run, and Jev never read the day regions. All runs passed. - pointer-drag-distance: Jev dropped on role:region "Onsite" in all 12 runs. Its drop question grew by 40 tokens for the main area's heading. - scroll-sideways: Jev scrolled role:region "Photos" right in all 12 runs. Its question grew by 52 tokens for the main area's heading. - kanban-move, scroll-into-view, scroll-back-up, scroll-halfway, scroll-in-frame, scroll-panel-chunk, and middle-click-tab: the same lines, all chosen by Jev. The other changes in Jev's share do not come from this change. In large-directory, the same 1,446-token intent request led to a pick in 1 of 3 runs per model, against 0 before. The 43-task session before was built without the quoted-line fix of 5805069: its synonym-toolbar descriptions lack "heading: Draft: Quarterly update", which comes from a quoted heading line. So the Jev requests of synonym-toolbar and saved-google-flights, a page with 49 quoted lines, grew because of that fix. The session cost $0.1048: $0.0584 for gemini-3.1-flash-lite and $0.0464 for gpt-5.6-luna, Jev's requests included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No eval task ran these ACT paths: scrollIntoView, scrollRight, a middle click, and dragAndDrop on boards with their own pointer-event drag code. Each new task has its own page: - scroll-into-view: "scroll to the pricing section" on a product page. Its Pricing section fetches its plans only when an IntersectionObserver sees it. The assert checks that the Team plan shows. - scroll-sideways: "scroll the photo strip to the right" on a rental listing with an overflow-x photo strip. The strip must scroll, and the page must not. - middle-click-tab: "middle-click the Reports tab to close it" in a support console. A middle click closes a tab through auxclick, and a left click only selects it. Only the active tab has a close button. - pointer-drag-distance: "move Priya Patel to the Onsite column" on a hiring board whose drag starts after an 8px move. - pointer-drag-delay: "move the tram ride to Sunday" in a trip planner whose drag starts after a 300ms press. A move of more than 5px before that cancels the drag. Both boards use pointer events and no native drag and drop. They drop on the column under the pointer, with the dragged card set to pointer-events: none, as the DRAG test page does. Each flow was run with its ACT line replaced by the line it should produce, such as MIDDLECLICK role:tab Reports. Those lines pass. Wrong actions fail the asserts: a left or right click on the tab, a scroll of the page, a click on a card, and a drop on the wrong column. With raw mouse input, the planner cancels a drag held for less than 300ms, and the hiring board starts no drag on a 6px move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The local set has 59 tasks, and a local session targets 10 runs per task by default, so it is about 590 runs per model, not 480. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Five models ran the five new tasks three times each. All 75 runs pass. The models and Jev chose these lines: - scroll-into-view: SCROLL role:heading "Pricing" 11 times, and the Pricing section's own region 4 times (gpt-5.6-terra 3, gpt-5.6-luna 1). The region has no name in the snapshot, because its name comes from its own heading. - scroll-sideways: SCROLL role:region "Photos" right every time. - middle-click-tab: MIDDLECLICK role:tab "Reports" every time. - pointer-drag-distance: DRAG from Priya Patel's card to role:region "Onsite". Jev dragged the name inside the card, which also works, because the pointerdown reaches the card. - pointer-drag-delay: DRAG from the Tram 28 ride item to the Sunday heading, list, or region. With --jev, Jev planned 24 of 30 actions without the model: every action in the first four tasks. For "move the tram ride to Sunday", Jev split its vote on the kind of action (drag 0.44, click 0.28, select 0.17), so the model planned all 6 runs of pointer-drag-delay. For "move Priya Patel to the Onsite column", Jev voted drag 1.0. The session cost $0.0525, and gpt-5.6-terra spent $0.0350 of it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SNAPSHOT name locator compares one element instead of the full page, as in SNAPSHOT cart testid:cart. The locator follows the name and takes prefixed segments only, like an ACT scope. It must match exactly one element and resolves again for every capture. Masks still resolve from the page, and percentages count the element's pixels. Playwright 1.62.1 resolves the element once per locator.screenshot() and throws when the page replaces it, so the shim retries detached, zero-size, and not-visible captures within the step timeout. After a stable mismatch, a capture that times out checks the target once without waiting: a missing or hidden target fails as a timeout instead of a mismatch on old pixels. The snapshot command carries a required target, so the shim protocol moves to 4. JSON and HTML reports record the target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test page replaced the snapshot target every 30 ms forever, so on the slower CI runners a capture never found a steady element within the step's 3 s. The page now stops replacing it after the first second, and the step gets 5 s: early captures still have to find the replaced target again, and a slow machine still gets a stable one in time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add ACT evals for five untested pointer methods
Add element snapshots
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Preserve Playwright names, refs, and quoted snapshot lines. Add five pointer evals, snapshot masks and tolerances, and element snapshots with their browser coverage.
Recovers the intended changes from #31, #32, #33, #34, #43, which were merged into the old development stack but did not reach
main. Preserves the newerlithos-llmpin from #54.The remaining work on
pointer-buttonsis organized into the following dependent PRs. Each PR targets the preceding recovery branch so its diff shows only its own stage.Merge in order with merge commits, then retarget the next PR to
main. The last three PRs remain draft for architecture/provider review; the crate-layout ADR is still draft. No release tags are created by this recovery.Completeness check: the final branch contains every commit reachable from
pointer-buttonsat1ced9e6and frommainat163dca6. Its tree differs frompointer-buttonsonly in the manifest and lockfile lines retaining the newerlithos-llmpin from #54. Repeated history from earlier branch updates is retained without duplicating the implementation.Validation:
CARGO_NET_OFFLINE=true mise run checkpassed on this branch. This runs formatting, Clippy, local Rust/browser tests, shim checks, eval tooling, and workflow checks. Paid model evals, the external BrowserSim round trip, and live Kernel checks were not rerun.