Skip to content

Read snapshot lines that Playwright quotes - #31

Merged
brynary merged 1 commit into
pointer-buttonsfrom
snapshot-quoted-lines
Oct 5, 2026
Merged

brynary merged 1 commit into
pointer-buttonsfrom
snapshot-quoted-lines

Conversation

@brynary

@brynary brynary commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Stacked on #30. Playwright wraps an element line of its AI snapshot in YAML single quotes when the name holds : , #, a brace, a backtick, or a control character: - 'button "Status: live" [ref=e2]'. Both Rust parsers read the role from the start of the line, so they read 'button. The element dropped out of Jev's role-based candidates, and reports showed CLICK role:'button "Status: live". The bug is older than this stack; #28 found it.

Playwright's rule (1.62.1)

renderAriaTree builds each key from the role, the name as JSON.stringify(name), and marks such as [ref=e4]. It wraps the key in '…', doubling any inner ', when yamlStringNeedsQuotes(key) is true. Values are only ever double-quoted, never single-quoted. Observed:

- 'button "Status: live" [ref=e2]': b0
- 'button "It''s: here" [ref=e7]': b5
- 'region "Q3: plan" [ref=e37]':
- 'textbox "Note: x" [active] [ref=e57]': "Status: live"
- 'link "x #y" [ref=e64] [cursor=pointer]':
  - /url: "#top"

Fix

One helper, unquote_line, removes the wrapping quotes and turns '' back into '. It keeps the indent and the text after the closing quote. PageSnapshot, its condense step, and Jev's Outline all read each line through it, and every other line is unchanged. The model also reads unwrapped lines, so every element line has one form, and it cannot copy a doubled '' into an argument.

The shim's iframe names (#28) stay unwrapped: Rust now reads both forms, and the model never sees the quotes.

Tests

  • Unit: both parsers read the exact wrapped lines above to the same nodes and the same model text as the unwrapped lines.
  • ACT: the model clicks a button named "Status: live", and the report line is CLICK role:button "Status: live"; with --jev, Jev offers that button among the buttons and picks it with no model call.
  • With the helper turned off, all four new unit tests and both acceptance tests fail, and the old report line reads CLICK role:'button "Status: live".
  • mise run check passes. SPEC §7.4, the shim protocol doc, and the CHANGELOG describe the change.

Found, not fixed

  • A name that looks like a regex, such as - button /api/ [ref=e34], gets no JSON quotes; both parsers read no name, and Jev's parser also loses the ref.
  • \xHH escapes in values are not decoded.
  • parse_line takes the first [ref= on a line, even one inside a name.

🤖 Generated with Claude Code

@brynary
brynary added this pull request to stack #55 October 5, 2026 13:05
Base automatically changed from drop-file to pointer-buttons October 5, 2026 13:21
Playwright's AI snapshot is YAML. When an element's role, name, and
marks hold text that YAML would misread, Playwright puts that text in
YAML single quotes and writes each ' inside it as ''. A button named
"Status: live" comes out as

  - 'button "Status: live" [ref=e4]'

Both Rust parsers of the snapshot read the role from the start of the
line, so they read it as 'button. ACT reported such a click as
CLICK role:'button "Status: live", and a ' in a name stayed as ''.
With --jev, the element fell out of Jev's lists of buttons, fields, and
scroll areas, because Outline sorts elements by role.

In playwright-core 1.62.1, renderAriaTree builds each key from the
role, the name in JSON double quotes, and marks such as [ref=e4]. It
wraps the key with yamlEscapeKeyIfNeeded when yamlStringNeedsQuotes is
true. For a key, that is when it holds ": ", whitespace before "#",
"{", "}", a backtick, or a control character that JSON does not escape
(DEL and U+0080 to U+009F). The text after a key, "- text:" lines, and
property lines such as "- /url:" never get single quotes:
yamlEscapeValueIfNeeded puts them in double quotes with backslash
escapes, which the parsers already read. A probe page with names and
texts that hold each YAML indicator gave these forms.

A new unquote_line in snapshot.rs removes the quotes from one line and
turns '' back into '. The indent and the text after the closing quote
stay as they are, and every other line comes back unchanged.
PageSnapshot and Outline call it on each line before they parse it, so
a quoted line reads as the same line without quotes does. The text the
model reads goes through it too, so the model sees
- button "It's: here" [ref=e7], not - 'button "It''s: here" [ref=e7]',
and cannot copy the doubled quote into an option.

The shim still writes a named iframe line without single quotes. Rust
now reads both forms, and the model never sees the quotes, so wrapping
would add code and change nothing. Its comment and the shim protocol
doc now give that reason, and a shim test keeps the line unwrapped.

Unit tests give both parsers the exact lines from the probe and check
that each gives the same result as the lines without quotes. Two
acceptance tests click a button named "Status: live": the model's
answer runs as CLICK role:button "Status: live", and with --jev, Jev
picks the button from the page's buttons without a model call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary
brynary force-pushed the snapshot-quoted-lines branch from a9df34d to b99e1d5 Compare October 5, 2026 13:21
@brynary
brynary merged commit 4e0db1d into pointer-buttons Oct 5, 2026
3 checks passed
@brynary
brynary deleted the snapshot-quoted-lines branch October 5, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant