Repository navigation
Name iframes in the ACT snapshot - #28
Merged
Merged
Conversation
Playwright's AI snapshot shows every iframe without a name, even one with a title. On the framed-feed eval page, the frame for "scroll down 50% inside the incident history" was only `- iframe [ref=e4]:`, so with --jev Jev was not sure it was the incident history, and the model planned the step in 5 of 6 runs. Right after the snapshot, the shim now resolves each iframe line's ref with aria-ref=, which works in nested frames and across origins. It adds the iframe's aria-label, or else its title, as the name: `- iframe "Incident history" [ref=e4]:`. Whole-page and scoped snapshots both get names. The name reaches the model's prompt, Jev's candidates, and the report lines, which already read names from snapshot lines. An iframe without a name, a ref that no longer resolves, a failed read, or a spent budget leaves the line as it was, so the names never fail a step. The shim does not wrap a line in YAML quotes as Playwright does for a name with ": ", because Rust's line parsers read the role from the start of the line. Tests cover the shim's rewrite rules, both Rust parsers on a named line, a Jev scroll inside a titled frame with no model call, a scoped snapshot with nested, labelled, cross-origin, and unnamed frames, and a cross-origin frame in Firefox and WebKit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both jev: variants ran the 11 local tasks that use frames three times each on 10b05d2. All 66 runs pass, as before. In scroll-in-frame, Jev now chooses 6 of 6 actions, up from 1 of 6: it sees the frame as "Incident history" and needs no model call. The step's median ACT time falls from 1.1s to 0.4s with Gemini and from 2.5s to 0.4s with GPT-5.6 Luna, and its cost from about $0.0003 to $0.0001 a run. The other 10 frame tasks do not change: every action is still Jev's, with the same model calls and Jev requests. Their frames' titles now reach Jev as the named sections around the button, even five frames deep and across sites. Their median ACT time moves from 472ms to 498ms and 483ms, within the spread between runs. The session cost about $0.006. The summary is kept apart from today's committed summaries, which the script would have overwritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Sep 27, 2026
Both jev: variants ran all 54 local tasks three times each on d9afb8a. Every run passes. Jev now chooses 84% and 85% of actions, up from 83% on #27, and model calls fall from 0.28 to 0.26 and 0.25 per task. Jev's share changed on three tasks: - scroll-in-frame: 1 of 6 to 6 of 6. The iframe's title names it. - saved-google-flights: 0 of 6 to 3 of 6. The page has two iframes. - plan-by-seats: 6 of 6 to 5 of 6. The page has no iframe, so this is run-to-run noise. This summary replaces the frame-only one; it includes those runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #27. Playwright's AI snapshot shows every iframe without a name, even one with a
title. So inscroll-in-frame("scroll down 50% inside the incident history") Jev could not tell which frame was meant, and it passed 5 of 6 runs to the model.Change
After the shim takes the
ACTsnapshot, whole-page or scoped, it names each unnamed iframe with itsaria-label, or else itstitle:- iframe [ref=e4]:becomes- iframe "Incident history" [ref=e4]:. The snapshot is still one string, so the model's prompt, Jev's candidates, and report lines all get the name, with no Rust or protocol change.aria-ref=, including nested and cross-origin frames (f1e3).:), because Rust's snapshot parsers read the role from the start of the line.Evals
Both
jev:variants, the 11 frame tasks, three runs each. Summary:evals/act/results/2026-09-27-local-jev-iframe-title.md.scroll-in-framescroll-in-frameThe other 10 frame tasks are unchanged: Jev still chooses every action, with the same requests and model calls.
Tests
ACT: Jev's request carries both frames' titles and Jev scrolls inside "Incident history" with no model call (SCROLL role:iframe "Incident history" to 50%); a scopedACTnames a nested frame, prefersaria-labeltotitle, names a cross-origin frame, and leaves an untitled frame as it was.mise run checkpasses. SPEC §7.4, the shim protocol doc, and the CHANGELOG describe the change.Full set
Both
jev:variants then ran all 54 local tasks three times each ond9afb8a. The summary isevals/act/results/2026-09-27-local-jev-iframe-title.md, which replaces the frame-only one. Every run passes. Jev now chooses 84% and 85% of actions, up from 83% on #27, and model calls fall from 0.28 to 0.26 and 0.25 per task. Jev's share changed on three tasks:🤖 Generated with Claude Code