Repository navigation
Add RIGHTCLICK and MIDDLECLICK - #24
Merged
Merged
Conversation
A flow can now click with the right or middle mouse button, to test a page's own context menu or auxclick handling. The verbs share CLICK's text default engine and actionability checks. CLICK carries its button to the shim, which passes it to Playwright and waits for the event that button fires: contextmenu for right, auxclick for middle. ACT runs them when the model answers click with right or middle. The prompt asks for that argument again, and an unknown button fails with act-invalid-decision. With --jev, Jev already named the mouse button, so it now plans these clicks instead of leaving them to the model. The Space press on a native checkbox stays for left clicks only. A middle click on a link differs by engine: Firefox opens a popup that POPUP can name, Chromium opens a tab without an opener, and WebKit follows the link. The SPEC says so, and a nightly test checks each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CHANGELOG, the SPEC, and code comments no longer name Stagehand. The eval tasks that run on saved copies of real sites are now saved-*, and their pages moved to local/site/saved/. Three things keep the name on purpose. prompt.rs and jev/intent.rs contain ported code, and the MIT license requires its copyright notice and a note of what it covers. The saved pages' README records where the copies came from and that they carry no license. The committed eval results keep the task names they ran under. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Whirl drives every browser through Playwright. ACT is modeled on Stagehand's act(), its prompts and parts of the Jev planner are ported from Stagehand under its MIT license, and some ACT evals run on pages saved from Stagehand's eval sites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DRAG source to target drags one element onto another. A bare `to` separates the two locators, which both use the text default engine and the strictness rules; a quoted "to" is text, and `whirl fmt` keeps its quotes. The shim runs the gesture itself, because Playwright's dragTo cannot hold: it hovers the source, presses, holds 500 ms, waits for the target to be visible and stable, moves in 10 steps, waits two animation frames, and releases, releasing even when a step fails. The hold serves drag code that starts after a press delay, and the steps serve code that starts after a short move. The target check scrolls it into view instead of a trial hover, which stops a native drag in Chromium. ACT runs dragAndDrop, whose one argument is the ref of the element to drop on; an unknown ref or the dragged element itself fails with act-invalid-decision. With --jev, drags go to the model. Tests cover a native board, a board that starts after an 8px move, one that starts after a 300ms press, drags into and across frames in all three engines, an ambiguous target, and the ACT paths. In WebKit a page that took a native drag gets no pointerdown until it loads again; the SPEC says so, and the board test runs its pointer drags first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
Actions already scroll their elements into view, so SCROLL is for what scrolling itself does: content that loads on scroll, controls that react to the position, and boxes that scroll on their own. - SCROLL locator brings the element into view. - SCROLL [locator] down, up, left, or right scrolls one visible height or width. - SCROLL [locator] to N% scrolls to a vertical position, with a new percent literal from 0% to 100%. The box that scrolls is the element when it can scroll in that direction, else its nearest ancestor that can, else its document; html and body stand for the page, and no locator means the page. The shim sets the position at once, even on a page with smooth scrolling, and ends when it holds for two frames; a box already there stays put. A trailing bare direction word and a bare `to` are keywords, and `whirl fmt` keeps the quotes on text that spells them. ACT runs scrollIntoView, scrollTo, nextChunk, prevChunk, scrollLeft, and scrollRight. In a whole-page snapshot the first element is <body>, so a scroll of it runs as the page form, such as SCROLL down. scrollTo needs a percent; anything else fails with act-invalid-decision. With --jev, scrolls go to the model. Tests cover each form on a long page, a lazy-load trigger, a panel in a dialog, a terms box, a sideways board, and a scrollable iframe in all three engines, plus the ACT paths. Four eval tasks scroll halfway, by a chunk in a panel, back up, and inside a frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first eval run of the scroll tasks showed models naming the container they mean, not the box that scrolls. Four of five models answered "scroll down one chunk in the filters panel" with the dialog, which cannot scroll, so the page behind it scrolled. Failing runs of "scroll inside the incident history" named the iframe element, which cannot scroll either, so nothing moved. The scroll box is now the element when it can scroll, else the largest box inside it that can, else its nearest scrollable ancestor, else its document. An iframe element scrolls the page inside it: the shim enters the frame with contentFrame(), which works across origins. The browser test now scrolls a dialog's list through the dialog, and a same-origin and a cross-origin frame through their iframe elements, in all three engines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Five models ran the six new tasks three times each: right-click-menu, kanban-move, and four scroll tasks. All 90 runs pass. Models still name the filters dialog and the iframe element when asked to scroll inside them, and the scroll box rule now reaches the list and the frame's page. Before that rule, the same session passed 0.67 to 0.94: the dialog scrolled the page behind it, and the iframe element scrolled nothing. With --jev, Jev plans every right click without the model; drags and scrolls go to the model by design. The two sessions cost about $0.09. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
With --jev, drags and scrolls went to the model. Now Jev plans them. For a drag, Jev picks the element to drag from every named element, then where to drop it: first from the parts of the page that take a drop, such as regions, lists, dialogs, and frames, then from every named element, never the dragged element or anything inside it. For a scroll, the first request asks two more questions: which way the scroll goes (down, up, left, right, to a position, or into view), and whether the whole page or a part of it scrolls. The page scrolls through the snapshot's root element. Jev picks the part, from the same candidates as a drop, or the element to bring into view. A position comes from the instruction's words: a percent, a fraction such as 0.75, halfway, the top, or the bottom; without one, the model plans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both jev: variants ran all 54 local tasks three times each on c9a7f55. Every run passes, and Jev still chooses 83% of actions with a median ACT time of 0.5s, so the two scroll questions in the first request cost nothing elsewhere. On the six pointer tasks Jev now chooses 31 of 36 actions, up from 6 (right clicks only): every drag and every page and panel scroll. In scroll-in-frame it chose 1 of 6: the iframe has no name in the snapshot, so Jev is unsure it is "the incident history", and the model plans it. The summary is kept apart from today's five-model summary of the new tasks, which the script would have overwritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Playwright's AI snapshot shows every iframe without a name, even one with a title. On the framed-feed eval page, the frame for "scroll down 50% inside the incident history" was only `- iframe [ref=e4]:`, so with --jev Jev was not sure it was the incident history, and the model planned the step in 5 of 6 runs. Right after the snapshot, the shim now resolves each iframe line's ref with aria-ref=, which works in nested frames and across origins. It adds the iframe's aria-label, or else its title, as the name: `- iframe "Incident history" [ref=e4]:`. Whole-page and scoped snapshots both get names. The name reaches the model's prompt, Jev's candidates, and the report lines, which already read names from snapshot lines. An iframe without a name, a ref that no longer resolves, a failed read, or a spent budget leaves the line as it was, so the names never fail a step. The shim does not wrap a line in YAML quotes as Playwright does for a name with ": ", because Rust's line parsers read the role from the start of the line. Tests cover the shim's rewrite rules, both Rust parsers on a named line, a Jev scroll inside a titled frame with no model call, a scoped snapshot with nested, labelled, cross-origin, and unnamed frames, and a cross-origin frame in Firefox and WebKit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both jev: variants ran the 11 local tasks that use frames three times each on 10b05d2. All 66 runs pass, as before. In scroll-in-frame, Jev now chooses 6 of 6 actions, up from 1 of 6: it sees the frame as "Incident history" and needs no model call. The step's median ACT time falls from 1.1s to 0.4s with Gemini and from 2.5s to 0.4s with GPT-5.6 Luna, and its cost from about $0.0003 to $0.0001 a run. The other 10 frame tasks do not change: every action is still Jev's, with the same model calls and Jev requests. Their frames' titles now reach Jev as the named sections around the button, even five frames deep and across sites. Their median ACT time moves from 472ms to 498ms and 483ms, within the spread between runs. The session cost about $0.006. The summary is kept apart from today's committed summaries, which the script would have overwritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With --video on Chromium, a short flow on a still page could fail as a runtime error: "video recording failed: the page produced no frames". On macOS CI this failed one run of combined_report_keeps_embedded_media_and_missing_scenarios_offline, and the rerun passed. Chrome sends a screencast frame only when the page paints, and the frame reaches the shim some time after the paint: 5 to 450 ms on a loaded machine. The recorder starts the screencast on the blank page, and a flow such as VISIT then SCREENSHOT can reach endFlow first. The recorder then stopped the screencast, dropped the frame on its way, and failed the file. With logging added, each such frame arrived 10 to 33 ms after the stop. Now, when no frame has arrived, the recorder captures the page once with Page.captureScreenshot and holds that frame from the start of the recording at the constant rate. When the capture fails or takes more than 5 seconds, as for a crashed page, the shim skips the recording and says why in the new endFlow field videoSkipped. Rust reports it as a warning, and the file's result does not change, as for SCREENSHOT. The recorder now takes a frame source, so its unit tests can use a page that sends no frame. Before the fix, the CI test failed 3 of 50 runs with 10 copies at once on low-priority cores, and a new test of four short flows failed 20 of 20 runs. After the fix, they pass 100 of 100 and 20 of 20. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A skipped recording kept runtime.videoFps in the JSON report, but the machine-output doc says the field is absent without a recording. Rust now clears it when the shim skips the recording. The fake shim starts a 60 fps recording before it reports the skip, so the test sees the field go away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Many upload widgets are drop zones with no file input, so UPLOAD, which sets an input's files, cannot test them. DROP locator file:path drops one file on the element. It parses like UPLOAD, and the path resolves beside the .whirl file. An unprefixed locator uses the text default engine, because a drop zone says what it takes, such as "Drop files here". The shim calls Playwright's locator.drop, which waits for the element as other pointer actions do and fires dragenter, dragover, and drop with the file. Playwright reads the file and gives it its name and a type from its extension. When the zone's dragover does not call preventDefault, the page rejects the drop, and the step fails at once with an action error that names the target. A missing file fails before the shim looks for the element, with the path that Whirl tried. ACT has no method for DROP, as it has none for UPLOAD, because it cannot choose a file. The SPEC now says so. Tests drop files on a zone, into a same-origin frame, and into a cross-origin frame, and cover a rejected drop, a missing file, and an ambiguous zone, in all three engines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A drop zone that appears only while a drag is over the page cannot be the target, because Playwright waits for the element to be visible before it fires any drag event. The events are synthetic, so a page that checks isTrusted rejects them. Authors hit both, so the SPEC says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With --video on Chromium, an ffmpeg failure at endFlow failed endFlow, so the file became a runtime error (exit 3). But a recording is evidence, not a result (SPEC 14.1). Now the shim discards the recording and says why in videoSkipped, as it does for a page that it cannot capture. Rust reports a warning, and the file's result does not change. A missing ffmpeg at startFlow is still a runtime error (SPEC 13). The recorder now waits for ffmpeg's "close" event, not "exit". Node can send "exit" before the last stderr output, and the warning then loses ffmpeg's error text. A new unit test runs a fake ffmpeg that writes part of its output, prints an error, and exits with status 1. The recorder skips the recording with that error and leaves no file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On GitHub Actions ubuntu-24.04, a lone VISIT /stable.html in short_video_flows_on_still_pages_pass_with_a_recording passed with "warning: video recording skipped: the page produced no frames, and capturing it failed: ... Unable to capture screenshot". The test expects a recording, so it failed. Chrome gives that error when the page has not painted yet, as right after a navigation or on a new tab: it has nothing to copy. With 10 copies of the test at once on low-priority cores, the old recorder failed 7 of 250 runs this way: 6 times right after a VISIT, and once on the blank page of an HTTP-only flow. Each failed capture took 2 to 11 ms. With logging added, a new capture 25 ms later, or the one after it, worked every time, within 360 ms. Now, when a capture finds nothing painted, the recorder waits 50 ms and tries again, for up to 1 second. The screencast still feeds the recorder while it tries, so a frame that arrives is used as a capture would be. When the recorder still has no frame, or the capture fails for another reason, as for a crashed page, it holds a white frame of the viewport's size for the whole recording, as Playwright's recorder does. All the captures together still stop after 5 seconds. Only an ffmpeg failure skips a recording now. The white frame is a JPEG from jpegjs in playwright-core's utilsBundle, the encoder that Playwright's recorder uses for its own white frame. The package's export map includes that bundle, and the shim already takes the image comparator from its core bundle. Node has no JPEG encoder, and Playwright's ffmpeg decodes only MJPEG and VP8, so a PNG frame cannot go into the stream. The frame source now gives the retry time, so the unit tests use 200 ms, not 1 second. New tests cover a capture that works on the second try, a page that never paints (its decoded frames are white), a crashed page (a white frame with no retry), and a screencast frame that arrives between tries. The fake shim test now reports an ffmpeg failure, the only reason for a skip. After the change, under the same load, the short test passed 250 of 250 runs. For 100 of them, rounds of the old and the new recorder alternated, so both had the same load; the old recorder failed 4 of its 100. In the 1,000 new recordings, 7 captures found nothing painted, and each of those recordings got a frame on the second or third try, within 650 ms. No recording needed the white frame, and none was skipped. The combined-report test passed 100 of 100 runs, as it passed 50 of 50 before. Its 18 recordings without a frame at the stop needed no retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the finished no-frames fix from #29: retries for a page that has not painted, a white frame when there is still none, and a skip only when ffmpeg fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With --video on Chromium, a page that the recorder cannot capture, as a crashed one, gets a recording that holds a white frame. The report then showed a white video with no explanation. Now the shim says why in the new endFlow field videoBlank, and Rust reports "warning: video recording is blank: <reason>", masked like other warnings. The recording stays in the report with its frame rate, and the file's result does not change. A page that has not painted after 1 second of tries also gets the warning, with the reason "the page did not paint within 1000ms". Its video is white too, so it needs the same explanation. A capture that takes more than 5 seconds gives "capturing the page took more than 5000ms". For a crashed page, Chrome answers "cdpSession.send: Protocol error (Page.captureScreenshot): Internal error", and the warning shows that text after "capturing the page failed:". The recorder's unit tests now check the reason for a page that never paints and for a crashed page. A real frame or a working capture still gives no reason. The fake shim has a new FAKE_SHIM_VIDEO_BLANK option, and a new acceptance test checks the masked warning in stdout and in the JSON report, the passed status, the listed recording, and the frame rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the blank-recording warning from #29. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both jev: variants ran all 54 local tasks three times each on d9afb8a. Every run passes. Jev now chooses 84% and 85% of actions, up from 83% on #27, and model calls fall from 0.28 to 0.26 and 0.25 per task. Jev's share changed on three tasks: - scroll-in-frame: 1 of 6 to 6 of 6. The iframe's title names it. - saved-google-flights: 0 of 6 to 3 of 6. The page has two iframes. - plan-by-seats: 6 of 6 to 5 of 6. The page has no iframe, so this is run-to-run noise. This summary replaces the frame-only one; it includes those runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With --video on Chromium, a short flow on a still page could fail as a runtime error: "video recording failed: the page produced no frames". On macOS CI this failed one run of combined_report_keeps_embedded_media_and_missing_scenarios_offline, and the rerun passed. Chrome sends a screencast frame only when the page paints, and the frame reaches the shim some time after the paint: 5 to 450 ms on a loaded machine. The recorder starts the screencast on the blank page, and a flow such as VISIT then SCREENSHOT can reach endFlow first. The recorder then stopped the screencast, dropped the frame on its way, and failed the file. With logging added, each such frame arrived 10 to 33 ms after the stop. Now, when no frame has arrived, the recorder captures the page once with Page.captureScreenshot and holds that frame from the start of the recording at the constant rate. When the capture fails or takes more than 5 seconds, as for a crashed page, the shim skips the recording and says why in the new endFlow field videoSkipped. Rust reports it as a warning, and the file's result does not change, as for SCREENSHOT. The recorder now takes a frame source, so its unit tests can use a page that sends no frame. Before the fix, the CI test failed 3 of 50 runs with 10 copies at once on low-priority cores, and a new test of four short flows failed 20 of 20 runs. After the fix, they pass 100 of 100 and 20 of 20. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A skipped recording kept runtime.videoFps in the JSON report, but the machine-output doc says the field is absent without a recording. Rust now clears it when the shim skips the recording. The fake shim starts a 60 fps recording before it reports the skip, so the test sees the field go away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With --video on Chromium, an ffmpeg failure at endFlow failed endFlow, so the file became a runtime error (exit 3). But a recording is evidence, not a result (SPEC 14.1). Now the shim discards the recording and says why in videoSkipped, as it does for a page that it cannot capture. Rust reports a warning, and the file's result does not change. A missing ffmpeg at startFlow is still a runtime error (SPEC 13). The recorder now waits for ffmpeg's "close" event, not "exit". Node can send "exit" before the last stderr output, and the warning then loses ffmpeg's error text. A new unit test runs a fake ffmpeg that writes part of its output, prints an error, and exits with status 1. The recorder skips the recording with that error and leaves no file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On GitHub Actions ubuntu-24.04, a lone VISIT /stable.html in short_video_flows_on_still_pages_pass_with_a_recording passed with "warning: video recording skipped: the page produced no frames, and capturing it failed: ... Unable to capture screenshot". The test expects a recording, so it failed. Chrome gives that error when the page has not painted yet, as right after a navigation or on a new tab: it has nothing to copy. With 10 copies of the test at once on low-priority cores, the old recorder failed 7 of 250 runs this way: 6 times right after a VISIT, and once on the blank page of an HTTP-only flow. Each failed capture took 2 to 11 ms. With logging added, a new capture 25 ms later, or the one after it, worked every time, within 360 ms. Now, when a capture finds nothing painted, the recorder waits 50 ms and tries again, for up to 1 second. The screencast still feeds the recorder while it tries, so a frame that arrives is used as a capture would be. When the recorder still has no frame, or the capture fails for another reason, as for a crashed page, it holds a white frame of the viewport's size for the whole recording, as Playwright's recorder does. All the captures together still stop after 5 seconds. Only an ffmpeg failure skips a recording now. The white frame is a JPEG from jpegjs in playwright-core's utilsBundle, the encoder that Playwright's recorder uses for its own white frame. The package's export map includes that bundle, and the shim already takes the image comparator from its core bundle. Node has no JPEG encoder, and Playwright's ffmpeg decodes only MJPEG and VP8, so a PNG frame cannot go into the stream. The frame source now gives the retry time, so the unit tests use 200 ms, not 1 second. New tests cover a capture that works on the second try, a page that never paints (its decoded frames are white), a crashed page (a white frame with no retry), and a screencast frame that arrives between tries. The fake shim test now reports an ffmpeg failure, the only reason for a skip. After the change, under the same load, the short test passed 250 of 250 runs. For 100 of them, rounds of the old and the new recorder alternated, so both had the same load; the old recorder failed 4 of its 100. In the 1,000 new recordings, 7 captures found nothing painted, and each of those recordings got a frame on the second or third try, within 650 ms. No recording needed the white frame, and none was skipped. The combined-report test passed 100 of 100 runs, as it passed 50 of 50 before. Its 18 recordings without a frame at the stop needed no retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With --video on Chromium, a page that the recorder cannot capture, as a crashed one, gets a recording that holds a white frame. The report then showed a white video with no explanation. Now the shim says why in the new endFlow field videoBlank, and Rust reports "warning: video recording is blank: <reason>", masked like other warnings. The recording stays in the report with its frame rate, and the file's result does not change. A page that has not painted after 1 second of tries also gets the warning, with the reason "the page did not paint within 1000ms". Its video is white too, so it needs the same explanation. A capture that takes more than 5 seconds gives "capturing the page took more than 5000ms". For a crashed page, Chrome answers "cdpSession.send: Protocol error (Page.captureScreenshot): Internal error", and the warning shows that text after "capturing the page failed:". The recorder's unit tests now check the reason for a page that never paints and for a crashed page. A real frame or a working capture still gives no reason. The fake shim has a new FAKE_SHIM_VIDEO_BLANK option, and a new acceptance test checks the masked warning in stdout and in the JSON report, the passed status, the listed recording, and the frame rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brynary
added this pull request to stack #56
October 5, 2026 13:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #23. The first of three PRs that add pointer actions to flows and to
ACT. The next two add drag and drop, then scrolling.Commits
auxclickhandling. They find elements by text by default and useCLICK's actionability checks. The shim'sclickcommand takes abutton. Its click receipt waits for the event that button fires:click,contextmenu, orauxclick.clicktakes an optionalleft,right, ormiddleargument, and the prompt asks for it again. Any other value fails withact-invalid-decision. With--jev, Jev already names the mouse button, so it now plans these clicks itself instead of passing them to the model. The Space press on a native checkbox stays for left clicks only.saved-*, inlocal/site/saved/. The MIT license headers of the ported code, the saved pages' source note, and past eval results keep the name.act()ACTis modeled on and whose prompts, Jev planner parts, and eval pages Whirl uses.A middle click on a link differs by engine
POPUPcan namePOPUPcannot nameSPEC §7 says so and tells authors to use
CLICKfor links that open a new tab. A nightly test checks each engine.Tests
ACT: a model answer withright, an unknown button, and a Jev right click with no model call.fmtround trip, the button argument and its errors, and Jev's button read.right-click-menutask that says "open the context menu" without naming the button.mise run checkpasses. SPEC §6.1, §7, §7.4, §17, and §18, the shim protocol doc, and the CHANGELOG describe the change.🤖 Generated with Claude Code