Repository navigation
Add ACT evals for five untested pointer methods - #32
Merged
Merged
Conversation
No eval task ran these ACT paths: scrollIntoView, scrollRight, a middle click, and dragAndDrop on boards with their own pointer-event drag code. Each new task has its own page: - scroll-into-view: "scroll to the pricing section" on a product page. Its Pricing section fetches its plans only when an IntersectionObserver sees it. The assert checks that the Team plan shows. - scroll-sideways: "scroll the photo strip to the right" on a rental listing with an overflow-x photo strip. The strip must scroll, and the page must not. - middle-click-tab: "middle-click the Reports tab to close it" in a support console. A middle click closes a tab through auxclick, and a left click only selects it. Only the active tab has a close button. - pointer-drag-distance: "move Priya Patel to the Onsite column" on a hiring board whose drag starts after an 8px move. - pointer-drag-delay: "move the tram ride to Sunday" in a trip planner whose drag starts after a 300ms press. A move of more than 5px before that cancels the drag. Both boards use pointer events and no native drag and drop. They drop on the column under the pointer, with the dragged card set to pointer-events: none, as the DRAG test page does. Each flow was run with its ACT line replaced by the line it should produce, such as MIDDLECLICK role:tab Reports. Those lines pass. Wrong actions fail the asserts: a left or right click on the tab, a scroll of the page, a click on a card, and a drop on the wrong column. With raw mouse input, the planner cancels a drag held for less than 300ms, and the hiring board starts no drag on a 6px move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Five models ran the five new tasks three times each. All 75 runs pass. The models and Jev chose these lines: - scroll-into-view: SCROLL role:heading "Pricing" 11 times, and the Pricing section's own region 4 times (gpt-5.6-terra 3, gpt-5.6-luna 1). The region has no name in the snapshot, because its name comes from its own heading. - scroll-sideways: SCROLL role:region "Photos" right every time. - middle-click-tab: MIDDLECLICK role:tab "Reports" every time. - pointer-drag-distance: DRAG from Priya Patel's card to role:region "Onsite". Jev dragged the name inside the card, which also works, because the pointerdown reaches the card. - pointer-drag-delay: DRAG from the Tram 28 ride item to the Sunday heading, list, or region. With --jev, Jev planned 24 of 30 actions without the model: every action in the first four tasks. For "move the tram ride to Sunday", Jev split its vote on the kind of action (drag 0.44, click 0.28, select 0.17), so the model planned all 6 runs of pointer-drag-delay. For "move Priya Patel to the Onsite column", Jev voted drag 1.0. The session cost $0.0525, and gpt-5.6-terra spent $0.0350 of it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #31. Five
ACTpaths from this stack had no eval task. This PR adds one for each, with a realistic page, and runs them on the five models the earlier sessions compared.Tasks
scroll-into-viewrole:heading Team visiblescroll-sidewaysmiddle-click-tabpointer-drag-distancepointer-drag-delayEach page was checked first without a model. The 11 right deterministic lines pass, such as
MIDDLECLICK role:tab ReportsandDRAGonto the column, its heading, a card, or its list. The 17 wrong ones fail at the assert: a left or right click on the tab, a page scroll instead of the strip,SCROLL to 100%instead of into view, a drop on the wrong column. The boards' own rules hold with raw Playwright mouse input: the planner cancels a 0 ms or 200 ms hold and accepts 400 ms, and the hiring board starts no drag on a 6 px move.Results
Every task and model passes 3/3: 75 of 75 runs, for about $0.05. Summary:
evals/act/results/2026-09-27-local-pointer-tasks.md.SCROLL role:heading "Pricing"11,SCROLL role:region(the section) 4SCROLL role:region "Photos" right15MIDDLECLICK role:tab "Reports"15role:region "Onsite"9; Jev drags the name inside the card 6, which also worksJev chose 24 of 30 actions overall. On
pointer-drag-delay, Jev split its vote on the kind of action for "move the tram ride to Sunday" (drag 0.44, click 0.28, select 0.17), so the model planned the step, as designed. The task keeps its wording rather than being tuned for Jev.Found, not fixed
Playwright's snapshot drops a region's name when
aria-labelledbypoints at its own heading. Jev's descriptions then give the region the nearest heading before it, so the Sunday region reached Jev described asheading: Saturday, May 10. Jev still dropped correctly here, by choosing the Sunday list. On a page where only the region takes a drop, it could choose the wrong column.🤖 Generated with Claude Code