Repository navigation
Plan drags and scrolls with Jev - #27
Merged
Merged
Conversation
With --jev, drags and scrolls went to the model. Now Jev plans them. For a drag, Jev picks the element to drag from every named element, then where to drop it: first from the parts of the page that take a drop, such as regions, lists, dialogs, and frames, then from every named element, never the dragged element or anything inside it. For a scroll, the first request asks two more questions: which way the scroll goes (down, up, left, right, to a position, or into view), and whether the whole page or a part of it scrolls. The page scrolls through the snapshot's root element. Jev picks the part, from the same candidates as a drop, or the element to bring into view. A position comes from the instruction's words: a percent, a fraction such as 0.75, halfway, the top, or the bottom; without one, the model plans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both jev: variants ran all 54 local tasks three times each on c9a7f55. Every run passes, and Jev still chooses 83% of actions with a median ACT time of 0.5s, so the two scroll questions in the first request cost nothing elsewhere. On the six pointer tasks Jev now chooses 31 of 36 actions, up from 6 (right clicks only): every drag and every page and panel scroll. In scroll-in-frame it chose 1 of 6: the iframe has no name in the snapshot, so Jev is unsure it is "the incident history", and the model plans it. The summary is kept apart from today's five-model summary of the new tasks, which the script would have overwritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Sep 27, 2026
brynary
added a commit
that referenced
this pull request
Sep 27, 2026
Both jev: variants ran all 54 local tasks three times each on d9afb8a. Every run passes. Jev now chooses 84% and 85% of actions, up from 83% on #27, and model calls fall from 0.28 to 0.26 and 0.25 per task. Jev's share changed on three tasks: - scroll-in-frame: 1 of 6 to 6 of 6. The iframe's title names it. - saved-google-flights: 0 of 6 to 3 of 6. The page has two iframes. - plan-by-seats: 6 of 6 to 5 of 6. The page has no iframe, so this is run-to-run noise. This summary replaces the frame-only one; it includes those runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brynary
added this pull request to stack #55
October 5, 2026 13:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #26. With
--jev, drags and scrolls went to the model; now Jev plans them.Change
50%,75 percent,0.75, halfway, the top, or the bottom. Without one, or with two that differ, the model plans.Any unsure answer leaves the step to the model, as before.
Evals
Both
jev:variants, all 54 local tasks, three runs each. Summary:evals/act/results/2026-09-27-local-jev.md.The two extra intent questions change nothing on the older tasks. On the six pointer tasks, Jev now chooses 31 of 36 actions, up from 6 (right clicks only):
RIGHTCLICK role:listitemDRAG role:article to role:region "Done"SCROLL to 50%SCROLL upSCROLL role:dialog "All filters" downSCROLL role:iframe to 50%In
scroll-in-framethe iframe has no name in the snapshot (itstitledoes not show), so Jev is unsure it is "the incident history" and the model plans the step. All runs pass.Tests
ACT: Jev drags with three requests and no model call; an unsure drag goes to the model; Jev scrolls the page to the position the instruction names with one request; an unsure scroll goes to the model.mise run checkpasses. SPEC §7.4 and the CHANGELOG describe the change.🤖 Generated with Claude Code