Skip to content

Plan drags and scrolls with Jev - #27

Merged
brynary merged 2 commits into
scrollfrom
jev-drag-scroll
Oct 5, 2026
Merged

brynary merged 2 commits into
scrollfrom
jev-drag-scroll

Conversation

@brynary

@brynary brynary commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Stacked on #26. With --jev, drags and scrolls went to the model; now Jev plans them.

Change

Action How Jev plans it
Drag Picks the element to drag from every named element, then where to drop it: first from the parts of the page that take a drop (regions, lists, dialogs, frames, and similar), then from every named element, never the dragged element or anything inside it.
Scroll The first request asks two more questions: which way (down, up, left, right, to a position, into view), and whether the whole page or a part of it scrolls. The page scrolls through the snapshot's root element; Jev picks the part, from the same candidates as a drop, or the element to bring into view.
Position Read from the instruction's words: 50%, 75 percent, 0.75, halfway, the top, or the bottom. Without one, or with two that differ, the model plans.

Any unsure answer leaves the step to the model, as before.

Evals

Both jev: variants, all 54 local tasks, three runs each. Summary: evals/act/results/2026-09-27-local-jev.md.

pass by Jev p50 ACT p90 ACT $/task
Jev + Flash-Lite, #23 (48 tasks) 1.00 0.84 0.47s 1.13s $0.0004
Jev + Flash-Lite, this PR 1.00 0.83 0.5s 1.2s $0.0005
Jev + Luna, #23 (48 tasks) 1.00 0.83 0.47s 1.79s $0.0002
Jev + Luna, this PR 1.00 0.83 0.5s 2.3s $0.0003

The two extra intent questions change nothing on the older tasks. On the six pointer tasks, Jev now chooses 31 of 36 actions, up from 6 (right clicks only):

task by Jev line
right-click-menu 6/6 RIGHTCLICK role:listitem
kanban-move 6/6 DRAG role:article to role:region "Done"
scroll-halfway 6/6 SCROLL to 50%
scroll-back-up 6/6 SCROLL up
scroll-panel-chunk 6/6 SCROLL role:dialog "All filters" down
scroll-in-frame 1/6 SCROLL role:iframe to 50%

In scroll-in-frame the iframe has no name in the snapshot (its title does not show), so Jev is unsure it is "the incident history" and the model plans the step. All runs pass.

Tests

  • ACT: Jev drags with three requests and no model call; an unsure drag goes to the model; Jev scrolls the page to the position the instruction names with one request; an unsure scroll goes to the model.
  • Unit: position reading, the scroll intent, the container view, the root, and containment.
  • mise run check passes. SPEC §7.4 and the CHANGELOG describe the change.

🤖 Generated with Claude Code

brynary and others added 2 commits September 27, 2026 08:13
With --jev, drags and scrolls went to the model. Now Jev plans them.

For a drag, Jev picks the element to drag from every named element, then
where to drop it: first from the parts of the page that take a drop,
such as regions, lists, dialogs, and frames, then from every named
element, never the dragged element or anything inside it.

For a scroll, the first request asks two more questions: which way the
scroll goes (down, up, left, right, to a position, or into view), and
whether the whole page or a part of it scrolls. The page scrolls through
the snapshot's root element. Jev picks the part, from the same
candidates as a drop, or the element to bring into view. A position
comes from the instruction's words: a percent, a fraction such as 0.75,
halfway, the top, or the bottom; without one, the model plans.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both jev: variants ran all 54 local tasks three times each on c9a7f55.
Every run passes, and Jev still chooses 83% of actions with a median
ACT time of 0.5s, so the two scroll questions in the first request cost
nothing elsewhere.

On the six pointer tasks Jev now chooses 31 of 36 actions, up from 6
(right clicks only): every drag and every page and panel scroll. In
scroll-in-frame it chose 1 of 6: the iframe has no name in the
snapshot, so Jev is unsure it is "the incident history", and the model
plans it.

The summary is kept apart from today's five-model summary of the new
tasks, which the script would have overwritten.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brynary added a commit that referenced this pull request Sep 27, 2026
Both jev: variants ran all 54 local tasks three times each on d9afb8a.
Every run passes. Jev now chooses 84% and 85% of actions, up from 83%
on #27, and model calls fall from 0.28 to 0.26 and 0.25 per task.

Jev's share changed on three tasks:
- scroll-in-frame: 1 of 6 to 6 of 6. The iframe's title names it.
- saved-google-flights: 0 of 6 to 3 of 6. The page has two iframes.
- plan-by-seats: 6 of 6 to 5 of 6. The page has no iframe, so this is
  run-to-run noise.

This summary replaces the frame-only one; it includes those runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary
brynary added this pull request to stack #55 October 5, 2026 13:05
@brynary
brynary merged commit 6064828 into pointer-buttons Oct 5, 2026
5 of 6 checks passed
@brynary
brynary deleted the jev-drag-scroll branch October 5, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant