Skip to content

Tune the Jev planner, and choose from custom lists - #23

Merged
brynary merged 8 commits into
act-jev-evalsfrom
act-jev-tuning
Oct 5, 2026
Merged

brynary merged 8 commits into
act-jev-evalsfrom
act-jev-tuning

Conversation

@brynary

@brynary brynary commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #22. The Jev planner gets finer intent, candidate, and acceptance rules, and handles lists and dropdowns built without a native <select>. One commit per change, with a re-eval after each batch.

Commits

Commit Change
c82eba1 Fuller intent question. Ten action families, each with a description. A split vote between click and select, double-click, or press is sure when the pair reaches 0.85. The same request asks for the key, the mouse button, the checkbox end state, and whether typing ends with a suggestion. Right and middle clicks, suggestions, and a click that would undo a checkbox already in the asked state go to the model.
5e863ac Fill text. A lone placeholder is the text to type. Otherwise a fill_value question lets Jev choose among quoted strings and placeholders. For unquoted text, a small model call that sees only the instruction reads the text, and Instruction::span requires it to be the instruction's own words. The planners share one Arc<ModelClient>.
b812e61 Candidate tiers. The action's own view, then every element with a name or text. A list over 40 is first cut to its 30 best BM25 matches. If Jev rejects the cut list, the full list is asked, split into parts of up to 254 that nominate their top three in parallel. An exact quoted name is confirmed with one small request.
7efca9f Acceptance rules. A clear-leader rule: ≥ 0.5 confidence, none-of-these ≤ 0.1, and a leader ≥ 0.6 and 2.5× the runner-up. Held picks: an accepted pick with none-of-these > 0.5 waits while the next tier tries. Copies of one control in one row or item share their vote.
acaecb8 Fix: describe a table element by its table's caption or name and its column header. Commit 1 let "pick March 14" reach a Jev pick for the first time, and Jev clicked February 14 in all 6 runs: the two "14" buttons differed only by position.
8f9e159 Custom lists and dropdowns. When the named option shows, click it: first elements with option roles, then list items and clickable elements, confirmed by Jev. Otherwise pick the control that opens the list as step one, and click the named option in step two, which no longer always goes to the model. Each pick asks its own question (target, opener, option). Also fixes a description bug: an element with its own text no longer takes its neighbour's text as a label.
7dbaa0b, 197f63c Eval results after commits 1–5 and after commit 6.

Results (local set, 48 tasks, -n 3)

pass mean ACT median p90 model calls by Jev $/task
Flash-Lite 0.99 1.14s 0.99s 1.81s 1.09 - $0.0009
Luna 0.99 1.71s 1.30s 3.23s 1.10 - $0.0004
Jev + Flash-Lite, #22 0.97 0.85s 0.59s 1.59s 0.49 0.58 $0.0007
Jev + Flash-Lite, commits 1–5 0.98 0.74s 0.51s 1.32s 0.38 0.76 $0.0004
Jev + Flash-Lite, this PR 1.00 0.66s 0.47s 1.13s 0.28 0.84 $0.0004
Jev + Luna, #22 1.00 1.31s 0.64s 2.94s 0.50 0.56 $0.0004
Jev + Luna, commits 1–5 1.00 1.00s 0.47s 2.38s 0.38 0.75 $0.0003
Jev + Luna, this PR 1.00 0.83s 0.47s 1.79s 0.28 0.83 $0.0002
  • Both Jev variants pass every run, and Jev chooses 83–84% of actions.
  • The mean ACT time falls 42% (Flash-Lite) and 51% (Luna) against the model alone.
  • Jev + Luna costs half as much per task as Luna alone.
  • The tasks where Jev still never decides are split intents: "cancel my membership", "show me my invoices", "reply to Grace Hopper's comment", "set the volume to 75". These fall back to the model by design; authors should name the action instead.

Tests

  • Acceptance: unquoted fill text through the small model call, which never sends the page; the broad tier; copies sharing a vote; a custom listbox; and a two-step custom dropdown with no model calls.
  • Unit: the intent merge, fill_value, span, splitting long lists, exact names, the acceptance rules, table places, option tiers, and neighbour labels.
  • mise run check passes. SPEC §7.4 and CHANGELOG describe each change.

🤖 Generated with Claude Code

brynary and others added 8 commits September 26, 2026 22:10
Jev was unsure of the intent for instructions such as "delete invoice
1037", so it never picked an element. The intent request now follows
Stagehand's (browserbase/stagehand#2953):

- Stagehand's ten action families and their richer descriptions.
- A split vote between click and select, double-click, or press counts
  as sure when the pair reaches 0.85. A select on a page without a
  native select is then a click on a custom control.
- The same request asks for the key, the mouse button, the end state of
  a checkbox or switch, and whether typing ends with a suggestion.
  Right and middle clicks, suggestions, and a click that would undo a
  checkbox already in the asked state go to the model.

The family descriptions carry Stagehand's MIT notice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jev gave up on a fill when the instruction quoted two strings or quoted
none, as in "type Lovelace into the last name field". Following
Stagehand (browserbase/stagehand#2953):

- A lone placeholder is the text to type.
- Otherwise the intent request adds a fill_value question: which quoted
  string or placeholder is the text to type, or none of them.
- When none is, a small structured model call reads the text from the
  instruction alone, without the page. Stagehand's prompt carries over.
  Instruction::span then requires the text to be words of the
  instruction, and Whirl types the instruction's own characters.

The planners now share one Arc<ModelClient>, since the Jev planner
makes this call itself. The call counts as a model call in the report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jev picked only from the list of controls for the action, and gave up
when a long list had no word match or when it rejected the shortened
list. Following Stagehand's pickTarget (browserbase/stagehand#2952):

- Tiers: the action's own view, then a broad view of every element with
  a name or text, which catches custom controls built from plain
  elements. A tier with the same candidates as an earlier one is
  skipped.
- Within a tier, a list over 40 is first cut to its 30 best BM25
  matches. When Jev rejects the cut list, the full list is asked, split
  into parts of at most 254 options and 60,000 characters. The parts
  nominate their top three in parallel, and the nominees are judged
  together. The broad tier asks one request at most.
- When exactly one candidate has the name the instruction quotes, one
  small request confirms it before any ranking.
- When Jev sees plausible targets but cannot choose, the search stops
  and hands its shortlist to the model.

The Jev client is now shared through an Arc, so parallel requests run
as tokio tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jev's pick was accepted only at 0.7 confidence with none-of-these at
0.9 or less. Following Stagehand's pickCandidate and pickTarget
(browserbase/stagehand#2952):

- A relaxed rule: at least 0.5 confidence, none-of-these at 0.1 or
  less, and a clear leader, at least 0.6 and 2.5 times the runner-up.
  Stagehand found split-but-right picks here, and a 0.56 against 0.38
  split that was wrong, which the leader rule refuses.
- A held pick: an accepted pick with none-of-these above 0.5 waits
  while the next tier tries. It is used at the end only while
  none-of-these stays at 0.7 or less.
- Copies of one control in one table row, list item, or article, such
  as a card's repeated "Add to cart", share their vote when their
  descriptions match. Identical buttons in different rows do not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After Jev's fuller intent question, "pick March 14 as the check-in
date" reached a Jev pick for the first time, and Jev clicked February
14 in every eval run. The two "14" buttons differed only by position:
each table is unnamed, and its month is a caption child.

An element inside a table, grid, or treegrid is now described with
the table's name or caption and the header of its column, as
Stagehand's descriptions carry a table cell's row and column
(browserbase/stagehand#2952).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both jev: variants ran again, three runs of all 48 tasks each, after
the four changes borrowed from Stagehand and the table description fix.
Jev now chooses 75-76% of actions, up from 56-58%. The median ACT time
is 0.5s, and model calls fall to 0.38 per task. Cost per task is the
lowest of all rows: $0.0004 with Flash-Lite and $0.0003 with Luna.
Jev + Luna passed every run. Every failed Jev + Flash-Lite run failed
on an answer the fallback model gave.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jev was sure that "choose Portugal from the country list" is a select,
but the planner handled select only for a native <select>, so every
list or dropdown built from other elements went to the model. Now:

- When the named option shows, Whirl clicks it. It looks for elements
  with an option role whose label the instruction says as whole words,
  then list items and clickable elements. Two different named labels
  are ambiguous. Jev confirms the match with one small request.
- When no named option shows, Jev picks the control that opens the list,
  and the answer is the first of two steps. Step two, which always went
  to the model, now clicks the named option on the fresh snapshot, and
  goes to the model only when none shows.
- Each pick asks its own question: the instruction's target, the
  control that opens the list, or the option to choose. Stagehand
  likewise asks each pick what it looks for.

One description fix came with it: an element with its own text no
longer takes its neighbour's text as a label. Jev had read the "Blue"
list item as labelled "Red" and rejected it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both jev: variants ran again, three runs of all 48 tasks each. Both now
pass every run. Jev chooses 83-84% of actions, up from 75-76%. The
median ACT time is 0.5s, the p90 is 1.1s with Flash-Lite, and model
calls fall to 0.28 per task. Jev + Luna costs $0.0002 per task, half
of Luna alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brynary brynary changed the title Borrow more of Stagehand's Jev path, and choose from custom lists Tune the Jev planner, and choose from custom lists Sep 27, 2026
@brynary
brynary added this pull request to stack #56 October 5, 2026 13:22
@brynary
brynary merged commit fc13a32 into main Oct 5, 2026
3 checks passed
@brynary
brynary deleted the act-jev-tuning branch October 5, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant