Repository navigation
Tune the Jev planner, and choose from custom lists - #23
Merged
Merged
Conversation
Jev was unsure of the intent for instructions such as "delete invoice 1037", so it never picked an element. The intent request now follows Stagehand's (browserbase/stagehand#2953): - Stagehand's ten action families and their richer descriptions. - A split vote between click and select, double-click, or press counts as sure when the pair reaches 0.85. A select on a page without a native select is then a click on a custom control. - The same request asks for the key, the mouse button, the end state of a checkbox or switch, and whether typing ends with a suggestion. Right and middle clicks, suggestions, and a click that would undo a checkbox already in the asked state go to the model. The family descriptions carry Stagehand's MIT notice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jev gave up on a fill when the instruction quoted two strings or quoted none, as in "type Lovelace into the last name field". Following Stagehand (browserbase/stagehand#2953): - A lone placeholder is the text to type. - Otherwise the intent request adds a fill_value question: which quoted string or placeholder is the text to type, or none of them. - When none is, a small structured model call reads the text from the instruction alone, without the page. Stagehand's prompt carries over. Instruction::span then requires the text to be words of the instruction, and Whirl types the instruction's own characters. The planners now share one Arc<ModelClient>, since the Jev planner makes this call itself. The call counts as a model call in the report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jev picked only from the list of controls for the action, and gave up when a long list had no word match or when it rejected the shortened list. Following Stagehand's pickTarget (browserbase/stagehand#2952): - Tiers: the action's own view, then a broad view of every element with a name or text, which catches custom controls built from plain elements. A tier with the same candidates as an earlier one is skipped. - Within a tier, a list over 40 is first cut to its 30 best BM25 matches. When Jev rejects the cut list, the full list is asked, split into parts of at most 254 options and 60,000 characters. The parts nominate their top three in parallel, and the nominees are judged together. The broad tier asks one request at most. - When exactly one candidate has the name the instruction quotes, one small request confirms it before any ranking. - When Jev sees plausible targets but cannot choose, the search stops and hands its shortlist to the model. The Jev client is now shared through an Arc, so parallel requests run as tokio tasks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jev's pick was accepted only at 0.7 confidence with none-of-these at 0.9 or less. Following Stagehand's pickCandidate and pickTarget (browserbase/stagehand#2952): - A relaxed rule: at least 0.5 confidence, none-of-these at 0.1 or less, and a clear leader, at least 0.6 and 2.5 times the runner-up. Stagehand found split-but-right picks here, and a 0.56 against 0.38 split that was wrong, which the leader rule refuses. - A held pick: an accepted pick with none-of-these above 0.5 waits while the next tier tries. It is used at the end only while none-of-these stays at 0.7 or less. - Copies of one control in one table row, list item, or article, such as a card's repeated "Add to cart", share their vote when their descriptions match. Identical buttons in different rows do not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After Jev's fuller intent question, "pick March 14 as the check-in date" reached a Jev pick for the first time, and Jev clicked February 14 in every eval run. The two "14" buttons differed only by position: each table is unnamed, and its month is a caption child. An element inside a table, grid, or treegrid is now described with the table's name or caption and the header of its column, as Stagehand's descriptions carry a table cell's row and column (browserbase/stagehand#2952). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both jev: variants ran again, three runs of all 48 tasks each, after the four changes borrowed from Stagehand and the table description fix. Jev now chooses 75-76% of actions, up from 56-58%. The median ACT time is 0.5s, and model calls fall to 0.38 per task. Cost per task is the lowest of all rows: $0.0004 with Flash-Lite and $0.0003 with Luna. Jev + Luna passed every run. Every failed Jev + Flash-Lite run failed on an answer the fallback model gave. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jev was sure that "choose Portugal from the country list" is a select, but the planner handled select only for a native <select>, so every list or dropdown built from other elements went to the model. Now: - When the named option shows, Whirl clicks it. It looks for elements with an option role whose label the instruction says as whole words, then list items and clickable elements. Two different named labels are ambiguous. Jev confirms the match with one small request. - When no named option shows, Jev picks the control that opens the list, and the answer is the first of two steps. Step two, which always went to the model, now clicks the named option on the fresh snapshot, and goes to the model only when none shows. - Each pick asks its own question: the instruction's target, the control that opens the list, or the option to choose. Stagehand likewise asks each pick what it looks for. One description fix came with it: an element with its own text no longer takes its neighbour's text as a label. Jev had read the "Blue" list item as labelled "Red" and rejected it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both jev: variants ran again, three runs of all 48 tasks each. Both now pass every run. Jev chooses 83-84% of actions, up from 75-76%. The median ACT time is 0.5s, the p90 is 1.1s with Flash-Lite, and model calls fall to 0.28 per task. Jev + Luna costs $0.0002 per task, half of Luna alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brynary
added this pull request to stack #56
October 5, 2026 13:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #22. The Jev planner gets finer intent, candidate, and acceptance rules, and handles lists and dropdowns built without a native
<select>. One commit per change, with a re-eval after each batch.Commits
fill_valuequestion lets Jev choose among quoted strings and placeholders. For unquoted text, a small model call that sees only the instruction reads the text, andInstruction::spanrequires it to be the instruction's own words. The planners share oneArc<ModelClient>.Results (local set, 48 tasks,
-n 3)Tests
fill_value,span, splitting long lists, exact names, the acceptance rules, table places, option tiers, and neighbour labels.mise run checkpasses. SPEC §7.4 and CHANGELOG describe each change.🤖 Generated with Claude Code