Repository navigation
Add a Jev planner for ACT behind --jev - #21
Merged
Merged
Conversation
With --jev, TypeSafe's Jev plans ACT first, and the model option plans when Jev is unsure. This follows Stagehand's experimental Jev act path (browserbase/stagehand#2951-#2953, #2960). Jev is a classifier: it answers closed questions quickly but cannot write text. One request asks which kind of action the instruction wants. A second picks the element from the snapshot elements that action can target, each described by its row or list item text, named sections, nearest heading, and place among look-alikes. It asks twice: "best" must choose, and "strict" may say none fits. Lists over 40 are cut to the 30 best BM25 word matches. Arguments come from the instruction: the one quoted string or placeholder that is not the field's name, the named key, or the one native select option the instruction names. Jev's choice goes through PageSnapshot::decide like a model answer. When Jev is not confident, has no argument, is asked for step two, or fails, the LLM planner plans the step, with Jev's likely matches in its prompt. A breaker pauses Jev after a rejected key or three failures in a row. TYPESAFE_API_KEY holds the key; WHIRL_JEV_ENDPOINT points at another server. Reports add the planner, each action's plannedBy, and Jev's requests and tokens. Tests use twin-openai's systemone route, pinned to the twins commit that logs request bodies (lithoscomputer/twins#14). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A step that Jev planned alone made no model call, so its usage had no cost and the report left costUsdMicros out, as if a call were unpriced. It now reports 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Jev planner called TypeSafe's /v1/systemone with its own reqwest client, but lithos-llm already serves Jev: the built-in catalog has typesafe/jev-latest on the systemone codec, and Client::evaluate checks each verdict, retries through the shared middleware, keeps the ACT deadline, and prices the verdict from the catalog. JevClient is now a thin wrapper over a lithos-llm client with the built-in catalog. WHIRL_JEV_ENDPOINT is an overlay that points the typesafe provider's base_url at another server. The breaker stays. Jev is now priced ($0.042 per million input tokens). The report's jev usage gains costUsdMicros, and a step's costUsdMicros is the model's cost plus Jev's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #20. Steps 2–3 of the ACT planner plan. Depends on lithoscomputer/twins#14: the twins pin points at that PR's commit until it merges.
What
--jevmakes ACT ask TypeSafe's Jev first, with themodeloption as the fallback. It follows Stagehand's experimental Jev act path (browserbase/stagehand#2951–#2953, #2960).run/act/jev/client.rs): Jev throughlithos-llm's evaluation API astypesafe/jev-latest:Client::evaluate_with_contextwith the ACT deadline, the shared retry middleware, verdict validation, and catalog pricing ($0.042 per million input tokens).WHIRL_JEV_ENDPOINTis a catalog overlay that points thetypesafeprovider at another server. A small breaker pauses Jev for 60 s after a rejected key, or for 30 s after three failures in a row. (The first version had its ownreqwestclient; commit e6a806e replaced it.)jev/outline.rs): the raw snapshot parsed as a tree. There are separate views for pointer actions, fills, native selects, and key presses. Each candidate is described by role, name, value, row or list-item text, named sections, nearest heading, and "n of m" among look-alikes. Views over 40 are cut to the 30 best BM25 matches.jev.rs): one request for the intent, one for the pick (bestwithout a "none" option, andstrictwith one). It accepts at ≥ 0.7 unlessstrict's "none" is > 0.9. Arguments come from the instruction (jev/args.rs). Every pick still goes throughPageSnapshot::decide.JevPlannerholds its fallback as anotherArc<dyn ActPlanner>.act.planner,actions[].plannedBy, andusage.jev(requests, tokens, and cost). A step'scostUsdMicrosis the model's cost plus Jev's. All three are additive, so the report stays version 2. The HTML report shows them too.machine-output.md;report.schema.json; CHANGELOG.Tests
Acceptance tests against the model twin's systemone route:
--jevwithoutTYPESAFE_API_KEYexits 3.Unit tests cover the outline, the views, descriptions, BM25, arguments, the breaker, and the wire shapes.
mise run checkpasses.A live smoke run against TypeSafe with the Flash-Lite fallback:
native-select,login-with-secret, andsimilar-buttonswere planned by Jev alone in about 0.45–0.5 s, with no model call.delete-invoice-rowfell back: the intent was unsure for "delete invoice 1037".The evals in the next PR measure all tasks.
🤖 Generated with Claude Code