Repository navigation
Evaluate ACT with the Jev planner - #22
Merged
Merged
Conversation
A model name that starts with jev: now runs Whirl with --jev, with the rest of the name as the fallback model, and keeps its own runs. The leaderboard adds "by Jev": the share of actions Jev chose without the model. A step with no model call counts as free, since older reports leave its cost out. The local summary adds jev:gemini/gemini-3.1-flash-lite and jev:openrouter/gpt-5.6-luna, three runs of all 48 tasks each. Jev chose 56-57% of actions, halved the model calls, and cut the median ACT time from 1.0-1.3s to 0.6s, at the same pass rate. Every failed Jev run failed on an action the fallback model chose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jev now goes through lithos-llm, which prices its requests, so costs include Jev. The two jev: variants ran again, three runs of all 48 tasks each. Jev chose 56-58% of actions, halved the model calls, and cut the median ACT time to 0.6s. Every failed Jev run failed on an action the fallback model chose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #21. Step 4 of the ACT planner plan: evals for both planners.
What
eval_act.py: a model name that starts withjev:runs Whirl with--jev, with the rest of the name as the fallback model, for example-m jev:gemini/gemini-3.1-flash-lite. Its runs are kept apart, so one session compares both planners on one leaderboard.costUsdMicros: 0for such a step.)jev:prefix andTYPESAFE_API_KEY.Results (local set, 48 tasks,
-n 3)Jev rows use the
lithos-llmclient from #21 (e6a806e), so costs include Jev's requests ($0.042 per million input tokens).Speed: Jev halves the model calls and the median ACT time. The p90 barely moves, because the slow steps are the ones Jev leaves to the model.
Accuracy: unchanged within noise. Every failed Jev run failed on an action the fallback model chose: Flash-Lite gave
clickan argument 3 times and choseTYPEon the slider once. Across both Jev runs (576 runs in all), Jev chose no wrong action.Cost: a small saving with Flash-Lite and none with Luna. Jev costs about 20–70 µ$ per step. The steps it leaves to the model are the large pages, where the model cost is.
Where Jev never decided:
These are the next things to tune.
The three LLM-only rows are the earlier runs from Read back ACT fills, type quoted text as written, and add an 80-row eval #19. The planner seam (Put ACT planning behind a planner trait #20) did not change their code path.
Tests
scripts/test_eval_act.pycoversjev:names, slugs, the "by Jev" share, and zero-call cost.mise run checkpasses.🤖 Generated with Claude Code