Repository navigation
Add EXTRACT, JUDGE, GOAL, and measured AI improvements - #60
Merged
Merged
Conversation
EXTRACT name [locator] "instruction" asks the model to read a typed value from the page, shaped by an optional JSON Schema in a documented subset. extract:NAME reads it in later checks and captures. Link fields come back as absolute URLs, and numbers keep their exact text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JUDGE [locator] "claim" is a check line that asks the model whether a claim about the page holds. The model sees the AI snapshot and a settled screenshot of the viewport or the element. yes passes, no fails with judge-false, and unsure passes with the judge-unsure warning. The shim moves to protocol 7 for judgeScreenshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GOAL "goal" asks the model to reach a goal with one action at a time on a fresh snapshot, until it answers done or impossible, 20 actions have run, or the time (2 minutes by default) runs out. A failed action goes back to the model. The AI cache records the path as a goal entry; a later run replays it, and a line that no longer fits heals from the current page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The eval server holds a response for delay=MS, so tasks can load their data late, as many real sites do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The wrapped value property was required and not nullable, so a model could not say that the page does not show the value. It answered "null", "", or 0. The evals found it: extract-missing-coupon failed on every model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before ACT, GOAL, ai: targets, EXTRACT, and JUDGE take the snapshot a model reads, the shim waits until no request has been open for 500 ms (streams and requests open for 2 s do not count), for at least 100 ms and at most 5 s or half of the step's time. Cached locator checks do not wait. Shim protocol 8 adds `settle` to ariaSnapshot. Local evals, 3 models x 84 tasks x 3 runs: pass rate 0.929-0.940 -> 0.960-0.964. The slow-loading tasks went from 0/9, 0/9, and 5/9 to 9/9 each; no other task changed beyond noise. A model step's p50 time grew by about 0.5 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a cached GOAL line misses because its element's role or name changed, ask the model to find that element from the cached role and name, run the line on it with its own method and arguments, and replay the rest of the path. The model then confirms the goal is done. Other misses plan the rest of the goal as before. Evals, goal-heal-renamed (a complete cached path with one renamed button), 3 models x 10 runs: 10/10 passes both ways; model calls 4 -> 2, input tokens about -63%, p50 time about -50%. goal-heal-checkout (an incomplete path) costs one more call and still passes 10/10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A GOAL answer now holds a list of actions: one, or one for each field of a form that needs a value. They run in order on the page the snapshot showed, and the first that fails stops the rest. Each counts toward the 20-action limit. Evals, 6 GOAL tasks x 3 models x 6 runs: Gemini 3.1 Flash-Lite model calls 4.5 -> 3.2 per task and p50 time 6.7 s -> 4.7 s at 1.00 pass; gpt-5.6-luna pass 0.75 -> 0.83 with 10% fewer calls; gpt-5.6-terra flat within noise (0.75 -> 0.72, one run of 36). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
goal-login-with-secret asked the model to sign in on a page whose button did nothing, so a careful model clicked until it ran out of actions. The page now says who signed in, and the task checks it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Consecutive JUDGE lines with the same scope and timeout share one snapshot, one screenshot, and one model call that answers each claim on its own. Each line still passes or fails on its own answer; the later lines report no model call. Evals, 3 models x 10 runs: four claims took 1 call instead of 4, input tokens 6.4k -> 1.7k, p50 time 6.4-8.9 s -> 2.0-3.0 s, with 10/10 passes both ways; a batch with a false claim still fails on it 10/10. One caution: for a claim whose evidence is out of view, batched Gemini Flash-Lite answered no where alone it answered unsure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Which ideas were kept or discarded, with the numbers of each comparison and the full-set summaries before and after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both go through OpenRouter passthrough, since the lithos-llm catalog does not list them. The five-model local summary is in results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It passed 0.84 of the local set: its answers often add arguments that click does not take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A click keeps only an argument that names a mouse button; any other, such as an empty string or the element's text, is a left click instead of act-invalid-decision. A method that takes no arguments, such as hover, ignores what the answer gave. Methods whose arguments carry meaning stay strict. Evals, full local set x 3 runs: default models unchanged beyond noise (luna 0.992 -> 1.000, terra 0.996 -> 0.992, both Geminis flat); GLM 5.3 Flash, which often adds such arguments, ACT pass 0.806 -> 0.883. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Where the schema wants a number and does not allow a string, a string that is a plain number (an optional minus sign, currency sign, and thousands commas, and an optional trailing %) reads as that number. Any other string still fails the schema check. Evals, 6 EXTRACT tasks x 5 models x 6 runs: Gemini 3.5 Flash-Lite 0.778 -> 0.944 (extract-slow-products 0/6 -> 6/6, where it answered "$12.00"). The change only touches answers that would fail the schema check, so it cannot turn a numeric answer into a failure; the other models' differences are noise on extract-cheapest-price, whose failures are wrong or missing prices both ways. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the fix for the replaced-target snapshot test up the stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the fix for the replaced-target snapshot test up the stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the fix for the replaced-target snapshot test up the stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the fix for the replaced-target snapshot test up the stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the fix for the replaced-target snapshot test up the stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The site failed the task's precheck on every run, so the task never reached a model step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add structured extraction, visual judgments, and multi-action goals. Include page-settle waits, batched judgments and form fills, cached-goal healing, and the measured click/number handling fixes.
Recovers the intended changes from #38, #39, #40, #41, #42, which were merged into the old development stack but did not reach
main. Preserves the newerlithos-llmpin from #54.Stage 4 of 10. Based on #59. Full recovery inventory and merge order: #57.
Merge the recovery PRs in order with merge commits. Once the preceding PR lands, retarget this PR to
main.Validation:
CARGO_NET_OFFLINE=true mise run checkpassed on this branch. This runs formatting, Clippy, local Rust/browser tests, shim checks, eval tooling, and workflow checks. Paid model evals, the external BrowserSim round trip, and live Kernel checks were not rerun.