feat(extension): experimental Jev WebMCP tool selection in act()

[miguelg719]

Stack

Stack: #2951 snapshot editable ids → #2952 library → #2953 act pipeline → #2954 observe + cache check → #2955 extract → this PR (6). Base: jev/5-extract.

Why

A page that registers WebMCP tools has already said what it can do. "Which of these tools fulfils the request" is a closed-set choice, which is what Jev answers in ~250 ms, and most tool arguments are words of the instruction (a span Jev can point at) or an enum/boolean. So act("put 2 of item p_102 in my basket") can call the page's add_to_cart instead of hunting for buttons.

Opt-in: experimentalJevAct.tools (default off). Sends tool names, descriptions and parameter names to TypeSafe.

What

• No round trip of its own. The page's tools are listed while act() waits for the DOM to settle, and the tool questions ride in the intent request every Jev act already makes (pipeline.ts). A page without tools adds no question; a page with tools adds no request.
• jevAct/tools.ts: builds and reads those questions. tool_best, tool_strict (with "none of these"), and a guard: does the instruction name a control? "click the Add to cart button" always takes the element path even when add_to_cart exists; the user asked for a click. A tool is used only at ≥ 0.8 with none ≤ 0.2.
• Arguments in the same request. Enum/boolean choices, and string/number parameters as a choice over spans of the instruction (quoted strings + 1–4 word runs). The argument questions of the two tools that share most words with the instruction ride along, so a confident call is usually one request; a different winner costs one more. Accepted only when every parameter, stated or judged "not stated", is ≥ 0.8, all required ones are filled, and no span fills two parameters (live: "search the store for mugs" put "mugs" into query and category).
• Argument-only LLM call (inference.toolArguments) for what is not a span (ISO dates, lists, nested objects): one tool in the prompt, not the catalog. The tool's own schema is the response format when every property is required; otherwise the input travels as a JSON string, because strict structured-output providers reject optional properties (found live with OpenAI). When the schema alone shows the lexical favourite will need this call, it starts alongside the Jev request.
• jevAct/toolAct.ts: invokes and waits. Once a tool was invoked the act is over, success or error; it never also clicks through the UI. Any doubt before that → the ordinary act continues from the intent it already has.
• The result is an ordinary ActResult whose action is { method: "webmcp", selector: "webmcp:<tool>", arguments: [<input JSON>] } with the tool output in message. Not cached (replay only knows element actions). Skipped for locator-scoped acts. %variables% are redacted towards TypeSafe, resolved only for the invocation, and returned as placeholders.
• Schema flag + regenerated artifacts; EVAL_JEV_TOOLS=1. No SDK example: the example-parity rule wants one per language and the switch is TS-env only, so the usage snippet lives in jevAct/README.md.

process.env.STAGEHAND_EXPERIMENTAL_JEV_ACT = JSON.stringify({ apiKey, tools: true });
const stagehand = await Stagehand.create({ browser, model });
await page.goto("https://browserbase.github.io/stagehand-eval-sites/sites/webmcp-test/");

await stagehand.act("add 19 and 23 together");
// → { method: "webmcp", selector: "webmcp:calculateSum", arguments: ['{"a":19,"b":23}'] }
//   'Invoked WebMCP tool calculateSum: {"a":19,"b":23,"sum":42}'  (one Jev request, 0 LLM tokens)

await stagehand.act("click the Calculate button");
// → names a control → ordinary element path

Results

Offline (not part of this PR): 166 tool descriptors harvested through page.tools() from six live sites (five from the webmcp.com directory, 9–57 tools each, plus our test site); 379 requests written by claude-sonnet-5 (direct + indirect phrasing per tool, plus 48 requests no tool can do, half near-misses). Tools were not invoked.

one merged Jev request (choice + guard + arguments of 2 favourites)
latency	p50 301 ms, p90 383 ms (~9K input tokens with the larger catalogs)
finished by Jev alone	189/379 (50%); 160 of them in a single request
of those, tool and arguments right	183/189 (96.8%); wrong tool accepted: 3
needs the argument LLM	106 (28%)
left to the element path (no tool / unsure / names a control)	84 (22%), which includes the 48 requests no tool can do

Baseline for scale: gemini-3.8-flash with the full catalog, thinking off: 364/378 (96.3%) tool + arguments right, p50 1.38 s.

Live, local Chrome, caller-side time for the whole act() (with #2993's settle overlap / without):

	path	with overlap	without
"what's in my cart?" (nekuda store, 9 tools)	jev-tool	398 ms	785 ms
"empty my cart"	jev-tool	244 ms	672 ms
"add 19 and 23 together" (test site)	jev-tool	n/a	850 ms (pipeline 333 ms)
"search the store for mugs"	jev-tool+arg-llm	n/a	2.9 s (gpt-5.4-mini argument call 2.1 s)
"click the Calculate button"	element path	unchanged	unchanged

Caveats: synthetic single-turn requests; thresholds were picked on this set; a request Jev is unsure about goes to the element path, not to an LLM tool chooser. The gain over an LLM is smaller than for element acts because a tool catalog is a small prompt; the bigger win is skipping UI work altogether.

Testing

jevTools.test.ts (stubbed Jev, through runJevActPipeline): no-input tool inside the intent request, span/enum/number filling in the same request, second request for a non-favourite winner, control guard, none / split choice, argument-LLM hand-off and skip without it, speculative argument call starts before Jev answers, one span in two parameters, missing required argument, variable redaction + resolution, tool error ≠ UI fallback, no WebMCP support adds no question; toolArguments schema-shaped and JSON-string routes. Full extension/protocol/sdk-ts/root suites, typecheck, lint, fmt, extensionpack --check pass locally. Live runs above.

[changeset-bot]

🦋 Changeset detected

Latest commit: 676e2fc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages

Name	Type
@browserbasehq/stagehand-extension	Patch
@browserbasehq/stagehand-go	Patch
@browserbasehq/stagehand	Patch
@browserbasehq/stagehand-python	Patch
browse	Patch
@browserbasehq/stagehand-integrations	Patch
@browserbasehq/stagehand-integrations-example-eve-facade	Patch
@browserbasehq/stagehand-integrations-example-pi-facade	Patch
@browserbasehq/stagehand-integrations-claude-agent-sdk	Patch
@browserbasehq/stagehand-integrations-example-claude-code-facade	Patch
@browserbasehq/stagehand-integrations-codex-sdk	Patch
@browserbasehq/stagehand-integrations-example-codex-facade	Patch
@browserbasehq/stagehand-integrations-cursor-sdk	Patch
@browserbasehq/stagehand-integrations-deepagents-sdk	Patch
@browserbasehq/stagehand-integrations-eve-sdk	Patch
@browserbasehq/stagehand-integrations-fx-sdk	Patch
@browserbasehq/stagehand-integrations-mastra-sdk	Patch
@browserbasehq/stagehand-integrations-example-mastra-facade	Patch
@browserbasehq/stagehand-integrations-pi-sdk	Patch
@browserbasehq/stagehand-integrations-example-vercel-ai-facade	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR