feat(extension): rank large Jev candidate lists with BM25

[antonvishal]

What changed

Use a small, local BM25 scorer for the first 30 candidates when a Jev candidate view has more than 40 elements. Keep the current role views, field weights, zero-overlap behavior, and full-list retry. The scorer shares the existing Unicode tokenization and stop words. No dependency is added.

The previous overlap scorer remains available for comparison. scripts/compare-jev-rankers.ts provides reproducible generated outlines and an optional --jev request using TYPESAFE_API_KEY from .env.

Why

In a generated table of 81 Delete buttons, the instruction Delete the Acme account tied the rare Acme row with 60 earlier rows containing Account. The previous top-30 shortlist omitted the target. BM25 ranks the rare row first. In one live TypeSafe comparison of these two shortlists, Jev chose the wrong first row from the previous shortlist (0.73 best, 0.87 none) and the intended row from the BM25 shortlist (0.92 best, 0.17 none). The requests took 1391 ms and 1383 ms respectively.

Across four generated cases, the target was in the top 30 for 2/4 with the previous scorer and 3/4 with BM25. Both missed the semantic-mismatch case. Local scoring added about 0.04–0.05 ms on the 80-candidate cases. These are targeted regression cases, not a representative live-site benchmark or a general success-rate estimate.

Checks

• Extension suite: 49 files passed, 375 tests passed, 10 todo.
• Extension and tooling TypeScript checks passed.
• Lint passed for changed files.
• One optional live TypeSafe comparison completed; it did not run browser actions.

This targets #2952 only. The downstream stack can incorporate it through the existing parent-child merge flow.

---

Summary by cubic

Swaps the overlap scorer for a local BM25 scorer when a Jev candidate view has more than 40 elements and is pruned to a top-30 shortlist. BM25 keeps rare target rows that the overlap scorer dropped, like Delete the Acme account in an 81-row table, with no dependency added.

New Features

• The BM25 scorer reuses the existing field weights, Unicode tokenization, and stop words; zero-overlap pruning and full-list retry are unchanged.
• The old overlap scorer stays exported for scripts/compare-jev-rankers.ts, which reproduces generated outlines and can run a live TypeSafe comparison via TYPESAFE_API_KEY.
• Across four generated cases, the target is in the top 30 for 2/4 with the old scorer and 3/4 with BM25, though both miss the semantic-mismatch case.
• One live TypeSafe comparison chose the intended row from the BM25 shortlist and the wrong row from the old one.
• Targets feat(extension): decision-model client (TypeSafe Jev) and candidate-picking library #2952 only.

^{Written for commit 0ad3f86. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 0ad3f86

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions]

This PR is from an external contributor and must be approved by a stagehand team member with write access before CI can run.
Approving the latest commit mirrors it into an internal PR owned by the approver.
If new commits are pushed later, the internal PR stays open but is marked stale until someone approves the latest external commit and refreshes it.