[STG-2515] feat(evals): cursor_sdk full-harness adapter

[shrey150]

What

Part 5 of 5 — stacked on #2340 (openai_agents_sdk) (base: shrey/evals-harness-openai-agents-sdk; GitHub shows only this PR's delta). When the parent merges, retarget this PR's base and rebase.

Registers --harness cursor_sdk via @cursor/sdk (343K npm dl/wk): Agent.create → send → stream/wait against Cursor's managed local agent. Classification: FULL harness, on the smart tier next to claude_code/codex — "the same runtime, harness, and models that power Cursor" — NOT the bare tier, despite being a new SDK. It ships Cursor's complete loop, planning, and tool behaviors, which is exactly what makes it a useful smart-tier comparison point.

• Exposes exactly one custom tool (browse, same allowed-command gate as every other external harness) and prompts the agent to use only that tool. Known limitation (recorded in the design doc): the SDK exposes no tool allow-list, so browse-only discipline is prompt + custom-tool based rather than a canUseTool-style hard gate; the agent's native shell still runs inside the per-run temp cwd.
• cursorAdapter reverse-maps the SDKMessage stream (tool_call / assistant / thinking / usage) into a verifier Trajectory — only terminal (completed|error) tool_call events become steps (an interrupted run can't record an in-flight call as a successful step), duplicate terminal events collapse by call_id, buffered assistant/thinking text folds into the call's reasoning, per-turn usage sums.
• Cursor models only (cursor/composer-2.5 default; prefix stripped). No SDK-level turn cap exists — like claude_code, it inherits harness-native limits.
• Auth: CURSOR_API_KEY (the SDK's own env var). Adds the @cursor/sdk dependency (pinned 1.0.23) — this PR carries its own lockfile delta.
• Testability: sdk injection seam (Agent.create → send → stream/wait mock).

External harnesses design + usage: packages/evals/README.md#external-harnesses (in #2337). Linear: STG-2515.

Comparability fixes (2026-07-09 audit)

Two comparability bugs in the custom browse tool:

• execute() returned the bare output string and discarded ok from runBareBrowseCommand. Since that helper never throws, the SDK had no signal a browse command failed — every tool call read back as status: "completed", and cursorAdapter (which derives ok from that status) recorded every call as ok:true regardless of outcome. Now returns @cursor/sdk's own SDKCustomToolResult shape ({ content, isError }, per options.d.ts) with isError: !ok.
• The tool called runBareBrowseCommand directly, bypassing the 20k-char output clip every other harness gets via createBareLoopToolRecorder's readToolOutputLimit. Now clipped the same way, so tool-output sizes are comparable across all five harnesses.

Strengthened the test: the previous mock hardcoded tool_call status "completed" regardless of the tool result. Added a case that fails the underlying browse command and asserts isError:true out of the custom tool, then that it lands as ok:false once that isError flows through a terminal tool_call event into the recorded trajectory.

Open question for reviewers

@cursor/sdk's options.d.ts exposes an undocumented sandboxOptions: { enabled } toggle on LocalAgentOptions that this runner doesn't set. Unknown whether enabling it narrows the shell-gating gap noted above (the SDK has no tool allow-list to hard-disable its native shell/file tools, so browse-only discipline here is prompt + custom-tool based, not a hard gate). Flagging for input before deciding whether to wire it — no doc coverage found beyond the type declaration itself.

E2E Test Matrix

Command / flow	Observed output	Confidence / sufficiency
cursor_sdk live run	NOT RUN — no CURSOR_API_KEY available. Live validation pending an API key	Honest gap. Implemented + unit-tested against a mocked SDK whose shapes were taken from the real @cursor/sdk@1.0.23 type declarations (Agent.create/send, Run.stream/wait, SDKMessage, TokenUsage), not from docs alone
pnpm exec vitest run (this branch)	Test Files 53 passed, Tests 385 passed — adds cursorSdkRunner.test.ts (agent creation with gated custom tool wired to a real temp binary, prompt composition = arm prompt + browse-only instruction + task, stream collection, close() on both paths, metrics; plus cursorAdapter mapping tests: reasoning folding, duplicate call_id collapse, usage summing, error tool_calls)	Mocked-SDK coverage only until a key exists
tsc --noEmit + prettier + eslint	clean	Types/style only
pnpm exec vitest run (this branch, post-fix, stack tip)	Test Files 57 passed, Tests 419 passed — adds the isError/ok:false propagation test, the in-flight (running-only) tool_call exclusion test, and the duplicate-terminal-event metric dedup case	Full stack green with all comparability + review-round fixes applied; still mocked-SDK only pending a real CURSOR_API_KEY
pnpm -w exec tsc --noEmit + prettier + eslint (post-fix)	clean	Types/style only

No changeset: packages/evals is private.

🤖 Generated with Claude Code

---

Summary by cubic

Adds a full cursor_sdk harness powered by @cursor/sdk to run Cursor’s managed local agent and feed results into our verifier. Implements STG-2515 and enables smart-tier comparisons with correct tool-failure signaling, terminal-only tool_call handling, and consistent tool-output clipping.

• New Features

  • Registers --harness cursor_sdk via @cursor/sdk and adds it to the bench registry.
  • Adds a browse custom tool with the allowed-command gate; returns SDK-shaped { content, isError } and clips output to the standard limit.
  • Maps SDK events with cursorAdapter: only terminal tool_calls become steps, duplicate call_ids deduped, assistant/thinking folded into reasoning, usage summed; adds deduped cursor_tool_calls and token metrics.
  • Adds unit tests with a mocked SDK, covering failure propagation, in-flight tool_call exclusion, metric dedup, and usage aggregation.
  • Adds @cursor/sdk@1.0.23.

• Migration

  • Set CURSOR_API_KEY to run this harness.
  • Use Cursor models only; default is cursor/composer-2.5 (provider prefix is stripped).

^{Written for commit ab81880. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: ab81880

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 8 files

Confidence score: 5/5

• Safe to merge after the addressed issues were fixed.

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>