[STG-2515] feat(evals): external-harness scaffolding — skill modes, bare-loop core, shared browse adapter

[shrey150]

What

Part 1 of 5 (base of the stack). Scaffolding + design doc for four upcoming external-harness adapters — no runner is registered here; each lands in its own stacked PR so the runners can be reviewed in isolation:

1. [STG-2515] feat(evals): external-harness scaffolding — skill modes, bare-loop core, shared browse adapter #2337 (this PR) — scaffolding + design doc
2. [STG-2515] feat(evals): vercel_ai_sdk bare-loop harness adapter #2338 — vercel_ai_sdk runner (stacked on this)
3. [STG-2515] feat(evals): anthropic_sdk bare-loop harness adapter #2339 — anthropic_sdk runner
4. [STG-2515] feat(evals): openai_agents_sdk harness adapter #2340 — openai_agents_sdk runner
5. [STG-2515] feat(evals): cursor_sdk full-harness adapter #2341 — cursor_sdk runner

External harnesses design + usage: packages/evals/README.md#external-harnesses

Linear: STG-2515

What's in this PR:

• Harness union grows the four names + classification helpers (BARE_LOOP_HARNESSES, EXTERNAL_HARNESSES). The new names are parseable/plannable (dry-run works) but not executable — EXECUTABLE_BENCH_HARNESSES stays at stagehand/claude_code/codex, and each runner PR flips its own bit. Every mid-stack state is coherent.
• --skill-mode <none|prompt_show|injected> — per-run skill-delivery arm, orthogonal to harness (A/B/C experiment arms). prompt_show is gated on the browse release carrying [STG-2511] feat: add browse skills show and agent-facing --help header #2335 (unreleased); the adapter logs a loud warning with the installed CLI version until then. claude_code/codex provisioning is untouched.
• externalHarnessToolAdapter.ts — shared browse provisioning via the same prepareBrowseCliHarnessAdapter contract codex delegates to, plus isAllowedBrowseCommand gating (single browse ... per tool call, no shell metacharacters) and the bare one-liner system prompt mirroring the sandbox templates verbatim (the bareness is the instrument).
• bareLoopRunner.ts core — execution-time NormalizedToolCall recording, 20k-char tool-output clip (EVAL_BARE_LOOP_TOOL_OUTPUT_LIMIT), shared EVAL_RESULT parsing, and gradeExternalTrajectory (feat(evals): add verifier benchmark instrumentation #2138) wiring unchanged from claude_code/codex: rubric hydration → gemini-2.5-flash judge → persisted trajectory → outcome/process folded into TaskResult.
• buildAdapterBackedHarness factory in benchHarness.ts (registrations come with each runner PR).
• 40-step bare-loop default cap (the 2026-07-09 3-provider smoke showed 20 is cap-binding) + per-harness env overrides; per-harness default-model env vars.

Why

Measure browse CLI + SKILL.md performance across the harness-richness spectrum, so skill-content improvements can be validated regardless of model intelligence:

• Duckbill (STG-2448): browse 1/5 vs managed Agents 3/5 — gap was scaffolding, not model.
• A 3-provider bare-loop smoke (2026-07-09) found 5 systematic SKILL.md/template issues in 3 runs that months of Claude Code usage never surfaced — smart harnesses compensate for doc flaws; bare loops execute skill text literally.
• Adoption reality: raw provider SDKs dominate bare-loop reality (openai 84M + anthropic 37.5M PyPI dl/wk), Vercel AI SDK (15.1M npm/wk) is the JS default and genuinely bare, OpenAI Agents SDK (6.8M PyPI/wk) is bare-ish, Cursor SDK (343K npm/wk) is a full harness — it sits on the smart tier next to claude_code/codex.

TS-only note: real-world anthropic/openai bare loops are mostly Python; the loop shape is language-invariant and packages/evals is TS — deliberate simplification, recorded in the doc.

Comparability fixes (2026-07-09 audit)

An independent comparability audit across all five harnesses found several cross-cutting issues. The two landing in this PR's layer (shared bareLoopRunner.ts / evalResultParser.ts):

• camelCase TaskResult field names. finalizeBareLoopResult built [`${prefix}Status`] / [`${prefix}StopReason`] directly off the harness id, so mixed-case ids (vercel_ai_sdk, openai_agents_sdk) produced awkward field names like vercel_ai_sdkStatus. Now camelCased: vercelAiSdkStatus, openaiAgentsSdkStatus, anthropicSdkStatus, etc. Metrics keys (${prefix}_steps, ${prefix}_tool_calls, ...) are unchanged — they intentionally keep the snake_case harness id, matching the metrics-naming convention shared with claude_code/codex.
• EVAL_RESULT parser follow-up noted, not folded in. claudeCodeRunner.parseClaudeCodeResult and codexRunner.parseCodexResult are two more hand-forked copies of this module's parseEvalResultText (predating it) that have already drifted slightly from each other and from this one (codexRunner has an extra first-line fallback on the no-marker path that claudeCodeRunner doesn't). Consolidating all three isn't a same-behavior mechanical change, so it's left as a TODO in evalResultParser.ts for a dedicated follow-up PR instead of risked here.

The other three findings (cursor_sdk isError/output-clip propagation, vercel_ai_sdk/openai_agents_sdk step-cap detection, anthropic_sdk native abort wiring) land in their owning stacked PRs — #2338, #2339, #2340, #2341.

Review-round fixes (Shrey + cubic)

Comment-style sweep (per review): code comments now state present-tense constraints only — dated smoke references, "A/B/C experiment arm" vocabulary, and PR-number release status moved out of code; "skill-delivery arm" → "skill-delivery mode" everywhere. The EVAL_RESULT parser consolidation is tracked as STG-2516.

cubic-round fixes in this layer: line-anchored EVAL_RESULT parsing (marker inside a JSON payload can't hijack _success), stderr kept in successful browse output, no on-disk SKILL.md for external harnesses (installSkill: false; claude_code/codex unchanged), verifier-carrier deregistration survives adapter-cleanup failures, ADAPTER_BACKED_HARNESSES (= bare loops + cursor_sdk) now types the factory / gates planner skillMode metadata / backs --skill-mode validation, Testcase.metadata.skillMode typed, preview header shows the planned skill mode, loop-error messages redact API keys/bearer tokens/signed params/secret-bearing JSON fields, plus focused tests for the parser, bare-loop core, bare-loop adapter, preview header, and --skill-mode flag path.

Compose order vs. open PRs

Based on origin/main; composes with either merge order of #2299 (this stack only imports from claudeCodeToolAdapter.ts, no edits) and #2334 (injected mode reads the same BROWSER_SKILL_SOURCE constant, so it inherits whichever SKILL.md lands).

E2E Test Matrix

The three live smokes ran on the pre-split combined branch, whose tip content is identical to this stack's tip (verified by git diff — only intentional restructuring deltas: dynamic error-message lists, split registry tests, exported factory). Live rows live in the stacked PR that owns each runner; this base PR carries the shared rows:

Command / flow	Observed output	Confidence / sufficiency
pnpm exec vitest run (packages/evals, this branch)	Test Files 50 passed, Tests 367 passed — includes new externalHarnessToolAdapter.test.ts (harness classification, skill-mode parsing, verbatim bare prompt, quote-aware arg tokenizing, command gating incl. metacharacter rejection, real temp-binary execution)	Proves shared scaffolding behavior; runner loops proven in their own PRs
pnpm -w exec tsc -p packages/evals/tsconfig.json --noEmit + prettier + eslint	all clean	Types/style only
evals run b:webtailbench --harness vercel_ai_sdk -m anthropic/claude-haiku-4-5-20251001 -l 1 --dry-run (this branch)	matrix row resolves: toolSurface: browse_cli, startupProfile: tool_launch_local, browseCliVersion: 0.9.4	Proves planner/CLI plumbing for not-yet-executable harnesses
evals run b:webtailbench --harness vercel_ai_sdk ... -t 1 (this branch, non-dry)	Harness "vercel_ai_sdk" is dry-run only for now. Use an executable harness (stagehand, claude_code, codex) ...	Proves the mid-stack execution gate degrades gracefully
pnpm exec vitest run (this branch, post-fix)	Test Files 53 passed, Tests 394 passed (this layer)	Includes the new parser/bare-loop/adapter/preview/flag tests from the review rounds; full stack green at the tip (#2341): Test Files 57 passed, Tests 419 passed
pnpm -w exec tsc --noEmit + prettier + eslint (post-fix, this branch and stack tip)	all clean	Types/style only

No changeset: packages/evals is private (not released).

Stack mechanics for review

Merge top-down (#2337 first). Each stacked PR's base is the previous branch, so GitHub shows only its own delta. After a parent merges, the child needs its base retargeted to main (GitHub usually auto-retargets on branch delete) + a rebase.

🤖 Generated with Claude Code

[changeset-bot]

⚠️ No Changeset found

Latest commit: 75d9810

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

6 issues found across 15 files

Confidence score: 3/5

• In packages/evals/tui/commands/run.ts, --preview can hide the selected skill-delivery arm (prompt_show vs injected), so two materially different plans may look the same and reviewers could approve the wrong run configuration—surface runOptions.skillMode explicitly in preview output before merging.
• The new bare-loop execution path across packages/evals/framework/bareLoopRunner.ts, packages/evals/framework/harnesses/bareLoopAdapter.ts, and packages/evals/framework/evalResultParser.ts has little focused coverage, which raises regression risk in tool-call passthrough, result/status shaping, marker parsing, and usage/finalAnswer mapping—add targeted unit tests for these contracts to de-risk merge.
• packages/evals/tui/commands/help.ts currently presents external harnesses as normal --harness options even though execution is still gated, which can lead users to choose paths that fail later—label these harnesses as preview/plan-only in help text until execution support is enabled.
• The new flag flow in packages/evals/tui/commands/parse.ts is untested, so invalid-value handling and stagehand rejection could break silently while lower-level parsing tests still pass—add focused parseRunArgs/resolveRunOptions tests before merging.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/framework/harnesses/bareLoopAdapter.ts">

<violation number="1" location="packages/evals/framework/harnesses/bareLoopAdapter.ts:30">
P3: Bare-loop trajectory conversion is new executable behavior but has no focused test coverage, so regressions in tool-call passthrough, default status, finalAnswer, or usage mapping would be caught only indirectly. A small unit test for `BareLoopTrajectoryAdapter.fromHarnessResult()` covering the happy path and default status would lock the intended identity-style mapping.

(Based on your team's feedback about adding unit tests for new behavior.) [FEEDBACK_USED]</violation>
</file>

<file name="packages/evals/tui/commands/help.ts">

<violation number="1" location="packages/evals/tui/commands/help.ts:95">
P3: `evals run --help` now presents the four external harnesses as normal runnable `--harness` choices, but this PR only makes them parseable/plannable and leaves execution gated. Consider marking these entries as preview-only/non-executable so users do not start runs that later fail unexpectedly.</violation>
</file>

<file name="packages/evals/tui/commands/run.ts">

<violation number="1" location="packages/evals/tui/commands/run.ts:406">
P2: `--preview` hides the selected skill-delivery arm, so runs planned with `--skill-mode prompt_show` or `injected` can look identical in the human-readable plan. Since this new `runOptions.skillMode` field feeds both renderers, the preview header should also surface non-null skill mode.</violation>
</file>

<file name="packages/evals/tui/commands/parse.ts">

<violation number="1" location="packages/evals/tui/commands/parse.ts:279">
P3: The new TUI flag path is untested, so parsing, invalid-value handling, and the stagehand rejection could regress while `parseSkillDeliveryMode` tests still pass. A couple of focused `parseRunArgs`/`resolveRunOptions` cases would lock in the CLI behavior.

(Based on your team's feedback about adding unit tests for new behavior.) [FEEDBACK_USED]</violation>
</file>

<file name="packages/evals/framework/evalResultParser.ts">

<violation number="1" location="packages/evals/framework/evalResultParser.ts:35">
P3: This shared parser is now a core result-contract path for bare-loop adapters, but it has no focused tests for marker parsing, trailing SDK text, malformed JSON, or no-marker fallback. Adding direct unit tests would lock the new `_success` behavior before downstream harnesses depend on it.

(Based on your team's feedback about adding unit tests for new behavior.) [FEEDBACK_USED]</violation>
</file>

<file name="packages/evals/framework/bareLoopRunner.ts">

<violation number="1" location="packages/evals/framework/bareLoopRunner.ts:147">
P3: This new bare-loop core currently has no focused unit coverage, leaving prompt construction, clipping/recording, TaskResult shaping, and verifier handoff regressions easy to miss. Consider adding tests for the normal finalization path plus edge cases like invalid env limits, clipped failed tool output, and verifier failure fallback.

(Based on your team's feedback about adding unit tests for new behavior.) [FEEDBACK_USED]</violation>
</file>

Architecture diagram

sequenceDiagram
    participant CLI as CLI (tui/commands)
    participant Planner as Bench Planner
    participant HarnessReg as Harness Registry
    participant AdapterFactory as buildAdapterBackedHarness
    participant ToolAdapter as ExternalHarnessToolAdapter
    participant BrowseProvision as prepareBrowseCliHarnessAdapter
    participant BareLoopRunner as bareLoopRunner.ts
    participant EvalParser as evalResultParser.ts
    participant Verifier as Verifier (gradeExternalTrajectory)
    participant TrajectoryAdapter as bareLoopAdapter.ts
    
    Note over CLI,TrajectoryAdapter: NEW: External Harness Scaffolding Flow
    
    CLI->>CLI: Parse --skill-mode flag
    CLI->>CLI: Validate skillMode only for external harnesses
    
    CLI->>Planner: generateBenchTestcases(harness, skillMode)
    Note over Planner: NEW: resolveExternalHarnessStartupProfile
    
    alt External Harness (vercel_ai_sdk, anthropic_sdk, etc.)
        Planner->>Planner: Set toolSurface = "browse_cli"
        Planner->>Planner: Resolve models from EXTERNAL_HARNESS_MODEL_DEFAULTS
        Planner->>Planner: Resolve startup profile
        Planner->>Planner: Build row with skillMode in config
    else Stagehand
        Planner->>Planner: Legacy paths unchanged
    end
    
    Planner-->>CLI: Bench matrix rows (dry-run only for new harnesses)
    
    alt Dry-run mode
        CLI->>CLI: Print matrix, exit (harnesses not yet executable)
    else Executable run (harness registered)
        CLI->>HarnessReg: Lookup harness runner
        
        HarnessReg->>AdapterFactory: Create BenchHarness for external harness
        Note over AdapterFactory: NEW: Factory shared by all 4 external harnesses
        
        AdapterFactory->>AdapterFactory: buildVerifierCarrierV3()
        AdapterFactory->>ToolAdapter: prepareExternalHarnessAdapter(skillMode)
        
        alt skillMode = "none"
            ToolAdapter->>ToolAdapter: Use BARE_LOOP_DEFAULT_SYSTEM_PROMPT (verbatim one-liner)
        else skillMode = "prompt_show"
            ToolAdapter->>ToolAdapter: Append PROMPT_SHOW_SYSTEM_PROMPT_ADDENDUM
            ToolAdapter->>ToolAdapter: Log warning if browse CLI lacks #2335
        else skillMode = "injected"
            ToolAdapter->>ToolAdapter: Read SKILL.md from disk
            ToolAdapter->>ToolAdapter: Embed skill text in system prompt
        end
        
        ToolAdapter->>BrowseProvision: prepareBrowseCliHarnessAdapter()
        Note over BrowseProvision: Shared with codex: temp cwd, session name, browse wrapper
        
        BrowseProvision-->>ToolAdapter: adapter with cwd, env, browseBinPath, cleanup
        
        alt skillMode = "prompt_show"
            ToolAdapter->>ToolAdapter: Warn about unreleased browse CLI support
        end
        
        ToolAdapter-->>AdapterFactory: PreparedExternalHarnessAdapter with systemPromptAddendum
        
        AdapterFactory->>BareLoopRunner: runner({plan, model, toolAdapter, verifier})
        
        Loop For each tool call (up to DEFAULT_BARE_LOOP_MAX_STEPS = 40)
            BareLoopRunner->>BareLoopRunner: Build system prompt + user prompt
            BareLoopRunner->>BareLoopRunner: buildBareLoopUserPrompt(plan)
            Note over BareLoopRunner: User prompt includes EVAL_RESULT instructions
            
            BareLoopRunner->>BareLoopRunner: createBareLoopToolRecorder()
            
            BareLoopRunner->>ToolAdapter: runBareBrowseCommand(args)
            ToolAdapter->>ToolAdapter: tokenizeBrowseArgs(args)
            ToolAdapter->>ToolAdapter: isAllowedBrowseCommand(candidate)
            
            alt Command rejected (metacharacters)
                ToolAdapter-->>BareLoopRunner: ok=false, rejected message
            else Command allowed
                ToolAdapter->>BrowseProvision: execFile(browseBinPath, argv)
                BrowseProvision-->>ToolAdapter: stdout/stderr
                ToolAdapter-->>BareLoopRunner: ok=true, clipped output (20k char limit)
            end
            
            BareLoopRunner->>BareLoopRunner: Record NormalizedToolCall (name, args, result, ok, error, reasoning)
            BareLoopRunner->>BareLoopRunner: Log duration + status
        End
        
        BareLoopRunner->>EvalParser: parseEvalResultText(finalText)
        Note over EvalParser: Find EVAL_RESULT: marker, parse JSON
        EvalParser-->>BareLoopRunner: ParsedEvalResult (success, summary, finalAnswer)
        
        BareLoopRunner->>BareLoopRunner: Build base TaskResult with camelCase field names
        Note over BareLoopRunner: "vercel_ai_sdk" -> "vercelAiSdkStatus"
        
        alt Verifier configured
            BareLoopRunner->>TrajectoryAdapter: fromHarnessResult(toolCalls, finalAnswer, status)
            Note over TrajectoryAdapter: Near-identity: bare loop already has NormalizedToolCalls
            TrajectoryAdapter-->>BareLoopRunner: Trajectory object
            
            BareLoopRunner->>Verifier: gradeExternalTrajectory(trajectory, rubric)
            Note over Verifier: Same path as claude_code/codex: gemini-2.5-flash judge
            Verifier-->>BareLoopRunner: Graded TaskResult (overrides _success)
        end
        
        BareLoopRunner-->>AdapterFactory: Final TaskResult
        
        AdapterFactory->>ToolAdapter: cleanup() - stop --force, remove temp cwd
        AdapterFactory->>AdapterFactory: carrierV3.close() - deregister V3 instance
        AdapterFactory-->>HarnessReg: TaskResult complete
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 9 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 15 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 4 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>