[STG-2515] feat(evals): openai_agents_sdk harness adapter

[shrey150]

What

Part 4 of 5 — stacked on #2339 (anthropic_sdk) (base: shrey/evals-harness-anthropic-sdk; GitHub shows only this PR's delta). When the parent merges, retarget this PR's base and rebase.

Registers --harness openai_agents_sdk via @openai/agents (6.8M PyPI dl/wk for the Python twin). Classification: bare-ish — one notch above the raw provider loops. The SDK runs its own agent loop (turn management, tool dispatch, tracing) but ships no behavioral scaffolding: everything the agent knows comes from the dev-supplied instructions string, which carries only the configured skill-arm prompt. SDK defaults untouched except maxTurns (the SDK's own default of 10 is far too low for a browse task; shared 40 default, EVAL_OPENAI_AGENTS_SDK_MAX_TURNS override).

• OpenAI models only (openai/ prefix stripped; other providers rejected loudly).
• Tool parameters use a plain JSON schema instead of a zod schema to sidestep the SDK's zod version coupling (packages/evals pins zod v4).
• Usage summed from result.rawResponses.
• Adds the @openai/agents dependency (pinned 0.13.0) — this PR carries its own lockfile delta.
• Testability: sdk injection seam ({ Agent, run, tool } mock).

External harnesses design + usage: packages/evals/README.md#external-harnesses (in #2337). Linear: STG-2515.

Comparability fix (2026-07-09 audit)

@openai/agents-core throws MaxTurnsExceededError ("Max turns (N) exceeded", error.name === "MaxTurnsExceededError" per AgentsError's this.name = new.target.name) instead of returning a truncated result the way vercel_ai_sdk/anthropic_sdk do — the generic catch was folding that into openaiAgentsSdkStatus: "error", reporting the shared step cap as a harness/SDK failure. Now detects it (by error name, with a message-shape regex fallback for robustness) and reports status: "complete" + the same step cap reached (N) stopReason anthropic_sdk (#2339) uses. The previous single "SDK errors" test (which happened to throw exactly this message and was silently exercising the bug) is now three tests: a genuine unrelated SDK error (still "error"), the real MaxTurnsExceededError-shaped throw hitting the cap, and the bare message-shape fallback without the SDK's error name.

E2E Test Matrix

Live smoke ran on the pre-split combined branch (content identical to this stack's tip — verified by diff); results transplanted, not rerun.

Command / flow	Observed output	Confidence / sufficiency
evals run b:webtailbench --harness openai_agents_sdk -m openai/gpt-5.4-mini -l 1 -t 1 -c 1 (local build, real OPENAI + GEMINI keys, local Chrome via browse CLI)	Run completed (exit 0). 10 browse tool calls recorded; trajectory persisted; verifier graded: outcomeSuccess=false, processScore=1.0, 5 rubric criteria, no verifierError. Clean final answer: "No reliable flight availability/pricing could be retrieved for United direct flights ..." Usage: 24k in / 0.5k out tokens	Bar met: harness completes, trajectory captured, graded outcome, no verifierError. Task pass/fail is NOT the bar (united.com flights); process=1.0 with outcome=false is the verifier saying "sound procedure, unreachable goal"
pnpm exec vitest run (this branch)	Test Files 52 passed, Tests 380 passed — adds openaiAgentsSdkRunner.test.ts (instructions === verbatim arm prompt only, maxTurns passthrough, JSON-schema tool execution against a real temp binary, usage summing, model-prefix validation, MaxTurnsExceeded folding)	Mocked-SDK loop coverage; live behavior proven by the smoke row above
tsc --noEmit + prettier + eslint	clean	Types/style only
pnpm exec vitest run (this branch, post-fix)	Test Files 52 passed, Tests 385 passed — MaxTurnsExceeded now correctly reported as status: "completed" + step cap reached (N) (2 new dedicated tests: real error-name shape, message-shape fallback); the pre-existing "SDK errors" test now throws a genuinely unrelated error to keep testing the generic-error path	Proves the cap is no longer misreported as a harness failure, without losing coverage of real SDK errors
pnpm -w exec tsc --noEmit + prettier + eslint (post-fix)	clean	Types/style only

No changeset: packages/evals is private.

🤖 Generated with Claude Code

[changeset-bot]

⚠️ No Changeset found

Latest commit: 1fbd7c2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 7 files

Confidence score: 5/5

• Safe to merge after the addressed issues were fixed.

Architecture diagram

sequenceDiagram
    participant CLI as Evals CLI
    participant Registry as Harness Registry
    participant Runner as openaiAgentsSdkRunner
    participant AgentSDK as @openai/agents SDK
    participant Browse as Browse Binary
    participant Verifier as Verifier

    Note over CLI,Verifier: OPENAI AGENTS SDK HARNESS FLOW

    CLI->>Registry: registerHarness("openai_agents_sdk")
    Registry->>Registry: buildAdapterBackedHarness("openai_agents_sdk", runOpenAiAgentsSdkAgent)

    CLI->>Runner: runOpenAiAgentsSdkAgent({ plan, model, toolAdapter, logger })

    Note over Runner: normalizeOpenAiAgentsModel(model)
    alt Non-OpenAI provider
        Note over Runner: throws EvalsError("only accepts openai models")
    else OpenAI model
        Runner->>Runner: stripProviderPrefix("openai/...") → "gpt-5.4-mini"
    end

    Runner->>Runner: readBareLoopMaxSteps("EVAL_OPENAI_AGENTS_SDK_MAX_TURNS")
    Runner->>Runner: createBareLoopToolRecorder(toolAdapter, logger, "openai_agents_sdk")

    Runner->>Runner: sdk.tool({ name: "browse", parameters: { type: "object", properties: { args } }, execute: async (params) => ... })
    Runner->>AgentSDK: new Agent({ name, instructions: systemPromptAddendum, model, tools: [browseTool] })
    Runner->>AgentSDK: run(agent, buildBareLoopUserPrompt(plan), { maxTurns, signal })

    alt Run succeeds normally
        AgentSDK-->>Runner: result { finalOutput, rawResponses }
        Runner->>Runner: sumAgentsUsage(result.rawResponses)
        Note over Runner: Sums inputTokens/outputTokens from each rawResponse
        Runner-->>Verifier: finalizeBareLoopResult({ status: "complete", stopReason: undefined, usage, toolCalls })
        Verifier-->>CLI: TaskResult with graded metrics

    else SDK throws MaxTurnsExceededError
        AgentSDK--xRunner: Error("Max turns (N) exceeded") with name "MaxTurnsExceededError"
        Runner->>Runner: isMaxTurnsExceededError(error)
        Note over Runner: Checks error.name === "MaxTurnsExceededError" OR message matches /^Max turns \(\d+\) exceeded$/
        Runner->>Runner: cappedOut = true, logger.warn("hit max-turns cap")
        Runner-->>Verifier: finalizeBareLoopResult({ status: "complete", stopReason: "step cap reached (N)", usage: {0,0} })
        Verifier-->>CLI: TaskResult with capped-out completion

    else SDK throws other error
        AgentSDK--xRunner: Error("network error talking to the model provider")
        Runner->>Runner: loopError = error, stringifyLoopError(error)
        Runner-->>Verifier: finalizeBareLoopResult({ status: "error", stopReason: "network error...", usage: {0,0} })
        Verifier-->>CLI: TaskResult with failed status
    end

    Note over Runner,Browse: Tool Execution (during SDK run loop)
    AgentSDK->>+Runner: execute({ args: "open https://example.com" })
    Runner->>Runner: recorder.execute(args)
    Runner->>Browse: browse "open https://example.com"
    Browse-->>Runner: stdout output
    Runner-->>-AgentSDK: execution result string

    Note over Runner,AgentSDK: SDK manages its own turn loop, tool dispatch, and tracing

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>