fix(framework): remove the harness system prompt for CLI agents (AI-1034) by seanoliver · Pull Request #241 · supabase/evals

seanoliver · 2026-08-25T22:17:43Z

This is stacked on #240. Review that first; this diff is against it. No results are touched.

Problem

We wrote our own system prompt into every eval run, for all four agents:

You are an agent solving a Supabase eval task in a Linux workspace. Use the provided tools to inspect and modify the workspace and run commands. When you are done, end your turn with a short summary of what you did.

Claude Code, Codex and OpenCode ship their own system prompt. Ours was layered on top, and Codex and OpenCode have no system-prompt flag, so for them it went onto the user prompt.
It described bash and files_read, which only the in-process ai-sdk agent has.
"End your turn with a short summary" shapes agentReport, which the LLM judge reads. That is a scoring change.
It says Supabase, and "project". An agent that would otherwise pick a database has been handed the answer (Base prompt #164).

Changes

Agent	Before	After
`ai-sdk`	framing + tool list + skills list	one neutral line + tool list + skills list
`claude-code`	ours appended to its own	nothing
`codex`	ours prepended to the user prompt	nothing
`opencode`	ours prepended to the user prompt	nothing

CLI agents take no system prompt at all. createCliAgent no longer stages a system-prompt file or passes a path to the runner; --append-system-prompt-file and the Codex/OpenCode prepend are gone. A non-empty systemPrompt for a CLI agent throws.
The ai-sdk base prompt is now You are an agent. Use the provided tools to complete the task. It is the same in both modes. No Supabase, no project, no stopping instructions.
buildToolSurfaceAddendum and buildSkillsPrompt return '' for every CLI agent. The bare sandbox no longer takes an agent or returns a promptAddendum.
Prompt assembly moves out of run-eval.ts into system-prompt.ts so it can be unit tested.
run-eval.ts only starts a run when it is the entry point. It used to call main() on import, which the harness unit tests now trigger.
Each run records systemPrompt in its raw result.json. export-results drops it, so nothing reaches the site.

Since the first review: the system-prompt-file plumbing for CLI agents is deleted rather than made optional, and the ai-sdk base prompt is rewritten for #164. The branch is re-applied on the current #240 (the retry loop is gone there), so the commits are new. #240 now has main merged in, which brings the usage and duration fields from #270.

Testing

pnpm --filter @supabase-evals/core test, pnpm --filter @supabase-evals/sandbox test, pnpm --filter @supabase-evals/framework test, pnpm --filter @supabase-evals/web test: all pass locally.
typecheck for framework, sandbox and web passes. biome check . is clean.
Unverified against real CLIs since the deletion. The earlier sandbox runs (Claude Code 32785480269, Codex 32880822811, OpenCode 32880839623) recorded an empty prompt and still loaded the Supabase skills, and the path they exercised (no flag, task alone) is the one that remains.

Score movement

Full benchmark on both branches before this revision, artifact only, nothing published. Both used runs=2.

	#240, no prompt change	#241, prompt removed
all pairs	86.0%	84.7%	21 flips, 12 down 9 up
`claude-code`	92.3%	92.3%	no change
`codex`	81.5%	75.0%	down 6.5 points
`opencode`	82.6%	89.1%	up 6.5 points

This looks like run-to-run variability. Once #229 lands and scores average across runs, the numbers should settle.

Follow-ups

Re-run the benchmark comparison once feat: average eval scores across independent runs #229 lands and the numbers can carry attribution.
Drop the "end your turn with a short summary" line from evals/resolve-performance-001-slow-query-cpu-spike/PROMPT.md:15, the last one left. It sits in the task text so it hits every agent equally, and removing it invalidates that eval's recorded results, so it should be a separate change.

Ref AI-1034

vercel · 2026-08-25T22:17:51Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored	Preview	Sep 11, 2026 2:05pm UTC

mattrossman · 2026-09-07T16:57:53Z

Tested running pnpm eval -- --eval build-cli-002-declarative-schema --experiment codex-gpt-5.4-mini and I see Codex reading its natively supported .agents/skills files now instead of Claude's.

createCliAgent no longer stages a system prompt file or hands a path to the runner. Claude Code drops --append-system-prompt-file; Codex and OpenCode send the task alone instead of prepending a block to the user prompt. A non-empty systemPrompt now throws rather than being dropped.

) buildToolSurfaceAddendum and buildSkillsPrompt return '' for every CLI harness: those describe in-process tools only aiSdkAgent has, and each CLI discovers the installed skills itself. The bare sandbox no longer takes an agent or returns a promptAddendum.

AI-1034) Prompt assembly moves to system-prompt.ts so it can be unit tested. The ai-sdk base prompt no longer names Supabase, a project, or how to end the turn (#164), and is the same in both modes. CLI agents get an empty prompt. Each run records the system prompt it sent.

…arness-native-prompts # Conflicts: # apps/framework/harness/run-eval.ts # packages/core/src/agents/codex/runner.test.ts

run-eval.test.ts imports run-eval.ts for a helper, and the new `vitest run harness` script picks that test up, so the unconditional main() call ran on import. Also: drop a duplicated opencode test, assert the CLI engine refuses a system prompt before install, document the promptAddendum contract, and take buildSystemPrompt's args as an object.

…arness-native-prompts # Conflicts: # packages/core/src/agents/codex/runner.test.ts

## What's inside - ~215 lines of skills install changes ([skills.ts](https://github.com/supabase/evals/pull/240/changes#diff-badfb2869f8854f19b00a8e02d6e9d94ef4e1cea66004399b290c03b8ecfb1fa), [local-stack-runtime.ts](https://github.com/supabase/evals/pull/240/changes#diff-366863d3b55e1541a2a038a7af0efb7952e345a84cdcdcce85727ca5dd0aef7f)) - ~5 lines of plumbing that does nothing yet ([index.ts](https://github.com/supabase/evals/pull/240/changes#diff-9a4ceebe7c6f86856371906c3f061d3b56b7457022b05179884a113e7ced67e8)), for the PR stacked on this one - ~195 lines of tests, skimmable ([unit.test.ts](https://github.com/supabase/evals/pull/240/changes#diff-d9864aa0981cd32280dc8a1cca41ada618e91ef3684f52c4c49234e378aee648), [docker.test.ts](https://github.com/supabase/evals/pull/240/changes#diff-216a9cc0506fc98aac7b13fd029e70581f3e089c35dcde73fac0f2217e93b00f)) Bottom of a 2-PR stack under #241, which removes the system prompt we write for the CLI agents. Claude Tag wrote the first pass at this (previously #180, now closed). ## Problem Installing a skill means copying its folder somewhere an agent will look, and each agent reads a different directory. - `skills add` takes a flag naming which agents to install for, but we weren't not passing it. - Without that flag, the CLI tries to detect an installed agent, and when it cannot, it installs for all 71 it knows about. - So every sandbox got ~52 folders it had no use for: `.aider-desk`, `.factory`, `.kilocode`, `.windsurf`, `.zencoder`, and non-dotted `data/` and `skills/` among them. Each holds a copy of the same two Supabase skills. - It also didn't install Codex skills into the native `.agents/skills` folder, which required us to include an addendum to the Codex prompt to get it to look in the Claude skills folder (`.claude/skills`). ## Changes ### Install Name the three agents we run in this repo: ```bash skills add '<dir>' --agent claude-code codex opencode --skill '*' --copy --yes ``` That writes two directories, `.claude/skills` and `.agents/skills`, instead of 52. `--copy` is required so that `.claude/skills` gets created in an empty eval workspace. Without it only `.agents/skills` shows up. After installing, `installSkills` confirms every skill is actually present in both directories (`.claude/skills` and `.agents/skills`). It used to check `.claude/skills` only. ### Plumbing for the next PR The prompt builders now take the agent as an argument and ignore it. This is used by #241 to skip the prompt text for the CLI agents. ## How to review Confirm the new install works: ```bash pnpm --filter @supabase-evals/sandbox test:docker ``` If interested, run both install strategies side by side in a temp folder to see the difference: ```bash mkdir -p /tmp/probe/stage/skills/demo /tmp/probe/old /tmp/probe/new printf -- '---\nname: demo\ndescription: Demo.\n---\n\nBody.\n' > /tmp/probe/stage/skills/demo/SKILL.md # old, no --agent cd /tmp/probe/old env -u CLAUDECODE -u CLAUDE_CODE_ENTRYPOINT npx -y skills@1.5.11 add /tmp/probe/stage --skill '*' --copy --yes ls -A # new, three agents named cd /tmp/probe/new env -u CLAUDECODE -u CLAUDE_CODE_ENTRYPOINT npx -y skills@1.5.11 add /tmp/probe/stage --agent claude-code codex opencode --skill '*' --copy --yes ls -A ``` You should see 53 entries (old) vs. 3 (new). ## Follow up tasks - Remove `skills-lock.json` from the workspace after installing. The CLI drops it in the root and it ends up in the exported workspace, same as before this change. - Assert the full set of workspace directories in the docker test. It currently samples five of the ~52 the old fallback created, so it catches that fallback firing but does not prove nothing else was written. - Add a check that each agent actually surfaces the skills it finds. Both this PR and #241 assume it, and nothing tests it. Ref AI-1034 --------- Co-authored-by: Matt Rossman <22670878+mattrossman@users.noreply.github.com>

## What's inside Adds Grok Build, xAI's coding agent, as a fourth CLI harness alongside Claude Code, Codex and OpenCode. - ~640 lines of runner + parser, the only files needing real review (packages/core/src/agents/grok/) - ~400 lines of parser tests, skimmable - ~95 lines of experiments and plumbing: grok-4.6, grok-4.6-no-skills, XAI_API_KEY through the workflow, the registry entry, the web label - ~17k lines of results data, committed by the bot from two workflow runs Grok Build is xAI's coding agent. It calls the xAI API directly, so it needs XAI_API_KEY rather than the gateway key the other harnesses share. - GROK_HOME goes in the shell command, not the sandbox env map. docker exec --env passes values verbatim, so a $HOME in an env value reaches the CLI unexpanded. The CLI resolves it against the cwd, writes its config, sessions and binary into a directory named $HOME inside the scored workspace, and never reads the config.toml we stage. MCP servers silently fail to connect, and the run scores 0 with an empty report that looks like a model failure. - Grok reads .claude/skills natively, but the skills CLI has no grok agent id and rejects the whole install when one is named. SKILLS_PATH_BY_AGENT gives grok null, and the claude-code entry populates the directory it reads. Drop claude-code from that map and grok loses its skills with nothing failing. Follow-up below. ## Results - with skills (64/69 - 93%) - without (58/69 - 84%) 23 benchmark evals × 3 runs on Vercel Sandboxes. Not comparable to the other rows yet: they were last refreshed on 2026-09-09 (d462078), so they predate #278, #240 and #241, including the system-prompt removal that #241 calls a scoring change. ## How to review Needs XAI_API_KEY in .env. Use --concurrency 1 locally; Docker drops containers under load and reports it as a timeout with a 0 score. pnpm eval -- --experiment grok-4.6 --eval resolve-dataapi-001-empty-results ## Follow up tasks - Split "scope a harness reads" from "--agent ids the CLI accepts" in SKILLS_PATH_BY_AGENT, so grok declares .claude/skills and a missing install fails loudly instead of silently. - Refresh the other five benchmark experiments. Every non-Grok row predates #278/#240/#241, so the published table currently compares across harness versions. - The Grok CLI is pinned to 1.0.13; xAI shipped 12 releases in the following 12 days. The streaming-json schema is undocumented and its embedded docs already describe a stream it doesn't emit, so bump deliberately and re-check the parser against a real transcript. Ref AI-1143 --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Matt Rossman <22670878+mattrossman@users.noreply.github.com>

This was referenced Aug 25, 2026

fix(sandbox): install skills natively per harness (AI-1034) #240

Merged

fix(evals): remove synthetic system prompt hints, install skills natively per harness (AI-1034) #180

Closed

seanoliver marked this pull request as ready for review August 25, 2026 23:38

seanoliver requested review from a team and mattrossman and removed request for mattrossman August 25, 2026 23:49

mattrossman reviewed Sep 7, 2026

View reviewed changes

Comment thread apps/framework/harness/system-prompt.ts Outdated

mattrossman reviewed Sep 7, 2026

View reviewed changes

Comment thread packages/core/src/agents/engine.test.ts Outdated

mattrossman reviewed Sep 7, 2026

View reviewed changes

Comment thread apps/framework/harness/run-eval.ts Outdated

seanoliver added 3 commits September 8, 2026 13:50

seanoliver force-pushed the sean/ai-1034-harness-native-prompts branch from 5bed078 to 3cefa60 Compare September 8, 2026 20:51

seanoliver added 2 commits September 8, 2026 14:00

Merge branch 'sean/ai-1034-native-skills-install' into sean/ai-1034-h…

b737d6d

…arness-native-prompts # Conflicts: # apps/framework/harness/run-eval.ts # packages/core/src/agents/codex/runner.test.ts

test(core): restore the usage tests after the merge

17c738c

seanoliver requested a review from mattrossman September 8, 2026 21:16

seanoliver and others added 3 commits September 8, 2026 14:23

test(core): observe side effects when prompt guard rejects

15e2a23

Merge branch 'sean/ai-1034-native-skills-install' into sean/ai-1034-h…

1165068

…arness-native-prompts # Conflicts: # packages/core/src/agents/codex/runner.test.ts

vercel Bot deployed to Preview September 11, 2026 13:52

View deployment

docs: trim readme

cc83942

mattrossman approved these changes Sep 11, 2026

View reviewed changes

mattrossman merged commit cd5b8d0 into main Sep 11, 2026
7 checks passed

mattrossman mentioned this pull request Sep 11, 2026

Base prompt #164

Closed

NickLittman mentioned this pull request Sep 15, 2026

feat(core): add a native Grok Build CLI harness #300

Merged

Navigation Menu

Repository navigation

fix(framework): remove the harness system prompt for CLI agents (AI-1034) - #241

fix(framework): remove the harness system prompt for CLI agents (AI-1034)#241
mattrossman merged 9 commits into
sean/ai-1034-native-skills-installfrom
sean/ai-1034-harness-native-prompts

seanoliver commented Aug 25, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Aug 25, 2026 •

edited

Loading

Uh oh!

mattrossman commented Sep 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

seanoliver commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

Testing

Score movement

Follow-ups

Uh oh!

vercel Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattrossman commented Sep 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

seanoliver commented Aug 25, 2026 •

edited

Loading

vercel Bot commented Aug 25, 2026 •

edited

Loading