Repository navigation
fix(sandbox): install skills natively per harness (AI-1034) - #240
Merged
Merged
Conversation
Skills were installed with a flagless `skills add`, which — finding no agent CLI installed yet — falls back to every one of the 71 agents the CLI knows. That scattered ~53 stray roots across the workspace (`.adal`, `.factory`, and non-dotted `data/` and `skills/` among them), and the workspace is exported into run artifacts and scored. It is also order-dependent: had an agent CLI been installed first, the fallback would have quietly stopped producing `.claude/skills` altogether. `skills add` now names the three CLI harnesses explicitly, which installs into exactly the two project scopes they discover natively: - `.claude/skills/` — Claude Code - `.agents/skills/` — Codex and OpenCode This is the actual fix for Codex, which does not read `.claude/skills` at all and therefore saw no skills in any eval. All three are installed unconditionally: the ids collapse to two directories, an unused copy costs a few kilobytes, and no agent id has to be threaded through `createAgentEnvironment` for correctness. Argument order matters — `--agent` is variadic, so the source directory must precede it and `--skill` terminates the list. `--copy` stays: symlink mode skips agents whose top-level directory does not already exist. The post-install check now verifies every agent scope, so a skill missing from one of them fails loudly instead of leaving that harness silently skill-less. The session's harness id is threaded through to the two prompt-addendum builders (`buildSkillsPrompt`, `buildToolSurfaceAddendum`, the latter lifted out of `startSession` so it is testable), but neither branches on it yet: every harness still gets the same injected text as before. What each one should actually be told is a separate change. Refs AI-1034, #164
Review fixes on the explicit-agent install: - The no-`--agent` fallback does reach both `.claude/skills` and `.agents/skills`, so it was never the reason skills failed to arrive. It is a pollution and determinism problem: ~52 stray roots in the scored workspace, and detection that depends on the environment. The order-dependence the comment claimed does not reproduce on 1.5.11. - `LocalStackSessionArgs.agent` said the addendum builders tailor their text to it. They take it and ignore it here. - Quote the staging dir in the install command. - The docker test checks a sample of the fallback's stray roots, not the whole set. Say so, and include the non-dotted ones. - README described the injected-listing path as though it were the native one.
The README claimed the harness does not describe installed skills. It still does, for every harness, and this change's own unit test pins that.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Aug 25, 2026
seanoliver
marked this pull request as ready for review
August 25, 2026 23:38
mattrossman
reviewed
Aug 28, 2026
…kills-install # Conflicts: # apps/framework/harness/run-eval.ts
`SKILLS_INSTALL_AGENTS` and `SKILLS_INSTALL_DIRS` were two hand-maintained lists. Adding a harness meant remembering both; forgetting one left that harness running evals with no skills installed, silently. Both are now derived from `SKILLS_PATH_BY_AGENT`, a `Record<AgentHarnessId, string | null>`. The record is exhaustive by type, so adding an id to `agentHarnessIdSchema` fails to compile until its project scope is declared. Verified: adding a fifth id errors with TS2741 naming the missing harness. `SKILLS_INSTALL_DIR` now reads from the same map rather than repeating `.claude/skills`, so the prompt listing cannot drift from the install either. Refs AI-1034
mattrossman
reviewed
Sep 7, 2026
mattrossman
approved these changes
Sep 7, 2026
mattrossman
left a comment
mattrossman
left a comment
Collaborator
There was a problem hiding this comment.
Sanity checked pnpm eval -- --eval build-cli-002-declarative-schema --experiment codex-gpt-5.4-mini
In that run it was reading Claude's skill files instead but it sounds like that will be fixed by subsequent PR #241 in the stack by trimming the buildSkillsPrompt
mattrossman
added a commit
that referenced
this pull request
Sep 11, 2026
) (#241) This is stacked on #240. Review that first; this diff is against it. No results are touched. ## Problem We wrote our own system prompt into every eval run, for all four agents: > You are an agent solving a Supabase eval task in a Linux workspace. Use the provided tools to inspect and modify the workspace and run commands. When you are done, end your turn with a short summary of what you did. - Claude Code, Codex and OpenCode ship their own system prompt. Ours was layered on top, and Codex and OpenCode have no system-prompt flag, so for them it went onto the user prompt. - It described `bash` and `files_read`, which only the in-process `ai-sdk` agent has. - "End your turn with a short summary" shapes `agentReport`, which the LLM judge reads. That is a scoring change. - It says Supabase, and "project". An agent that would otherwise pick a database has been handed the answer (#164). ## Changes | Agent | Before | After | |---|---|---| | `ai-sdk` | framing + tool list + skills list | one neutral line + tool list + skills list | | `claude-code` | ours appended to its own | nothing | | `codex` | ours prepended to the user prompt | nothing | | `opencode` | ours prepended to the user prompt | nothing | - CLI agents take no system prompt at all. `createCliAgent` no longer stages a system-prompt file or passes a path to the runner; `--append-system-prompt-file` and the Codex/OpenCode prepend are gone. A non-empty `systemPrompt` for a CLI agent throws. - The `ai-sdk` base prompt is now `You are an agent. Use the provided tools to complete the task.` It is the same in both modes. No Supabase, no project, no stopping instructions. - `buildToolSurfaceAddendum` and `buildSkillsPrompt` return `''` for every CLI agent. The bare sandbox no longer takes an `agent` or returns a `promptAddendum`. - Prompt assembly moves out of `run-eval.ts` into `system-prompt.ts` so it can be unit tested. - `run-eval.ts` only starts a run when it is the entry point. It used to call `main()` on import, which the harness unit tests now trigger. - Each run records `systemPrompt` in its raw `result.json`. `export-results` drops it, so nothing reaches the site. Since the first review: the system-prompt-file plumbing for CLI agents is deleted rather than made optional, and the ai-sdk base prompt is rewritten for #164. The branch is re-applied on the current #240 (the retry loop is gone there), so the commits are new. #240 now has `main` merged in, which brings the usage and duration fields from #270. ## Testing - `pnpm --filter @supabase-evals/core test`, `pnpm --filter @supabase-evals/sandbox test`, `pnpm --filter @supabase-evals/framework test`, `pnpm --filter @supabase-evals/web test`: all pass locally. - `typecheck` for framework, sandbox and web passes. `biome check .` is clean. - Unverified against real CLIs since the deletion. The earlier sandbox runs (Claude Code [32785480269](https://github.com/supabase/evals/actions/runs/32785480269), Codex [32880822811](https://github.com/supabase/evals/actions/runs/32880822811), OpenCode [32880839623](https://github.com/supabase/evals/actions/runs/32880839623)) recorded an empty prompt and still loaded the Supabase skills, and the path they exercised (no flag, task alone) is the one that remains. ## Score movement Full benchmark on both branches before this revision, artifact only, nothing published. Both used `runs=2`. | | #240, no prompt change | #241, prompt removed | | |---|---|---|---| | all pairs | 86.0% | 84.7% | 21 flips, 12 down 9 up | | `claude-code` | 92.3% | 92.3% | no change | | `codex` | 81.5% | 75.0% | down 6.5 points | | `opencode` | 82.6% | 89.1% | up 6.5 points | This looks like run-to-run variability. Once #229 lands and scores average across runs, the numbers should settle. ## Follow-ups - Re-run the benchmark comparison once #229 lands and the numbers can carry attribution. - Drop the "end your turn with a short summary" line from `evals/resolve-performance-001-slow-query-cpu-spike/PROMPT.md:15`, the last one left. It sits in the task text so it hits every agent equally, and removing it invalidates that eval's recorded results, so it should be a separate change. Ref AI-1034 --------- Co-authored-by: Matt Rossman <22670878+mattrossman@users.noreply.github.com>
Closed
NickLittman
added a commit
that referenced
this pull request
Sep 18, 2026
## What's inside Adds Grok Build, xAI's coding agent, as a fourth CLI harness alongside Claude Code, Codex and OpenCode. - ~640 lines of runner + parser, the only files needing real review (packages/core/src/agents/grok/) - ~400 lines of parser tests, skimmable - ~95 lines of experiments and plumbing: grok-4.6, grok-4.6-no-skills, XAI_API_KEY through the workflow, the registry entry, the web label - ~17k lines of results data, committed by the bot from two workflow runs Grok Build is xAI's coding agent. It calls the xAI API directly, so it needs XAI_API_KEY rather than the gateway key the other harnesses share. - GROK_HOME goes in the shell command, not the sandbox env map. docker exec --env passes values verbatim, so a $HOME in an env value reaches the CLI unexpanded. The CLI resolves it against the cwd, writes its config, sessions and binary into a directory named $HOME inside the scored workspace, and never reads the config.toml we stage. MCP servers silently fail to connect, and the run scores 0 with an empty report that looks like a model failure. - Grok reads .claude/skills natively, but the skills CLI has no grok agent id and rejects the whole install when one is named. SKILLS_PATH_BY_AGENT gives grok null, and the claude-code entry populates the directory it reads. Drop claude-code from that map and grok loses its skills with nothing failing. Follow-up below. ## Results - with skills (64/69 - 93%) - without (58/69 - 84%) 23 benchmark evals × 3 runs on Vercel Sandboxes. Not comparable to the other rows yet: they were last refreshed on 2026-09-09 (d462078), so they predate #278, #240 and #241, including the system-prompt removal that #241 calls a scoring change. ## How to review Needs XAI_API_KEY in .env. Use --concurrency 1 locally; Docker drops containers under load and reports it as a timeout with a 0 score. pnpm eval -- --experiment grok-4.6 --eval resolve-dataapi-001-empty-results ## Follow up tasks - Split "scope a harness reads" from "--agent ids the CLI accepts" in SKILLS_PATH_BY_AGENT, so grok declares .claude/skills and a missing install fails loudly instead of silently. - Refresh the other five benchmark experiments. Every non-Grok row predates #278/#240/#241, so the published table currently compares across harness versions. - The Grok CLI is pinned to 1.0.13; xAI shipped 12 releases in the following 12 days. The streaming-json schema is undocumented and its embedded docs already describe a stream it doesn't emit, so bump deliberately and re-check the parser against a real transcript. Ref AI-1143 --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Matt Rossman <22670878+mattrossman@users.noreply.github.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's inside
Bottom of a 2-PR stack under #241, which removes the system prompt we write for the CLI agents.
Claude Tag wrote the first pass at this (previously #180, now closed).
Problem
Installing a skill means copying its folder somewhere an agent will look, and each agent reads a different directory.
skills addtakes a flag naming which agents to install for, but we weren't not passing it..aider-desk,.factory,.kilocode,.windsurf,.zencoder, and non-dotteddata/andskills/among them. Each holds a copy of the same two Supabase skills..agents/skillsfolder, which required us to include an addendum to the Codex prompt to get it to look in the Claude skills folder (.claude/skills).Changes
Install
Name the three agents we run in this repo:
That writes two directories,
.claude/skillsand.agents/skills, instead of 52.--copyis required so that.claude/skillsgets created in an empty eval workspace. Without it only.agents/skillsshows up.After installing,
installSkillsconfirms every skill is actually present in both directories (.claude/skillsand.agents/skills). It used to check.claude/skillsonly.Plumbing for the next PR
The prompt builders now take the agent as an argument and ignore it. This is used by #241 to skip the prompt text for the CLI agents.
How to review
Confirm the new install works:
If interested, run both install strategies side by side in a temp folder to see the difference:
You should see 53 entries (old) vs. 3 (new).
Follow up tasks
skills-lock.jsonfrom the workspace after installing. The CLI drops it in the root and it ends up in the exported workspace, same as before this change.Ref AI-1034