feat(core): add a native Grok Build CLI harness by NickLittman · Pull Request #300 · supabase/evals

NickLittman · 2026-09-15T23:44:12Z

What's inside

Adds Grok Build, xAI's coding agent, as a fourth CLI harness alongside Claude Code, Codex and OpenCode.

~640 lines of runner + parser, the only files needing real review (packages/core/src/agents/grok/)
~400 lines of parser tests, skimmable
~95 lines of experiments and plumbing: grok-4.6, grok-4.6-no-skills, XAI_API_KEY through the workflow, the registry entry, the web label
~17k lines of results data, committed by the bot from two workflow runs

Grok Build is xAI's coding agent. It calls the xAI API directly, so it needs XAI_API_KEY rather than the gateway key the other harnesses share.

GROK_HOME goes in the shell command, not the sandbox env map. docker exec --env passes values verbatim, so a $HOME in an env value reaches the CLI unexpanded. The CLI resolves it against the cwd, writes its config, sessions and binary into a directory named $HOME inside the scored workspace, and never reads the config.toml we stage. MCP servers silently fail to connect, and the run scores 0 with an empty report that looks like a model failure.
Grok reads .claude/skills natively, but the skills CLI has no grok agent id and rejects the whole install when one is named. SKILLS_PATH_BY_AGENT gives grok null, and the claude-code entry populates the directory it reads. Drop claude-code from that map and grok loses its skills with nothing failing. Follow-up below.

Results

with skills (64/69 - 93%)
without (58/69 - 84%)

23 benchmark evals × 3 runs on Vercel Sandboxes.

Not comparable to the other rows yet: they were last refreshed on 2026-09-09 (d462078), so they predate #278, #240 and #241, including the system-prompt removal that #241 calls a scoring change.

How to review

Needs XAI_API_KEY in .env. Use --concurrency 1 locally; Docker drops containers under load and reports it as a timeout with a 0 score.

pnpm eval -- --experiment grok-4.6 --eval resolve-dataapi-001-empty-results

Follow up tasks

Split "scope a harness reads" from "--agent ids the CLI accepts" in SKILLS_PATH_BY_AGENT, so grok declares .claude/skills and a missing install fails loudly instead of silently.
Refresh the other five benchmark experiments. Every non-Grok row predates feat: record step and tool call counts for eval runs #278/fix(sandbox): install skills natively per harness (AI-1034) #240/fix(framework): remove the harness system prompt for CLI agents (AI-1034) #241, so the published table currently compares across harness versions.
The Grok CLI is pinned to 1.0.13; xAI shipped 12 releases in the following 12 days. The streaming-json schema is undocumented and its embedded docs already describe a stream it doesn't emit, so bump deliberately and re-check the parser against a real transcript.

Ref AI-1143

Grok Build is a fourth CLI agent, with Claude Code, Codex and OpenCode. It runs against xAI directly, and not through the AI Gateway. Its runner and parser hold four things that apply only to Grok. - GROK_HOME keeps a run separate from the user. If you do not set it, the CLI reads the `~/.grok` directory of the user. Then the scorers measure tools that the experiment did not declare. The runner sets GROK_HOME in the shell command, and not in the sandbox `env` map. The sandbox uses `docker exec --env`, which does not expand `$HOME`. An unexpanded value makes a directory with the name `$HOME` in the workspace, and the scorers then measure that directory. - The MCP configuration is TOML, not JSON. Each server becomes an `[mcp_servers.<name>]` table with an `env` table in it. The runner also stops the skills marketplace. Thus an evaluation uses only the skills that it declares. - The CLI sends one `text` or `thought` event for each token. The parser joins the adjacent events of one type into one event. One tool call sends more than one `tool_call_update` event, thus the parser sends one result at the end. - The CLI does not show the MCP tools directly. Grok finds a tool with `search_tool`, then starts it with `use_tool`. The parser divides the name and gives the scorers the same `{kind:'mcp', server, toolName}` data that Claude Code and Codex give. The usage data, the stop reasons and the tool names agree with the other harnesses. For example, a run that stops at the time limit gives no usage data, because Claude Code and Codex also give none. This change also adds `spacexai` as a model provider, and adds its labels and icon to the web app. It sends XAI_API_KEY to the eval-refresh workflow and to the Vercel sandbox runner, for remote runs. Grok Build reads `.claude/skills` natively. But the skills CLI has no `grok` agent id, thus `SKILLS_PATH_BY_AGENT` gives grok a null scope. The `claude-code` entry populates the directory that grok reads.

Every other benchmark agent has a `-no-skills` twin, which runs the same model with no skills. The difference between the two shows how much the Supabase skills help that model. Grok had no twin, thus it was absent from the no-skills comparison. The two experiments differ only in `suite` and `skills`.

The fixture carried the session id from the real run it came from. The parser does not read that field, and the Claude Code and Codex fixtures use short synthetic ids (`s1`, `t1`).

vercel · 2026-09-15T23:44:19Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Sep 18, 2026 9:53pm UTC

mattrossman · 2026-09-18T15:45:13Z

Interesting to see Grok Build topping the chart overall!

Before merging this I'll plan to pull this branch up to speed with main and refresh the results one more, as you mentioned we recently refreshed the full benchmark in #306 which shifted numbers around, primarily due to bias removal from terminal agents' system prompt in #241 as well as pulling forward tooling versions (e.g. latest Supabase CLI), and I'll want to make sure we're representing Grok Build fairly against the others.

mattrossman · 2026-09-18T18:44:09Z

reasoningEffort: 'high',

👍 Confirming I see "high" as the default on https://docs.x.ai/developers/grok-4-6#at-a-glance

mattrossman · 2026-09-18T20:53:23Z

Due to the quirk in AI-1005 I'm triggering the refresh in 2 phases.

With skills

gh workflow run eval-refresh.yml --ref feat/grok-native-harness -f suite=benchmark -f experiment_suite=benchmark -f experiments=grok-4.6 -f runs=3 -f merge=true -f commit_to_branch=true

Run: 🟢 https://github.com/supabase/evals/actions/runs/35393677080

Notes:

I thought build-functions-005-dual-auth-user-secret might need a re-run since it hit a 502 from edge functions runtime, which seemed related to the ensureEdgeRuntime() workaround from chore: bump harness and Supabase CLI pins to latest #291, but I'm pretty sure this is just the sandbox timing out from the legitimately slow agent solution

Without skills

gh workflow run eval-refresh.yml --ref feat/grok-native-harness -f suite=benchmark -f experiment_suite=no-skills -f experiments=grok-4.6-no-skills -f runs=3 -f merge=true -f commit_to_branch=true

Run: 🟢 https://github.com/supabase/evals/actions/runs/35395340881

mattrossman · 2026-09-18T21:36:29Z

Results have shuffled around since the earlier state, Grok no longer on top:

Benchmark	Without Skills

Looks like the score decrease is mainly because it's taking too long to finish (either due to newer Grok CLI, or newer Supabase CLI) and ends up getting disqualified by timeout, which (I think) is a legit failure reason. It was already timing out a lot on the earlier Grok version so I don't see this as a net new concern. We could consider revisiting build-functions-005 in a later PR to see why it's more likely to exhaust the time limit on it.

Brings in #300 (native Grok Build CLI harness). Two conflicts, both additive on each side, resolved by keeping both: - `.github/workflows/eval-refresh.yml`: `XAI_API_KEY` alongside the `SUPABASE_CLI_{STABLE,BETA}_VERSION` run pins, in both the job `env` block and the `.env` heredoc. - `apps/framework/scripts/run-vercel-evals.ts`: `XAI_API_KEY` added to `FORWARDED_ENV_NAMES` ahead of the CLI version pins.

NickLittman and others added 5 commits September 15, 2026 17:32

chore: refresh eval results

4693a3e

chore: refresh eval results

3a184ef

test(core): use a synthetic session id in the Grok fixture

2adb396

The fixture carried the session id from the real run it came from. The parser does not read that field, and the Claude Code and Codex fixtures use short synthetic ids (`s1`, `t1`).

NickLittman requested a review from a team as a code owner September 15, 2026 23:44