feat(cli): add the cli eval suite and experiment suite plumbing

[Coly010]

Final chunk of the closed #308. This is the piece @mattrossman asked for — the minimal CLI suite plumbing and CODEOWNERS, standalone, with no CLI experiment internals to review. My own commit is 258 lines across 14 files, and five of those files are 11–13 lines each.

Draft, and the diff is temporarily larger than the change. This depends on three unmerged PRs at once — #315 (the experiments/<owner>/ layout), #326 (typechecking experiments/), and #324 + #325 (the sandbox options the experiment files call). All three are merged into this branch so it actually typechecks and tests green. Once they land I rebase onto main, the diff collapses to just the commit above, and I mark this ready. Review 64722a8 alone in the meantime.

What it adds

cli as an eval suite and an experiment suite, @supabase/cli ownership over /evals/cli/, /experiments/cli/ and cli-eval-results.json, and the workflow wiring so eval-refresh and append-gh-pages-history know the suite exists.

The substance is five environment columns: one scenario run unchanged across forced CLI environments, so a failure isolates to which environment broke. They're thin now that experiments/presets.ts exists — the whole of the Docker-less column is:

export default defineExperiment({
  ...baselineExperiment,
  suite: ['cli'],
  // beta: the Docker-less path only exists in the managed stack, which ships in beta.
  localStack: localStackRuntime({ cliVersion: 'beta', docker: 'absent' }),
  skipEval: skipUnlessDockerless,
});

column	environment	picks up
codex-gpt-5.6-luna	repo-pinned CLI, Docker available	every interface: cli eval
…-cli-stable	npm latest	every interface: cli eval
…-cli-beta	npm beta	every interface: cli eval
…-cli-nodaemon	beta, daemon unreachable	also needs needsDocker: false + projectRunning: false
…-cli-absent	beta, no docker binary	also needs needsDocker: false + projectRunning: false

The pinned column is the existing baseline with 'cli' added to its suite array, not a new file — same as how the docs suite works today (codex-gpt-5.6-luna-no-skills carries 'docs'; the docs team never got a duplicate column). A dedicated file would be a byte-identical copy of baselineExperiment under a second name. That's the one line here outside CLI-owned paths, hence @supabase/ai on the review.

skipUnlessCli / skipUnlessDockerless are the only things left over from the old experiments/_lib/; they live in experiments/cli/lib/, which discovery ignores for free since it only matches *.experiment.ts directly under an owner directory.

Verified, including the parts tests can't reach

format:check, typecheck, and the sandbox (89), core (162), framework (8), test:cli-lib (11), vercel-runner (29) and web (83) suites all pass.

Two things unit tests can't cover, checked directly instead:

• Discovery, since it's filename-convention-based and fails silently: all 16 experiments resolve and load, experiments/cli/lib/ is correctly not treated as an experiment, and the five columns resolve to the intended runtimes (local-stack, local-stack-stable, local-stack-beta, local-stack-beta-no-daemon, local-stack-beta-absent).
• The workflow's version-resolution step, which only ever runs in CI. Ran its exact node --import tsx/esm body against live npm: stable: 2.117.0, beta: 2.118.0-beta.60, with the release-asset existence check passing on beta.60.

test:cli-lib carries --passWithNoTests deliberately: evals/cli/ has no scorer tests until #281/#314/#316 land, and without the flag the script's exit code depends on which of its two paths happens to be populated.

Next

#281, #314, #316 and #307 get re-parented onto this branch in one pass once the dependency chain above clears — they currently point at the closed #308's retained branch. /cc @Rodriguespn @kanadgupta

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored	Preview	Sep 22, 2026 7:30pm UTC