feat(framework): add cli suites, needsDocker, and CLI environment arms

[Coly010]

Current Behavior

There is no home for CLI-team evals: the only suites are benchmark, regression, docs, other; an experiment can only opt out of evals by id; every local-stack experiment installs the pinned CLI into a sandbox where Docker always works. "One task, run across the environments the CLI actually meets" — Docker available / daemon unreachable / absent, stable vs beta CLI — cannot be expressed without touching other teams' experiments.

Expected Behavior

Infrastructure only — no evals in this PR (the first one, build-database-002-stack-lifecycle, is stacked on it in #281). Mirrors how the docs suite was added in #282.

Framework

• A cli eval suite (evals/cli/, owned by @supabase/cli) and a cli experiment suite. Nightly runs cli × cli next to regression × regression; results export to apps/web/src/data/cli-eval-results.json with a cli-results.jsonl history on GitHub Pages. Pairing is by eval folder, so CLI experiment variants never pair with other teams' evals, and other teams' experiments never see CLI evals.
• A needsDocker frontmatter field (default true). An eval that can run — and is meaningful — without a Docker daemon sets needsDocker: false; Docker-less experiments select on it with skipEval: (ev) => ev.metadata.needsDocker !== false instead of maintaining id lists.
• README/CONTRIBUTING: suite list, the field, and a "CLI evals" section (ownership, experiments, add/refresh/history, unit-test command).

Experiment-land runtime — experiments/_lib/ (composes @supabase-evals/sandbox exports; no framework internals touched)

• cli-channel.ts: resolves stable/beta to a concrete Supabase CLI version from npm dist-tags (latest/beta), lazily inside startSession (never on --dry/list), memoised, overridable via SUPABASE_CLI_STABLE_VERSION/SUPABASE_CLI_BETA_VERSION, and failing loudly rather than falling back to the pin.
• docker-aware-local-stack.ts: dockerAwareLocalStackRuntime({ channel, docker }). docker: 'available' delegates to the stock runtime with the resolved CLI. 'no-daemon'/'absent' build the sandbox from the same exported pieces and make Docker unusable in layers the agent can't undo: DOCKER_HOST=tcp://127.0.0.1:1 on every command, a root-owned supabase PATH shim that re-exports it, removal of the real docker binaries, and for no-daemon a root-owned docker shim that still answers --version (so the CLI's managed-stack runtime probe, fix(cli): improve stack startup defaults and clean managed state cli#6563, selects Docker as on a real host) but fails everything else with the canonical daemon error. Permissions alone can't do this: the CI sandbox chmod 666s the socket. A marker at /tmp/supabase-eval-runtime.json records channel, resolved version, staged docker state and session start for scorers' metrics.

Experiments (Codex GPT-5.6 Luna, suite: ['cli'])

Experiment	CLI installed	Docker	Selects evals by
codex-gpt-5.6-luna (existing; cli added to its suites)	pinned	available	baseline column for CLI evals
-cli-stable	latest stable	available	interface: cli, not hosted
-cli-beta	latest beta (cut per merge to develop)	available	same
-cli-nodaemon	latest beta	client present, daemon unreachable	needsDocker: false
-cli-absent	latest beta	no docker binary	needsDocker: false

Docker-less arms use beta because the CLI's Docker-less path only exists in the managed stack, which ships there.

Review follow-ups (second commit)

Two independent reviews plus a triage of the first CI run of the cli suite (run 35274398896, via #307) led to:

• CI root cause — npm/GitHub release skew. npm's beta dist-tag pointed at 2.118.0-beta.52 while its GitHub release was still a draft, so the .deb download 404'd and every beta-channel pair errored before the agent started (stable passed for exactly that reason). The resolver now HEAD-checks the release asset; for beta it walks back to the newest published -beta.N, for stable it throws — never falls back to the pin. The workflow resolves both channels once in prepare (overridable via new cli_stable_version / cli_beta_version dispatch inputs) and forwards the pins into the Vercel sandbox, so one nightly never mixes CLI versions across pairs. Each session logs the version it installed.
• Structurally Docker-less. DockerSandboxOptions.mountDockerSocket (default true, 3 lines in packages/sandbox); the Docker-less arms create their sandbox without the socket and assert it's absent and that no docker resolves on the agent's PATH. DOCKER_HOST + shims stay as defence in depth.
• CI gates the new code. apps/framework/tsconfig.json's experiments include pointed at a nonexistent directory (experiments were never typechecked); fixed, and pnpm check now runs the _lib and evals/cli unit tests (test:cli-lib).
• Shared skipUnlessCli / skipUnlessDockerless predicates (mis-tagged evals skip instead of erroring), root-owned read-only runtime marker, single-quote shell escaping in the shims, needsDocker parse tests, CODEOWNERS for experiments/_lib/ and the arms, CONTRIBUTING spells out which evals each arm picks up.

Questions for the framework team, deliberately left as-is: needsDocker: boolean vs a requires: { docker } object (kanad's #307 already uses the boolean); moving the suite→experiment-suite mapping out of the workflow case; session facts (runtime id, resolved CLI version) on the scoring context to replace the marker file.

Related Issue(s)

Refs https://linear.app/supabase/issue/CLI-2398/add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle (first eval: #281, stacked on this)
Context: #team-ai feedback thread → Linear project "Evals Driven Development".

Test plan

• pnpm check and pnpm format:check green
• Unit tests for the runtime helpers (version resolver incl. memoisation/rejection, shim scripts, runtime ids): pnpm --filter @supabase-evals/framework exec vitest run --root ../.. experiments/_lib — these paths are outside every package test script, so pnpm check does not run them
• tsc --strict over experiments/ (not covered by pnpm check: apps/framework/tsconfig.json's experiments include resolves to a nonexistent dir — worth a one-line fix separately)
• pnpm eval:dry for regression × regression and benchmark × benchmark unchanged; cli × cli has no evals until feat(evals): add build-database-002-stack-lifecycle CLI eval (CLI-2398) #281
• Both workflow files parse
• Sandbox staging verified without an agent for all three docker states (shims hold after unset DOCKER_HOST; real supabase start errors surface; teardown leaves nothing behind), and with real agent runs via feat(evals): add build-database-002-stack-lifecycle CLI eval (CLI-2398) #281
• First nightly / dispatch after feat(evals): add build-database-002-stack-lifecycle CLI eval (CLI-2398) #281 lands: suite=cli, experiment_suite=cli

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Sep 21, 2026 10:57am UTC

[Coly010]

Closing in favour of the split agreed in the thread with @mattrossman and @Rodriguespn — this PR bundled sandbox plumbing, framework schema, CI wiring and CLI-team experiments into one 1365-line diff that couldn't be reviewed without CLI domain knowledge. It's now four PRs, each reviewable on its own:

#324	main	CLI channel tags (cliVersion: 'stable' | 'beta') into @supabase-evals/sandbox, plus the per-run version pins
#325	#324	Docker-less staging (docker: 'no-daemon' | 'absent'), mountDockerSocket, needsDocker
#326	#315	typecheck experiments/ (it never has been)
next	main after #315	the cli eval + experiment suites, CODEOWNERS, workflow wiring, and experiments/cli/

The substantive change versus what was here: dockerAwareLocalStackRuntime and experiments/_lib/ are gone entirely. Resolving a CLI channel and controlling Docker availability are now options on the shared localStackRuntime, per @mattrossman's suggestion, so they're available to everyone rather than living in CLI-team experiment land. What stays CLI-owned is just skipUnlessCli / skipUnlessDockerless and the experiment files themselves, which are ~6 lines each now that experiments/presets.ts exists.

Not deleting columferry/cli-2398-cli-suite-and-arms — #281, #314 and #316 are still based on it. They'll be re-parented once the cli suite PR lands, and #307 follows #281. /cc @kanadgupta