feat(evals): add build-cli-004-worktree-stacks (CLI-2400)

[kanadgupta]

Tracks CLI-2400 under the Slim CLI evals RFC. Supersedes #295.

Note

Stacked on #281 and uses its cli suite and needsDocker field. The diff shows Colum's commits until #281 lands; this PR's own change is the single evals/cli/build-cli-004-worktree-stacks/ folder.

The problem it's solving

The headline agent workflow for the Slim CLI launch is one human plus N coding agents, each working in its own git worktree, each needing its own isolated local Supabase stack. Nothing in this repo measured whether an agent can get there: three worktrees, three stacks, zero leakage.

What the PR adds

One new eval in the cli suite, evals/cli/build-cli-004-worktree-stacks/. Starting from an empty sandbox, the agent is asked to set up a repo with worktrees feature-a, feature-b, feature-c, start a local stack in each, and add a different table (widgets / gadgets / gizmos) plus one sample row per worktree. No seed data, no framework changes, no experiment edits: needsDocker: false puts it on the Docker-less arms, the interface rule puts it on the stable and beta arms, and the cli suite puts it on the pinned Codex Luna baseline.

A scorer that grades the end state, never the method. Because the harness's built-in ctx.query / stackStatus resolve a single stack from the workspace root, every check addresses stacks itself by cd-ing into each worktree:

• three real git worktrees on distinct branches, discovered by finding a repo in the workspace and asking git worktree list --porcelain (so worktrees placed outside the workspace still count; three plain directories don't);
• one live stack per worktree with three distinct database endpoints, which is what catches aliasing or reuse;
• per-table schema isolation via to_regclass against all three stacks;
• at least one row per table in its home stack;
• each table created by a migration file in its worktree;
• an always-passing metrics check reporting fleet wall-clock across the three stacks, CLI version and channel, per-stack backend and runtime, and the number of supabase start invocations.

Stack resolution asks the managed backend first (SUPABASE_EXPERIMENTAL_STACK=1 supabase stack status --env --output-format json) and falls back to the legacy one, the same shape #281 landed on. Environment-agnostic: the scorer never branches on which arm it runs under.

Deliberate choices

• Nothing tells the agent about the managed stack or its feature flag. Worktree isolation with no config edits only exists on the managed stack (beta channel, gated behind experimental.stack). Whether agents can discover that is part of what's measured.
• services: is omitted, for the reason feat(evals): add build-database-002-stack-lifecycle CLI eval (CLI-2398) #281 documents: the sandbox shim's legacy -x gotrue,kong,... is rejected by the managed backend.
• Known footgun left in. supabase init pins ports in config.toml; the managed stack treats them as exact intents, so the second worktree's start fails with Persisted database port is unavailable until the ports are removed. That is the "no port surgery" promise the launch makes.

What the first CI run showed (#295, 18 runs across six experiments)

Two passes, both on the legacy backend, each with a hand-edited project_id and a full set of shifted ports per worktree. That is real signal about what agents do today, and the metrics check records the backend so it stays distinguishable from a managed-stack pass.

The 16 failures were two harness defects, both fixed here:

• 12 agents stopped within ~10 s to ask for a repository. The prompt said "set this repo up" against an empty workspace. It now says the project is brand-new, following the wording of build-database-002-stack-lifecycle, and asks for the worktrees as folders in this directory, following deploy-self-hosting-001's location hint.
• 2 agents built everything correctly with worktrees at /tmp/feature-* and the scorer's name-based search under the workspace missed them. Discovery now goes through git.

Verification

Verified by hand against CLI 2.118.0-beta.37 on the native runtime: three sibling worktrees get three distinct stack identities with no config edits, and a table created in one is absent from the others.

• Scorer unit tests (scoring.test.ts, 28 cases; fixtures captured from real beta CLI output)
• pnpm typecheck, Biome
• pnpm eval:dry -- --suite cli --experiment-suite cli → 5 experiments × 2 evals
• Real scorer executed against three hand-built native stacks on the beta CLI, with worktrees both inside and outside the workspace: all checks pass; stopping a stack or leaking a table fails the right checks
• Agent-backed runs via the run-evals-changed label on this PR (no Anthropic or OpenAI keys locally)

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Sep 24, 2026 5:36pm UTC

[Coly010]

Heads-up: I rebased this branch (kanad-claude/cli-2400-worktree-stacks-cli-suite) with --force-with-lease and it now carries only your two commits (f3d48c7 eval, 3158ffd CI results) on top of #281's current head.

Why: #281 was split into two PRs this morning — #308 (the cli suite, needsDocker, experiments/_lib runtime and the -cli-* experiments) and #281 (the stack-lifecycle eval only, stacked on #308) — which rewrote the history this branch was built on, so the PR had started showing my superseded commits in its diff. Nothing in your commits changed; pnpm check, the scorer tests for both CLI evals (90) and pnpm eval:dry -- --suite cli --experiment-suite cli (5 experiments × 2 evals) all pass on the rebased branch.

Nothing you need to do. Your eval is already in evals/cli/ with needsDocker: false, which is exactly how the Docker-less arms now select evals (skipUnlessDockerless in experiments/_lib), so it runs under all five arms without the experiment allow-list edits from #295. Merge order is #308 → #281 → this.

Two things from the review round on #308 that touch your runs, for awareness only:

• The first CI run of the cli suite (35274398896, the one that produced your cli-eval-results.json) lost every beta-channel pair because npm's beta tag pointed at 2.118.0-beta.52 while its GitHub release was still a draft (.deb 404). The resolver now verifies the asset and falls back to the newest published beta, and the workflow pins one version per run — so the next refresh should fill in the -cli-beta/-cli-nodaemon/-cli-absent rows.
• The Docker-less arms no longer mount the Docker socket at all (mountDockerSocket: false) in addition to the DOCKER_HOST + shim layering.