feat(sandbox): stage Docker-less local stacks via a docker option

[Coly010]

Stacked on #324 — review that one first. Second of two PRs splitting #308's sandbox work out of CLI-team experiment land.

This is @mattrossman's suggestion from the thread taken literally: dockerAwareLocalStackRuntime dissolves into localStackRuntime, and being able to control Docker availability becomes something any experiment can ask for.

localStack: localStackRuntime({ cliVersion: 'beta', docker: 'absent' })

docker defaults to 'available', which is the existing code path. The two new states stage a sandbox where the daemon is unreachable ('no-daemon') or the binary is gone entirely ('absent').

What staging actually involves

None of this is decorative — each step exists because a weaker version of it let a "Docker-less" sandbox quietly still have Docker:

• The socket is not bind-mounted at all (mountDockerSocket: false on DockerSandbox), rather than hidden or blocked from inside the container, and its absence is asserted. CI's sandbox setup ends with chmod 666 /var/run/docker.sock, so anything short of omitting the mount is a no-op there.
• docker/dockerd/docker-proxy removal asserts its postcondition instead of trusting rm -f, which always exits 0 — and checks against both the root shell's PATH and the agent's own SANDBOX_PATH, which differ.
• For 'no-daemon', docker --version is captured before removal so the shim can keep echoing it. The CLI's managed-stack runtime probe (fix(cli): improve stack startup defaults and clean managed state cli#6563) reads that to choose Docker; if the shim didn't answer, the CLI would pick a different runtime and we would be testing the wrong thing.
• DOCKER_HOST points at tcp://127.0.0.1:1, which is never bound. Deliberately not 2375, which Docker Desktop can legitimately expose.

There are guard errors for projectRunning !== false and for a hosted link, since the harness can do neither without Docker.

The supabase shim looks CLI-specific but isn't

It injects -x <excluded services> into supabase start. On the normal path the harness runs supabase start and applies exclusions via buildSupabaseStartCommand; on a Docker-less path the agent runs it, so the shim is the only route by which the eval-level includeServices field is honoured at all. computeExcludedServices already lived in this package.

The environment marker

Each session writes /tmp/supabase-eval-runtime.json recording what was staged, exported now as LOCAL_STACK_MARKER_PATH / LocalStackEnvironmentMarker so scorers can import it instead of duplicating the schema. Scorers are meant to report it, never branch on docker to decide pass/fail — that would grade an eval against its own environment.

Two asymmetries worth knowing:

• On the Docker-less paths it is root-owned and 0444, so the agent cannot rewrite it to fake the environment it is being graded in, and a write failure is fatal — there the marker is the evidence the environment was staged as claimed. On the 'available' path it goes through the session's own non-root exec, so it is agent-writable and metrics-only, and a write failure only warns. That path is the one every experiment in the repo takes, and a transient /tmp failure shouldn't kill a benchmark run over a marker nothing reads.
• channel is optional: populated only when the runtime option named a channel and no per-eval cliVersion: pin overrode it. An exact-version pin records cliVersion with no channel.

Also here

needsDocker eval frontmatter (defaults to true), the eval-side counterpart of this option — it is how a Docker-less experiment decides which evals it can legitimately pick up.

What is deliberately not here

skipUnlessCli / skipUnlessDockerless stay CLI-team-owned and land with the cli suite, along with CODEOWNERS and the workflow wiring, once #315 is in.

Verification

pnpm format:check, pnpm typecheck, and the sandbox (89), core (162), framework (8), vercel-runner (28) and web (83) suites pass. test:docker needs a real daemon and free ports, so it was not run; pnpm check's test:framework stage needs AI_GATEWAY_API_KEY and is not run in CI.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored	Preview	Sep 23, 2026 9:34am UTC

[Rodriguespn]

Thanks for putting this together. A few things are still worth attention before calling this done:

1. After docker cp, have writeRootFile() run chown root:root and apply the requested mode in one checked root command.
2. Pass the Docker state into the prompt and tool builders so each state gets the right text.
3. Give the scorer the parsed marker or a direct container read. It should not depend on the shell or PATH.

After fixing these, please rerun the focused tests and the real-container ownership check.
You can run the Docker check with SANDBOX_DOCKER_TESTS=1 pnpm --filter @supabase-evals/sandbox test:docker.

Add the test case to test/docker.test.ts with localStackRuntime({ docker: 'no-daemon' }). Assert that stat -c "%u:%g %a" /tmp/supabase-eval-runtime.json /usr/local/sbin/supabase /usr/local/sbin/docker returns 0:0 444 for the marker and 0:0 755 for both shims, then confirm a write as node fails with permission denied. Run the prompt, tool, and scorer cases with pnpm --filter @supabase-evals/sandbox exec vitest run test/local-stack-docker.test.ts test/unit.test.ts.

[Rodriguespn]

Thanks for the update. I ran this locally: the focused tests, sandbox typecheck, and targeted real-container ownership check pass. The branch is two commits behind main but merges cleanly.

Can we update it from main and add the PATH-shadow regression by placing a fake cat first on PATH and asserting environmentMarker() still returns no-daemon?

Approving now to unblock you but e should validate that the CI is green. Thx!