Repository navigation
Build images from git repositories in the gateway - #1923
Conversation
The image service accepts a git source: the gateway pins the ref to a commit (the image id keys on it, never on the token), the build worker clones the repository as the build context and builds its Dockerfile, or the one nixpacks generates when there is none. railway.json/railway.toml start, build and dockerfilePath settings are honored so repositories written for Railway deploy unchanged. Tokens travel via GIT_ASKPASS and are stripped from private-worker requests. Bumps clip to the hard-link fix so nixpacks images resolve libz. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
A repository without a Dockerfile gets one rendered by the worker from its stack (Python, Node, Go, static) on the official image for the version it declares, honouring railway.json/Procfile start and build commands. No third-party binary in the worker image (only git), no ghcr.io base image at build time; hello-world builds drop from ~100s to ~15-30s. Fixes a nil cleanup call on prepare failures. Co-authored-by: Cursor <cursoragent@cursor.com>
Restore returns on the first state with a pid; the fake reported one immediately, so on a slow runner the assertion read the bundle before the fake restore had copied it. Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 584ea19)
jackf-beamcloud
left a comment
There was a problem hiding this comment.
Symlinks in the cloned repo aren't handled, and the checkout is attacker-controlled. readRepoFile's Clean("/"+rel) stops ../ but os.ReadFile and os.Stat follow links, so:
- working_dir pointing at a committed symlink to / passes the IsDir check and becomes the buildah context, so COPY . . pulls the worker host's filesystem into an image the user then runs.
- A Dockerfile, dockerfile_path or railway.json/Procfile that is a symlink (e.g. to /proc/self/environ) gets read on the worker host; Dockerfile parse errors get echoed into the user's build logs.
Suggest resolving with filepath.EvalSymlinks and requiring the result to stay under the checkout root (or os.Root / Lstat and reject links) for contextDir and every readRepoFile.
Non-blocking: on the fetch-by-commit path (the usual GitHub one), checkout --recurse-submodules in a fresh init won't populate submodules, so you'd need a git submodule update --init --depth 1. The gateway also runs ls-remote against any https URL, which is a small SSRF surface toward internal hosts. And the proto/swagger comments still say nixpacks, but the Dockerfile is rendered in-tree now.
@jackf-beamcloud can you fix both of these surgically? > 1. working_dir pointing at a committed symlink to / passes the IsDir check and becomes the buildah context, so COPY . . pulls the worker host's filesystem into an image the user then runs.
|
working_dir and every file read from the clone (Dockerfile, dockerfile_path, railway.json/toml, Procfile, ...) now resolve symlinks and must stay under the checkout root, so a committed link to / or /proc/self/environ can't become the build context or be read on the worker host.
|
Fixed both in 5070a7e. |
@jackf-beamcloud Looks good, lets approve. seems ready to merge. |
jackf-beamcloud
left a comment
There was a problem hiding this comment.
Approving. Symlink escapes are closed (5070a7e) and tests pass. The non-blocking notes from my earlier review (submodules on the fetch-by-commit path, ls-remote SSRF surface, stale nixpacks comments) can go in a follow-up.
What
The image service accepts a git source (
BuildImageRequest.git_source/VerifyImageBuildRequest.git_source):{repo_url, ref, token, working_dir, dockerfile_path, start_command, build_command}.Gateway pins the ref to a commit with
git ls-remoteup front, so a bad repo or ref fails at verify time with a precise message (repository … not found or not accessible,ref "x" not found in …). The image id hashes{clip version, repo, commit, working dir, dockerfile path, start/build command}, never the token or runtime env, so the same commit is the same image and a new push is a new one.Build worker clones the commit (GIT_ASKPASS for the token, nothing in argv or
.git/config) onto the disk-backed spool dir and builds, in Railway's resolution order:dockerfile_path(request orrailway.json/railway.tomlbuild.dockerfilePath), elseDockerfile, elsepkg/worker/git_dockerfile.go) from the detected stack: Python (requirements.txt/pyproject.toml; Django/FastAPI/Flask server detection), Node (lockfile → npm/yarn/pnpm,build/startscripts,engines.node), Go (multi-stage on thegodirective's version), static (index.htmlon nginx). Official images only; start/build commands from the request, railway config or Procfile win; gunicorn gets--bind 0.0.0.0:$PORT. Anything else:repository has no Dockerfile and no recognised stack; add a Dockerfile.The checkout is the buildah context, so the repository's own multi-stage Dockerfiles work unchanged.
docker/Dockerfile.worker: addsgit. Nothing else.ContainerRequest.BuildOptions.GitSourcetravels via proto;IsBuildRequestrecognises it;PrivateWorkerRequeststrips the token.Bumps clip to the hard-link fix (
1c772eb, already on MCP fixes from staging e2e, clip hard-link fix #1921).This is what lets internal-api drop its sandbox builder (beam-cloud/internal-api#309): the deploy agent calls
/images/buildwith agitSource, then/stubs+/stubs/deploy.Verified locally (k3d)
Through internal-api's deploy path against the local gateway, then HTTP to the deployed pod:
python:3.11-slim,railway.jsonstartCommand (hypercorn)node:20-slim, yarn +tscbuildpython:3.11-slim, gunicorn detected fromFlask(repository … not found or not accessibleat verifynoperef "nope" not found in …at verifyworking_dir: xworking directory "x" not found in repositoryTests: image-id keying, request normalisation, Dockerfile resolution order (explicit → railway config → repo → rendered, path escape, TOML quoting), rendered Dockerfiles for each stack, a real
gitcheckout by commit with branch fallback and error cleanup, proto round trip incl. token stripping.Not in this PR:
CAP_NET_BIND_SERVICEfor services binding :80 is #1922.