Skip to content

Build images from git repositories in the gateway - #1923

Merged
luke-lombardi merged 5 commits into
mainfrom
ll/build-from-git
Sep 24, 2026
Merged

luke-lombardi merged 5 commits into
mainfrom
ll/build-from-git

Conversation

@luke-lombardi

@luke-lombardi luke-lombardi commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

What

The image service accepts a git source (BuildImageRequest.git_source / VerifyImageBuildRequest.git_source): {repo_url, ref, token, working_dir, dockerfile_path, start_command, build_command}.

  • Gateway pins the ref to a commit with git ls-remote up front, so a bad repo or ref fails at verify time with a precise message (repository … not found or not accessible, ref "x" not found in …). The image id hashes {clip version, repo, commit, working dir, dockerfile path, start/build command}, never the token or runtime env, so the same commit is the same image and a new push is a new one.

  • Build worker clones the commit (GIT_ASKPASS for the token, nothing in argv or .git/config) onto the disk-backed spool dir and builds, in Railway's resolution order:

    1. dockerfile_path (request or railway.json/railway.toml build.dockerfilePath), else
    2. the repository's Dockerfile, else
    3. a Dockerfile rendered in-tree (pkg/worker/git_dockerfile.go) from the detected stack: Python (requirements.txt/pyproject.toml; Django/FastAPI/Flask server detection), Node (lockfile → npm/yarn/pnpm, build/start scripts, engines.node), Go (multi-stage on the go directive's version), static (index.html on nginx). Official images only; start/build commands from the request, railway config or Procfile win; gunicorn gets --bind 0.0.0.0:$PORT. Anything else: repository has no Dockerfile and no recognised stack; add a Dockerfile.

    The checkout is the buildah context, so the repository's own multi-stage Dockerfiles work unchanged.

  • docker/Dockerfile.worker: adds git. Nothing else.

  • ContainerRequest.BuildOptions.GitSource travels via proto; IsBuildRequest recognises it; PrivateWorkerRequest strips the token.

  • Bumps clip to the hard-link fix (1c772eb, already on MCP fixes from staging e2e, clip hard-link fix #1921).

This is what lets internal-api drop its sandbox builder (beam-cloud/internal-api#309): the deploy agent calls /images/build with a gitSource, then /stubs + /stubs/deploy.

Verified locally (k3d)

Through internal-api's deploy path against the local gateway, then HTTP to the deployed pod:

repo path build result
traefik/whoami repo Dockerfile (multi-stage) 72s image built
railwayapp-templates/fastapi rendered python:3.11-slim, railway.json startCommand (hypercorn) 22s 200
railwayapp-templates/expressjs rendered node:20-slim, yarn + tsc build 32s 200
railwayapp-templates/flask rendered python:3.11-slim, gunicorn detected from Flask( 12s 200
beam-cloud/does-not-exist — — repository … not found or not accessible at verify
expressjs @ nope — — ref "nope" not found in … at verify
flask, working_dir: x — — working directory "x" not found in repository

Tests: image-id keying, request normalisation, Dockerfile resolution order (explicit → railway config → repo → rendered, path escape, TOML quoting), rendered Dockerfiles for each stack, a real git checkout by commit with branch fallback and error cleanup, proto round trip incl. token stripping.

Not in this PR: CAP_NET_BIND_SERVICE for services binding :80 is #1922.

luke-lombardi and others added 4 commits September 24, 2026 08:53
The image service accepts a git source: the gateway pins the ref to a
commit (the image id keys on it, never on the token), the build worker
clones the repository as the build context and builds its Dockerfile, or
the one nixpacks generates when there is none. railway.json/railway.toml
start, build and dockerfilePath settings are honored so repositories
written for Railway deploy unchanged. Tokens travel via GIT_ASKPASS and
are stripped from private-worker requests.

Bumps clip to the hard-link fix so nixpacks images resolve libz.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
A repository without a Dockerfile gets one rendered by the worker from
its stack (Python, Node, Go, static) on the official image for the
version it declares, honouring railway.json/Procfile start and build
commands. No third-party binary in the worker image (only git), no
ghcr.io base image at build time; hello-world builds drop from ~100s to
~15-30s. Fixes a nil cleanup call on prepare failures.

Co-authored-by: Cursor <cursoragent@cursor.com>
Restore returns on the first state with a pid; the fake reported one
immediately, so on a slow runner the assertion read the bundle before the
fake restore had copied it.

Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit 584ea19)

@jackf-beamcloud jackf-beamcloud left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Symlinks in the cloned repo aren't handled, and the checkout is attacker-controlled. readRepoFile's Clean("/"+rel) stops ../ but os.ReadFile and os.Stat follow links, so:

  1. working_dir pointing at a committed symlink to / passes the IsDir check and becomes the buildah context, so COPY . . pulls the worker host's filesystem into an image the user then runs.
  2. A Dockerfile, dockerfile_path or railway.json/Procfile that is a symlink (e.g. to /proc/self/environ) gets read on the worker host; Dockerfile parse errors get echoed into the user's build logs.
    Suggest resolving with filepath.EvalSymlinks and requiring the result to stay under the checkout root (or os.Root / Lstat and reject links) for contextDir and every readRepoFile.
    Non-blocking: on the fetch-by-commit path (the usual GitHub one), checkout --recurse-submodules in a fresh init won't populate submodules, so you'd need a git submodule update --init --depth 1. The gateway also runs ls-remote against any https URL, which is a small SSRF surface toward internal hosts. And the proto/swagger comments still say nixpacks, but the Dockerfile is rendered in-tree now.

@luke-lombardi

Copy link
Copy Markdown
Contributor Author

Symlinks in the cloned repo aren't handled, and the checkout is attacker-controlled. readRepoFile's Clean("/"+rel) stops ../ but os.ReadFile and os.Stat follow links, so:

  1. working_dir pointing at a committed symlink to / passes the IsDir check and becomes the buildah context, so COPY . . pulls the worker host's filesystem into an image the user then runs.
  2. A Dockerfile, dockerfile_path or railway.json/Procfile that is a symlink (e.g. to /proc/self/environ) gets read on the worker host; Dockerfile parse errors get echoed into the user's build logs.
    Suggest resolving with filepath.EvalSymlinks and requiring the result to stay under the checkout root (or os.Root / Lstat and reject links) for contextDir and every readRepoFile.
    Non-blocking: on the fetch-by-commit path (the usual GitHub one), checkout --recurse-submodules in a fresh init won't populate submodules, so you'd need a git submodule update --init --depth 1. The gateway also runs ls-remote against any https URL, which is a small SSRF surface toward internal hosts. And the proto/swagger comments still say nixpacks, but the Dockerfile is rendered in-tree now.

@jackf-beamcloud can you fix both of these surgically? > 1. working_dir pointing at a committed symlink to / passes the IsDir check and becomes the buildah context, so COPY . . pulls the worker host's filesystem into an image the user then runs.

  1. A Dockerfile, dockerfile_path or railway.json/Procfile that is a symlink (e.g. to /proc/self/environ) gets read on the worker host; Dockerfile parse errors get echoed into the user's build logs.

working_dir and every file read from the clone (Dockerfile, dockerfile_path, railway.json/toml, Procfile, ...) now resolve symlinks and must stay under the checkout root, so a committed link to / or /proc/self/environ can't become the build context or be read on the worker host.
@jackf-beamcloud

Copy link
Copy Markdown
Contributor

Fixed both in 5070a7e. working_dir and every file read from the checkout (Dockerfile, dockerfile_path, railway.json/toml, Procfile, package.json, etc.) now go through repoPath, which runs EvalSymlinks and refuses anything that resolves outside the checkout root. Links that stay inside the repo still work. Added TestRepoFilesCannotEscapeThroughSymlinks; go test ./pkg/worker/ passes.

@luke-lombardi

Copy link
Copy Markdown
Contributor Author

Fixed both in 5070a7e. working_dir and every file read from the checkout (Dockerfile, dockerfile_path, railway.json/toml, Procfile, package.json, etc.) now go through repoPath, which runs EvalSymlinks and refuses anything that resolves outside the checkout root. Links that stay inside the repo still work. Added TestRepoFilesCannotEscapeThroughSymlinks; go test ./pkg/worker/ passes.

@jackf-beamcloud Looks good, lets approve. seems ready to merge.

@jackf-beamcloud jackf-beamcloud left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Symlink escapes are closed (5070a7e) and tests pass. The non-blocking notes from my earlier review (submodules on the fetch-by-commit path, ls-remote SSRF surface, stale nixpacks comments) can go in a follow-up.

@luke-lombardi
luke-lombardi merged commit 8532063 into main Sep 24, 2026
4 checks passed
@luke-lombardi
luke-lombardi deleted the ll/build-from-git branch September 24, 2026 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants