[jeremy]

Problem

A release is made by pushing any v* tag. Nothing is signed, and checksums.txt is written by the same job that builds the binaries, so it can't vouch for them. main has no protection and releases aren't immutable.

get.once.com and the self-updater install whatever GitHub reports as the latest release without checking it. The updater runs as root every 24 hours and accepts any tag that differs from its own, including older ones. So anything able to push to this repository can reach every ONCE install.

There's also an accident in the same class: make test-release force-pushes v0.0.1-test, which runs the real release workflow and produces a normal, non-prerelease release.

Design

Each release now carries release.txt and release.txt.sig:

once v0.4.0
commit <40-hex sha>
<sha256>  once-darwin-amd64
<sha256>  once-darwin-arm64
<sha256>  once-linux-amd64
<sha256>  once-linux-arm64

release.txt.sig is an ECDSA P-256 / SHA-256 signature (DER) made by openssl dgst -sha256 -sign. That format verifies with openssl on any target machine and with the Go standard library, so no new dependency is added. The version line binds the signature to one release, so an older signed manifest can't be served as a newer one.

• .github/workflows/release.yml runs only on workflow_dispatch on main, with a version input and an optional draft input. Each job holds one privilege:
  • gate (read-only) checks for main, a well-formed version, that no tag or release (draft or not) of that name exists, and that the version is newer than the latest release.
  • ci is the existing CI.
  • build (read-only, no secrets, no Go cache) runs make dist VERSION=….
  • sign runs in the release environment, so it is the only job that can read ONCE_RELEASE_SIGNING_KEY. It runs no repository code and checks out only release-key.pub. It checks the files against release.txt and checks the version and commit lines. Then it signs, and verifies the signature against the committed public key, so a mismatched secret fails before anything is published.
  • publish (contents: write, no key, no checkout) uploads to a draft at --target $GITHUB_SHA and reads the draft back to compare digests. It then publishes the draft as --latest, which creates the tag at the built commit, and confirms where the tag points. With draft: true it stops at the draft.
• Self-updater (internal/version):
  • verify.go and release_keys.go verify release.txt.sig against the built-in keys. Any one of them is enough, so a key can be rotated.
  • It requires release.txt to name the release's tag, and refuses a release without the two files.
  • It hashes the binary as it downloads and compares the hash with the manifest before replaceBinary. On a mismatch it removes the temp file and leaves the running binary untouched.
  • It moves only forward, and only to a full vX.Y.Z (isNewerRelease replaces the !=). A dev build never self-updates.
  • Every failure goes through the existing RecordSelfUpdate(err).
• installer/templates/install.sh downloads release.txt and .sig and verifies them with openssl against the keys pinned in RELEASE_KEYS. It checks the version line and the binary's hash, and exits before sudo install (and before once background install) on any failure. If openssl is missing, it says so and stops. There is no skip flag.
• Makefile:
  • dist writes release.txt, and still writes checksums.txt for anyone using it.
  • test-release now dispatches a signed draft (v0.0.1-test, draft only). A draft is never tagged, never latest, and never published.
  • make test now also runs the installer module's tests. CI didn't run them before.
• release-key.pub is a placeholder and is clearly marked as one. Tests hold release_keys.go and the install.sh copy byte-for-byte equal to it. While it is a placeholder, every self-update and every install refuses to proceed, so this must not be merged or deployed until the real key is in.

Tests

• Updater:
  • verified update
  • update signed by a second key
  • tampered binary (not replaced, no temp left)
  • untrusted key
  • edited manifest
  • unsigned release
  • manifest signed for another version
  • no keys built in
  • older, prerelease, unversioned and dev cases (no assets fetched)
  • an openssl-made signature verified by Go
  • embedded keys match release-key.pub
• Installer (runs the served script's real install_once, with downloads read from a fixture and sudo recorded):
  • verified install
  • second key
  • tampered binary
  • untrusted key
  • edited manifest
  • replayed older manifest
  • unsigned release
  • placeholder keys
  • missing openssl
  • In every refusal, no sudo is reached.
• Mutation check: disabling any one of the signature, version-binding, hash, newer-than or prerelease checks (Go), or the verify, version-line, hash or openssl checks (script), fails the suite.
• I ran make dist VERSION=v9.9.9, then the sign job's own steps with a throwaway key, then the rendered installer against the result. It verified, and refused a tampered binary and a mismatched tag.

Setup for the owners (settings, not in this PR)

1. Create the release environment:
   • required reviewers
   • prevent self-review
   • administrators can't bypass
   • deployment branches: main only, no tag patterns
2. Generate the key pair on a trusted machine:

   openssl ecparam -name prime256v1 -genkey -noout -out once-release-key.pem
   openssl pkey -in once-release-key.pem -pubout -out release-key.pub
   gh secret set ONCE_RELEASE_SIGNING_KEY --env release --repo basecamp/once < once-release-key.pem

   • Keep an offline copy of the private key in the team's password manager for recovery and rotation, then delete the local file.
   • The private key must exist nowhere else: not as a repository or organization secret, and not in any other environment.
3. Commit the public key. Replace release-key.pub with it, and paste the identical text into releaseKeys in internal/version/release_keys.go and RELEASE_KEYS in installer/templates/install.sh. make test fails until all three match.
4. Turn on immutable releases for this repository.
5. Make main PR-only with required review and no bypass. The signature attests "main's workflow built main's commit after a reviewer approved", so it's only as strong as main.

Migration

Binaries up to and including v0.3.3 can't be retrofitted. They take one last unverified update, and every update after that is verified.

1. Do the setup above, then merge this. Don't deploy the installer yet. It would refuse every install until a signed release exists.
2. Dispatch v0.4.0. Check it by hand with openssl dgst -sha256 -verify release-key.pub -signature release.txt.sig release.txt and sha256sum. Asset names are unchanged, so existing installs take v0.4.0 on their next daily check. From then on they verify.
3. kamal deploy the installer from a main checkout, so new installs verify from then on.
4. Installs with ONCE_NO_SELF_UPDATE set, or that are offline, stay on the old path until they run once self-update once. The release notes for v0.4.0 should ask them to.

Rotation: add the next public key to all three places alongside the current one, and ship a release. Then switch the environment secret to the next key. In a later release, drop the old public key.

What this PR does not do

• It changes no settings: no environment, key, ruleset, or immutable-releases change.
• It cuts no release and deploys no installer.
• It doesn't help installs already on ≤ v0.3.3 until they take v0.4.0.
• It doesn't verify application images (ghcr.io/basecamp/* pulled by tag) or the basecamp/kamal-proxy image.
• It doesn't change the installer's own trust root (TLS to get.once.com plus whoever deploys it) or the get.docker.com step.
• It doesn't skip the installer's "already installed" short-circuit. Re-running the installer doesn't upgrade an existing install.
• It doesn't add a scheduled check that the latest release verifies. That's a good follow-up for security.yml.
• Releases can still be created by other means (for example, a workflow using GITHUB_TOKEN). They now install nowhere, but they would make new installs fail until removed. A tag ruleset, or read-only default workflow permissions, closes that. Those are settings too.

Unrelated commit

Use strings.CutLast in cutLast fixes a golangci-lint v2.14 modernize finding in internal/docker/helpers.go. That finding fails CI's lint step on every open PR today, including #116. It's here only so this PR's tests run in CI. Drop it if it lands separately.