Skip to content

ci(docker): support arm64 builds and multi-arch manifest assembly #2132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nullPointerEnjoyer merged 4 commits into from
Oct 11, 2026
Merged

ci(docker): support arm64 builds and multi-arch manifest assembly #2132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
10473d0
ci(docker): support arm64 builds and multi-arch manifest assembly
erubboli Oct 6, 2026
430d64c
fix(ci): harden the multi-arch docker release workflow
nullPointerEnjoyer Oct 11, 2026
c7e9a81
Merge branch 'master' into ci-docker-multiarch
nullPointerEnjoyer Oct 11, 2026
e71a589
fix(ci): restrict docker dispatches and gate shared tag writes
nullPointerEnjoyer Oct 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
239 changes: 224 additions & 15 deletions .github/workflows/release_docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,68 @@ name: Build Docker

on:
workflow_call:
workflow_dispatch:
inputs:
version:
description: 'Version for image tags, e.g. 1.4.1 (defaults to the short commit SHA when empty)'
required: false
default: ''
platform:
description: 'Target CPU architecture (amd64 or arm64)'
required: false
default: 'amd64'
tag_suffix:
description: 'Suffix appended to every image tag, e.g. arm64 (leave empty for the default arch)'
required: false
default: ''
runner:
description: 'Runner label; only ubuntu-24.04 / ubuntu-24.04-arm are honored, anything else falls back to ubuntu-latest'
required: false
default: 'ubuntu-latest'
assemble_manifests:
description: "Skip image builds and only assemble multi-arch manifests from already-pushed '<tag>-amd64'/'<tag>-arm64' tags"
required: false
default: 'false'

# The workflow only checks out code and pushes images to Docker Hub via the
# DOCKERHUB_* secrets; it never needs more than read access to the repository.
permissions:
contents: read

jobs:
build-docker-images:
Comment thread
nullPointerEnjoyer marked this conversation as resolved.
runs-on: ubuntu-latest
# workflow_dispatch builds may push shared mutable tags ('<version>',
# 'v<version>', 'latest') with the repo's Docker Hub credentials, so they
# are restricted to protected refs; feature-branch dispatches cannot push.
# Tag refs are allowed at the same trust level as the release pipeline.
if: github.event_name == 'workflow_call' || (inputs.assemble_manifests != 'true' && (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/heads/release-') || startsWith(github.ref, 'refs/tags/v')))
permissions:
contents: read
# Explicit bound below the 360-minute GitHub default, with headroom for
# slow native arm64 builds; a stuck build releases the runner in 4h.
timeout-minutes: 240
# Serialize ALL registry mutations (builds and assemblies mutate the same
# tag space) behind one group. GitHub keeps at most one in-progress and
# one pending run per group; a newer run cancels an older PENDING run, so
# an overlapping dispatch can drop a queued run — including a release
# build from release.yml, and the per-arch flow itself (with amd64
# running, queueing arm64 makes it PENDING, and queueing the assembly
# then cancels it). Wait for each queued run to finish before dispatching
# the next, re-dispatch anything canceled, and re-run release.yml if a
# release run fails this way; in-progress runs are never canceled.
concurrency:
group: docker-release
cancel-in-progress: false
# Allowlist the runner label: a free-form `runs-on:` would let anyone with
# workflow-dispatch rights route this build (and its Docker Hub secrets)
# to an arbitrary, possibly untrusted runner.
runs-on: ${{ (inputs.runner == 'ubuntu-24.04' || inputs.runner == 'ubuntu-24.04-arm') && inputs.runner || 'ubuntu-latest' }}
Comment thread
nullPointerEnjoyer marked this conversation as resolved.
steps:
- name: Checkout code
uses: actions/checkout@v5
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5

- name: Setup Python
uses: actions/setup-python@v6
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version-file: './build-tools/.python-version'

Expand All @@ -22,42 +74,199 @@ jobs:
# # Install any other dependencies your script might need

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3

- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Determine version
id: version
env:
INPUT_VERSION: ${{ inputs.version }}
run: |
if [[ $GITHUB_REF == refs/tags/* ]]; then
VERSION=${GITHUB_REF#refs/tags/}
validate() {
# Validate the grammar up front (the version flows into docker build/push
# command lines inside build.py), so a non-semver tag fails here with a
# clear message. The leading 'v' is allowed here; build.py strips it.
if ! echo "$VERSION" | grep -Eq '^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$'; then
echo "derived version '$VERSION' is not [v]X.Y.Z[-suffix]; tag a semver release" >&2
# command lines inside build.py), so a bad version fails here with a
# clear message. Whole-value match: grep would compare line by line
# and pass a multi-line value, which would later expand into several
# tag arguments. The leading 'v' is allowed here; build.py strips it.
if ! [[ "$1" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]]; then
echo "derived version '$1' is not [v]X.Y.Z[-suffix]; tag a semver release or pass a valid version input" >&2
exit 1
fi
echo "VERSION=$VERSION" >> $GITHUB_OUTPUT
}
if [[ $GITHUB_REF == refs/tags/* ]]; then
VERSION=${GITHUB_REF#refs/tags/}
validate "$VERSION"
STAMP_LATEST=true
elif [[ -n "$INPUT_VERSION" ]]; then
VERSION="$INPUT_VERSION"
validate "$VERSION"
STAMP_LATEST=true
Comment on lines +106 to +109

@github-actions github-actions Bot Oct 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security · medium
For a workflow_dispatch on a protected ref, the 'version' input is accepted as any semver string without being tied to the checked-out ref (unlike the tag path, which derives VERSION from GITHUB_REF). Since builds push '', 'v' and, with --latest, the shared 'latest' tag, an operator (or a compromised token with dispatch rights) can repoint already-released version tags with different content. Consider requiring, for dispatches, that the version matches the ref (tag dispatches) or at least warning/gating when the target tags already exist in the registry.

else
echo "VERSION=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
# Ad-hoc dispatch default: the short commit SHA. These builds keep
# their own tags and must not repoint the production 'latest' tag.
VERSION=$(git rev-parse --short HEAD)
STAMP_LATEST=false
fi
echo "VERSION=$VERSION" >> $GITHUB_OUTPUT
echo "stamp_latest=$STAMP_LATEST" >> $GITHUB_OUTPUT

# Note: the version (and the Docker Hub username) are passed through environment
# variables instead of direct ${{ }} interpolation into the run block, so that a
# crafted tag/ref/input cannot inject shell into the step.
- name: Build and Push Docker images
env:
VERSION: ${{ steps.version.outputs.VERSION }}
STAMP_LATEST: ${{ steps.version.outputs.stamp_latest }}
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
PLATFORM: ${{ inputs.platform || 'amd64' }}
TAG_SUFFIX: ${{ inputs.tag_suffix }}
run: |
python build-tools/docker/build.py --push --docker-hub-user "$DOCKERHUB_USERNAME" \
--version "$VERSION" --latest
# Validate the suffix up front: it flows into image tags on docker
# command lines, so anything outside the Docker tag grammar must fail
# here with a clear message instead of producing a broken (or
# hostile) tag. The bash regex matches the whole value (grep would
# match line by line and pass a multi-line suffix). build.py validates
# it again for standalone use.
if [[ -n "$TAG_SUFFIX" ]] && ! [[ "$TAG_SUFFIX" =~ ^[a-zA-Z0-9]([a-zA-Z0-9_.-]{0,127})?$ ]]; then
echo "invalid tag_suffix '$TAG_SUFFIX'; expected [a-zA-Z0-9][a-zA-Z0-9_.-]{0,127}" >&2
exit 1
fi
args=(--push --docker-hub-user "$DOCKERHUB_USERNAME" \
--version "$VERSION" --platform "$PLATFORM")
# 'latest' is stamped only for release versions; the classification
# (semver vs commit SHA) happened in 'Determine version', so this
# step re-derives no grammar.
if [[ "$STAMP_LATEST" == 'true' ]]; then
args+=(--latest)
fi
if [[ -n "$TAG_SUFFIX" ]]; then
args+=(--tag_suffix "$TAG_SUFFIX")
fi
Comment thread
nullPointerEnjoyer marked this conversation as resolved.
python build-tools/docker/build.py "${args[@]}"

env:
DOCKER_BUILDKIT: 1

# Registry-side assembly of multi-arch manifest lists from images that were
# pushed earlier by separate per-arch dispatches of the build job above
# ("--platform arm64 --tag_suffix arm64"). Nothing is rebuilt: "docker
# buildx imagetools create" only references already-pushed manifests. Runs
# from the default branch via "gh workflow run ... --ref <branch>": GitHub
# validates the workflow registration against the default branch but reads
# the file content from the dispatched ref.
assemble-manifests:
if: github.event_name == 'workflow_dispatch' && inputs.assemble_manifests == 'true' && (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/heads/release-') || startsWith(github.ref, 'refs/tags/v'))
permissions:
contents: read
timeout-minutes: 60
# Builds and assemblies mutate the same tag space, so both share the
# 'docker-release' group (see the build job for the pending-run caveat).
concurrency:
group: docker-release
cancel-in-progress: false
Comment thread
nullPointerEnjoyer marked this conversation as resolved.
runs-on: ubuntu-latest
Comment thread
nullPointerEnjoyer marked this conversation as resolved.
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Assemble and push multi-arch manifests
env:
INPUT_VERSION: ${{ inputs.version }}
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
run: |
set -euo pipefail

VERSION="${INPUT_VERSION#v}"
# Whole-value match: grep would compare line by line and pass a
# multi-line value, which would later expand into several tag
# arguments.
if ! [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]]; then
echo "invalid version '$INPUT_VERSION'; expected [v]X.Y.Z[-suffix]" >&2
exit 1
fi
Comment thread
nullPointerEnjoyer marked this conversation as resolved.

assemble() {
local repo="$1" tag="$2"
local amd64_ready=0
# Pre-flight: confirm every source manifest before mutating
# anything, so a missing image fails the job here instead of
# leaving this tag half-assembled (part of the per-arch tags
# created, the base tag already repointed).
docker buildx imagetools inspect "$repo:$tag-amd64" >/dev/null 2>&1 && amd64_ready=1
if [ "$amd64_ready" -eq 0 ] && ! docker buildx imagetools inspect "$repo:$tag" >/dev/null 2>&1; then
echo "neither $repo:$tag-amd64 nor $repo:$tag found; push the amd64 images first" >&2
exit 1
fi
if ! docker buildx imagetools inspect "$repo:$tag-arm64" >/dev/null 2>&1; then
echo "$repo:$tag-arm64 not found; push the arm64 images first" >&2
exit 1
fi
# Snapshot the existing (historically amd64-only) manifest under an
# explicit arch tag, unless it's already there (idempotent re-runs).
if [ "$amd64_ready" -eq 0 ]; then
docker buildx imagetools create -t "$repo:$tag-amd64" "$repo:$tag"
fi
# Provenance: the two sources were pushed by independent dispatches
# and can originate from different commits. Full cross-arch
# integrity needs image-level attestation; until then, log both
# digests so the pairing is auditable after the fact.
echo "amd64 digest of $repo:$tag-amd64:"
docker buildx imagetools inspect "$repo:$tag-amd64" | grep '^Digest:'
echo "arm64 digest of $repo:$tag-arm64:"
docker buildx imagetools inspect "$repo:$tag-arm64" | grep '^Digest:'
docker buildx imagetools create -t "$repo:$tag" "$repo:$tag-amd64" "$repo:$tag-arm64"
Comment thread
nullPointerEnjoyer marked this conversation as resolved.
Comment on lines +225 to +229

@github-actions github-actions Bot Oct 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security · high
The assemble() function combines whatever 'latest-amd64'/'latest-arm64' manifests currently exist in the registry into the shared production 'latest' and 'v' tags, without verifying they were built from the same commit/version. Two independently dispatched per-arch builds from different commits (as the inline provenance comment acknowledges) can be published as one multi-arch release. The digest logging is post-hoc audit only, not a gate. Consider recording the source digest (e.g. as a workflow output or build-time attestation/label) during each per-arch build and comparing them here before running the final 'imagetools create', or pinning 'create' to the exact digests captured during pre-flight.

Suggestion:

Suggested change
echo "amd64 digest of $repo:$tag-amd64:"
docker buildx imagetools inspect "$repo:$tag-amd64" | grep '^Digest:'
echo "arm64 digest of $repo:$tag-arm64:"
docker buildx imagetools inspect "$repo:$tag-arm64" | grep '^Digest:'
docker buildx imagetools create -t "$repo:$tag" "$repo:$tag-amd64" "$repo:$tag-arm64"
amd64_digest="$(docker buildx imagetools inspect "$repo:$tag-amd64" | grep '^Digest:' | awk '{print $2}')"
arm64_digest="$(docker buildx imagetools inspect "$repo:$tag-arm64" | grep '^Digest:' | awk '{print $2}')
# Compare against the digests recorded by the per-arch builds before repointing the shared tag.
check_digests_match "$repo" "$tag" "$amd64_digest" "$arm64_digest"
docker buildx imagetools create -t "$repo:$tag" "$repo:$tag-amd64@$amd64_digest" "$repo:$tag-arm64@$arm64_digest"

}

# Only the app images are maintained on Docker Hub (build.py's
# push_instances pushes these six); the builder/runner-base images
# stay local to the CI runner, so there is nothing to assemble for
# them.
for repo in "$DOCKERHUB_USERNAME/node-daemon" \
"$DOCKERHUB_USERNAME/api-blockchain-scanner-daemon" \
"$DOCKERHUB_USERNAME/api-web-server" \
"$DOCKERHUB_USERNAME/wallet-cli" \
"$DOCKERHUB_USERNAME/wallet-rpc-daemon" \
"$DOCKERHUB_USERNAME/dns-server"; do
for tag in "$VERSION" "v$VERSION" latest; do
assemble "$repo" "$tag"
done
done

- name: Verify manifests
env:
INPUT_VERSION: ${{ inputs.version }}
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
run: |
VERSION="${INPUT_VERSION#v}"
for repo in node-daemon api-blockchain-scanner-daemon api-web-server wallet-cli wallet-rpc-daemon dns-server; do
echo "=== $DOCKERHUB_USERNAME/$repo:v$VERSION ==="
# A missing manifest must fail the job: reporting success here would
# defeat the purpose of the check.
if ! output="$(docker buildx imagetools inspect "$DOCKERHUB_USERNAME/$repo:v$VERSION")"; then
echo "manifest for $DOCKERHUB_USERNAME/$repo:v$VERSION is missing" >&2
exit 1
fi
echo "$output" | grep -E '^Name:|Platform:'
# Both architectures must be present in the manifest list; a stale
# single-arch tag must fail the job instead of shipping silently.
for platform in linux/amd64 linux/arm64; do
# imagetools output is tabwriter-aligned ("Platform: linux/amd64"),
# so tolerate any run of whitespace after the colon.
if ! grep -Eq "Platform:[[:space:]]+$platform" <<<"$output"; then
echo "manifest for $DOCKERHUB_USERNAME/$repo:v$VERSION lacks $platform" >&2
exit 1
fi
done
done
9 changes: 9 additions & 0 deletions build-tools/docker/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,12 @@ docker run -it --network=mintlayer-net -v ~/.mintlayer:/root/.mintlayer mintlaye
replace `<NETWORK>` with `mainnet` or `testnet` depending on what network you're running and `<IP_ADDRESS>` with the result of the command above.

This command mounts the same `~/.mintlayer` directory as a volume in the `wallet-cli` container and uses the `--rpc-cookie-file` option to specify the path to the cookie file.

## Multi-arch (amd64 / arm64) images

By default the build produces `linux/amd64` images, exactly as it always has. Two extra knobs exist for building other architectures:

- `build.py --platform arm64` builds `linux/arm64` images (must run on an arm64 machine, e.g. the `ubuntu-24.04-arm` GitHub runner, since the compilation happens inside the container).
- `build.py --tag_suffix arm64` appends the suffix to every built and pushed tag (e.g. `node-daemon:1.4.1-arm64`), so a non-default architecture never overwrites the default-arch tags.

To assemble the suffixed images into a proper multi-arch manifest list under the unsuffixed tags (`docker pull mintlayer/node-daemon` then auto-selects the architecture), dispatch the **"Assemble multi-arch Docker manifests"** workflow (`release_docker_manifest.yml`) with the release version after both architectures have been pushed.
Loading
Loading