Scope issue on push with multiple sub-repositories in GCR

[b4nst]

Using docker-credential-gcr configured like this:

before_script:
  - apk add --no-cache curl
  - curl -fsSL "https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v2.1.6/docker-credential-gcr_linux_amd64-2.1.6.tar.gz" | tar xz docker-credential-gcr
  - chmod +x docker-credential-gcr && mv docker-credential-gcr /usr/bin/
  - docker-credential-gcr config --token-source="env, store"
  - docker-credential-gcr configure-docker

I get a 401 Unauthorized with manifest-tool:

manifest-tool --docker-cfg /root/.docker/config.json --debug push from-args --platforms linux/amd64,linux/arm64 --template ${TARGET_REPO}/${TARGET_IMAGE}/ARCH:${TARGET_TAG} --target ${TARGET_REPO}/${TARGET_IMAGE}:${TARGET_TAG}
time="2023-02-20T15:54:00Z" level=info msg="Retrieving digests of member images"
time="2023-02-20T15:54:00Z" level=debug msg=resolving host=us.gcr.io
time="2023-02-20T15:54:00Z" level=debug msg="do request" host=us.gcr.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.8+unknown request.method=HEAD url="https://us.gcr.io/v2/[redacted]/multiarch-container/amd64/manifests/b712d0adccd4ebb07ceca7488d4b6e27d44bf6f8"
time="2023-02-20T15:54:00Z" level=debug msg="fetch response received" host=us.gcr.io response.header.accept-ranges=none response.header.cache-control=private response.header.content-type=application/json response.header.date="Mon, 20 Feb 2023 15:54:00 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server="Docker Registry" response.header.vary=Accept-Encoding response.header.www-authenticate="Bearer realm=\"https://us.gcr.io/v2/token\",service=\"us.gcr.io\",scope=\"repository:[redacted]/multiarch-container/amd64:pull\"" response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="401 Unauthorized" url="https://us.gcr.io/v2/[redacted]/multiarch-container/amd64/manifests/b712d0adccd4ebb07ceca7488d4b6e27d44bf6f8"
time="2023-02-20T15:54:00Z" level=debug msg=Unauthorized header="Bearer realm=\"https://us.gcr.io/v2/token\",service=\"us.gcr.io\",scope=\"repository:[redacted]/multiarch-container/amd64:pull\"" host=us.gcr.io
time="2023-02-20T15:54:00Z" level=debug msg="do request" host=us.gcr.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.8+unknown request.method=HEAD url="https://us.gcr.io/v2/[redacted]/multiarch-container/amd64/manifests/b712d0adccd4ebb07ceca7488d4b6e27d44bf6f8"
time="2023-02-20T15:54:00Z" level=debug msg="fetch response received" host=us.gcr.io response.header.content-length=424 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Mon, 20 Feb 2023 15:54:00 GMT" response.header.docker-content-digest="sha256:302d716b2105f09293dfd312491cd07f7b66dcf697114e81e97703eaff030673" response.header.docker-distribution-api-version=registry/2.0 response.header.server="Docker Registry" response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="200 OK" url="https://us.gcr.io/v2/[redacted]/multiarch-container/amd64/manifests/b712d0adccd4ebb07ceca7488d4b6e27d44bf6f8"
time="2023-02-20T15:54:00Z" level=debug msg=resolved desc.digest="sha256:302d716b2105f09293dfd312491cd07f7b66dcf697114e81e97703eaff030673" host=us.gcr.io
time="2023-02-20T15:54:00Z" level=debug msg=fetch digest="sha256:302d716b2105f09293dfd312491cd07f7b66dcf697114e81e97703eaff030673" mediatype=application/vnd.docker.distribution.manifest.v2+json size=424
time="2023-02-20T15:54:00Z" level=debug msg="do request" digest="sha256:302d716b2105f09293dfd312491cd07f7b66dcf697114e81e97703eaff030673" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.6.8+unknown request.method=GET size=424 url="https://us.gcr.io/v2/[redacted]/multiarch-container/amd64/manifests/sha256:302d716b2105f09293dfd312491cd07f7b66dcf697114e81e97703eaff030673"
time="2023-02-20T15:54:00Z" level=debug msg="fetch response received" digest="sha256:302d716b2105f09293dfd312491cd07f7b66dcf697114e81e97703eaff030673" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=424 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Mon, 20 Feb 2023 15:54:00 GMT" response.header.docker-content-digest="sha256:302d716b2105f09293dfd312491cd07f7b66dcf697114e81e97703eaff030673" response.header.docker-distribution-api-version=registry/2.0 response.header.server="Docker Registry" response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="200 OK" size=424 url="https://us.gcr.io/v2/[redacted]/multiarch-container/amd64/manifests/sha256:302d716b2105f09293dfd312491cd07f7b66dcf697114e81e97703eaff030673"
time="2023-02-20T15:54:00Z" level=debug msg=fetch digest="sha256:375bc02c1d6cf0ae34a7d48ea0cda91f07a5b982404bd1a587fe3439f2c7f4a3" mediatype=application/vnd.docker.container.image.v1+json size=822
time="2023-02-20T15:54:00Z" level=debug msg="do request" digest="sha256:375bc02c1d6cf0ae34a7d48ea0cda91f07a5b982404bd1a587fe3439f2c7f4a3" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.6.8+unknown request.method=GET size=822 url="https://us.gcr.io/v2/[redacted]/multiarch-container/amd64/blobs/sha256:375bc02c1d6cf0ae34a7d48ea0cda91f07a5b982404bd1a587fe3439f2c7f4a3"
time="2023-02-20T15:54:00Z" level=debug msg="fetch response received" digest="sha256:375bc02c1d6cf0ae34a7d48ea0cda91f07a5b982404bd1a587fe3439f2c7f4a3" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="private, max-age=0" response.header.content-length=822 response.header.content-type=application/octet-stream response.header.date="Mon, 20 Feb 2023 15:54:00 GMT" response.header.etag="\"2f634da39749f8da540faae808432222\"" response.header.expires="Mon, 20 Feb 2023 15:54:00 GMT" response.header.last-modified="Mon, 20 Feb 2023 15:53:46 GMT" response.header.server=UploadServer response.header.x-goog-generation=1676908426076193 response.header.x-goog-hash="crc32c=GKXPMA==" response.header.x-goog-hash.1="md5=L2NNo5dJ+NpUD6roCEMiIg==" response.header.x-goog-metageneration=1 response.header.x-goog-storage-class=STANDARD response.header.x-goog-stored-content-encoding=identity response.header.x-goog-stored-content-length=822 response.header.x-guploader-uploadid=ADPycdtUcYC_8rvJ5BcCcCg24r5ffFa5ssY2VdwofP9G7xxjBkpCpLSAd5WrEkCce8MGeRyqkNxc1hJzC-d1HSoL10gRlA response.status="200 OK" size=822 url="https://us.gcr.io/v2/[redacted]/multiarch-container/amd64/blobs/sha256:375bc02c1d6cf0ae34a7d48ea0cda91f07a5b982404bd1a587fe3439f2c7f4a3"
time="2023-02-20T15:54:00Z" level=debug msg=resolving host=us.gcr.io
time="2023-02-20T15:54:00Z" level=debug msg="do request" host=us.gcr.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.8+unknown request.method=HEAD url="https://us.gcr.io/v2/[redacted]/multiarch-container/arm64/manifests/b712d0adccd4ebb07ceca7488d4b6e27d44bf6f8"
time="2023-02-20T15:54:00Z" level=debug msg="fetch response received" host=us.gcr.io response.header.accept-ranges=none response.header.cache-control=private response.header.content-type=application/json response.header.date="Mon, 20 Feb 2023 15:54:00 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server="Docker Registry" response.header.vary=Accept-Encoding response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="401 Unauthorized" url="https://us.gcr.io/v2/[redacted]/multiarch-container/arm64/manifests/b712d0adccd4ebb07ceca7488d4b6e27d44bf6f8"
time="2023-02-20T15:54:00Z" level=debug msg=Unauthorized header= host=us.gcr.io
time="2023-02-20T15:54:00Z" level=fatal msg="Inspect of image \"us.gcr.io/[redacted]/multiarch-container/arm64:b712d0adccd4ebb07ceca7488d4b6e27d44bf6f8\" failed with error: pulling from host us.gcr.io failed with status code [manifests b712d0adccd4ebb07ceca7488d4b6e27d44bf6f8]: 401 Unauthorized"

I double-checked, image can be pulled with other tools.

[estesp]

What version of manifest-tool were you using? I can test this with my test GCR instance when I get a chance.

If you built from main in the last few weeks this was probably not working, but all released versions should work. I'm preparing to release v2.0.7 which has re-worked credential/Docker config support that isn't relying on the ORAS library import. I'll make sure to validate this working with GCR before releasing. I have already verified with DockerHub and AWS ECR.

[b4nst]

I was using mplatform/manifest-tool:alpine container image

[estesp]

I was using mplatform/manifest-tool:alpine container image

Ah! This is most likely a limitation of using manifest-tool from within a container. If you installed the credential helper that is on the host and configured as such on the host. The container, unless you mount all the right places inside the container, has no idea about the credential helper. I believe if you try manifest-tool installed on the host you should find that it works properly.

It might be an interesting enhancement to add popular credential helpers to the container image, but there also might be additional steps (e.g. some cred helpers are going to want access to environment variables that are not set inside the container without specifically adding them) to get them to work properly.

[b4nst]

Nah this is part of a CI build, I installed the credential helper directly in the container. This container is running on a k8s cluster, so even if I wanted I wouldn't be able to install the helper on the host.

[estesp]

Looks like this is not any issue with the credential helper anyway; I just verified/tested that there are no issues with using the GCR cred helper.

This might be a scope issue with the initial auth; looks like you are using subrepos for the "input" architectures? e.g. /multiarch-container/amd64 for 64-bit Intel, /multiarch-container/arm64 for ARM64v8, etc. The scope of the auth uses the initial repo (from the log: scope=\"repository:[redacted]/multiarch-container/amd64:pull\") and gets a token and is able to get the references needed from the amd64 image, but then it moves on to arm64 and uses the same token with the same scope and gets the 401. Not sure why I haven't seen this before as I have tested DockerHub at least with multiple repos as inputs. Let me dig a bit deeper

[b4nst]

Okay on my side I can try to push them in the same repo to confirm the issue. Thanks!

[waddles]

It would be great if you could handle creds inside the container the way kaniko does. I'm trying to create a bunch of multi-architecture images with kaniko and write a single image_manifest.yaml for each one

---
image: registry/repo:123456-linuxarm64v8
platform:
  os: linux
  architecture: arm64
  variant: v8

which in a later stage, I want to gather those artifacts and merge them with yq then push them to my registry with manifest-tool.

yq ea '. as $item ireduce([]; . + $item) | {"image": env(IMAGE_URL), "tags": ("${TAGS}" | envsubst | split(" ")), "manifests": . }' image_manifests/* > manifest.yaml
manifest-tool push from-spec manifest.yaml

The problem is I don't know which registry the developers want this image to go to and I'd like manifest-tool to handle the auth rather than the infrastructure.

[estesp]

@waddles seems reasonable to enhance the container with the cred helpers, but this issue has kind of morphed from what turns out the root cause was (looks like a scope issue in the auth transaction). I just opened a new issue (#216) to discuss/work towards better cred helper support when containerized, and, with @b4nst's approval, rename this issue to focus on the scope/authorizer issue.

[b4nst]

Sure, thanks for asking! Sorry I didn't have time to test the scope yet, will try today

[b4nst]

Sorry it's been a while. Just to confirm it does work with ${TARGET_REPO}/${TARGET_IMAGE}:${TARGET_TAG}-ARCH as a pattern. Putting the arch as part of the tag seems to resolve the scope issue