What happens
A hosted runner pre-pulls exactly one image at boot: the session_image in its runner_artifacts metadata, using a short-lived token from the VM metadata server (docker login … docker pull … docker logout). After that, when runnerd asks the Docker daemon to create a sandbox from an environment whose image is a different digest in the same private Artifact Registry repository, the daemon pulls with no credentials and the registry answers:
denied: Unauthenticated request. Unauthenticated requests do not have permission
"artifactregistry.repositories.downloadArtifacts" on resource ".../repositories/<images-repo>" (or it may not exist)
The session is marked failed about one second after creation. Observed 2026-09-12 on a dev cell while qualifying a canary environment on a newly published digest; the same command on an environment using the pre-pulled digest ran, and an environment identical to the canary except for the old digest ran, so the digest is the only variable.
Why it matters
rainier env create --image <digest> for anything but the runner's boot image cannot work on a hosted runner today, which makes the documented canary flow order-dependent on a runner roll (Cloud runbook default-environment-rollout.md step 5 before step 6).- Public images are unaffected as far as authentication goes; private ones in the operator's own repository are the ones that fail, which is the opposite of what an operator expects.
Proposed contract
runnerd sends registry credentials with each image pull:
- A pluggable credential source on the pull path (
RegistryAuth on the image-create call), resolved per pull so a short-lived token is never cached past its lifetime.
- Hosted default: a
--registry-token-command or a GCE-metadata source that exchanges the runner's service-account token for oauth2accesstoken on the registry host named by the image reference; self-hosted default: none (today's behaviour).
- Fail closed with a clear session failure reason (
image_pull_denied) that names the registry host, not the whole daemon error, so the CLI's Failure: line is actionable.
The runner service account already holds the reader role on the repository, so no IAM change is needed; this is a runnerd change plus one startup flag in the Cloud runner unit.
Out of scope
Caching policy, image allow-lists, and pull-through mirrors — each is its own discussion.
What happens
A hosted runner pre-pulls exactly one image at boot: the
session_imagein itsrunner_artifactsmetadata, using a short-lived token from the VM metadata server (docker login…docker pull…docker logout). After that, when runnerd asks the Docker daemon to create a sandbox from an environment whose image is a different digest in the same private Artifact Registry repository, the daemon pulls with no credentials and the registry answers:The session is marked failed about one second after creation. Observed 2026-09-12 on a dev cell while qualifying a canary environment on a newly published digest; the same command on an environment using the pre-pulled digest ran, and an environment identical to the canary except for the old digest ran, so the digest is the only variable.
Why it matters
rainier env create --image <digest>for anything but the runner's boot image cannot work on a hosted runner today, which makes the documented canary flow order-dependent on a runner roll (Cloud runbookdefault-environment-rollout.mdstep 5 before step 6).Proposed contract
runnerd sends registry credentials with each image pull:
RegistryAuthon the image-create call), resolved per pull so a short-lived token is never cached past its lifetime.--registry-token-commandor a GCE-metadata source that exchanges the runner's service-account token foroauth2accesstokenon the registry host named by the image reference; self-hosted default: none (today's behaviour).image_pull_denied) that names the registry host, not the whole daemon error, so the CLI'sFailure:line is actionable.The runner service account already holds the reader role on the repository, so no IAM change is needed; this is a runnerd change plus one startup flag in the Cloud runner unit.
Out of scope
Caching policy, image allow-lists, and pull-through mirrors — each is its own discussion.