dind: remove containerd, and use embedded containerd

[thaJeztah]

• stacked on dind: remove obsolete runtime dependencies #584
• relates to daemon: Add embedded containerd mode moby/moby#52898
• relates to containerd: fall back to embedded when binary is unavailable moby/moby#53388

We could even consider dropping the zfs dependency; reducing the image size we can go from 528MB -> 429MB (unpacked + content)

• removing redundant dependencies (528MB -> 519MB)
• removing containerd (519MB -> 471MB)
• removing ctr (471MB -> 442MB)
• removing zfs (442MB -> 429MB)

docker image ls dind
IMAGE           ID             DISK USAGE   CONTENT SIZE   EXTRA
dind:before     6f84404d1f9d        528MB          137MB
dind:lessdeps   7c41dc0bc569        519MB          134MB
dind:noc8d      b42924822491        471MB          122MB
dind:noctr      a6c8b5c25b14        442MB          114MB
dind:nozfs      ce720faa179a        429MB          111MB

[tianon]

If we do this, we should definitely include a comment here explaining why, probably with a pointer to something useful like moby/moby#53388 -- maybe something like this?

Suggested change

	--exclude 'docker/containerd' \
	--exclude 'docker/ctr' \
	# exclude containerd, relying on the "embedded containerd" instead: https://github.com/moby/moby/pull/53388 (makes for a smaller image)
	--exclude 'docker/containerd' \
	--exclude 'docker/ctr' \

[tianon]

IMO, one of the biggest and most attractive benefits to the containerd integration itself is access to the containerd content store, especially being able to trivially explore/query that via ctr, so I'm honestly not a huge fan of this change (and think going the other direction and embedding dockerd inside containerd/making it a plugin instead is saner), but I don't feel strongly enough about it to block this.

[tianon]

(it's a very niche use case either way)

[thaJeztah]

Still draft, but updated to keep the ctr binary for debugging.

Size compared to #586

docker image ls dind
IMAGE           ID             DISK USAGE   CONTENT SIZE   EXTRA
dind:before     66a85fa1e49a        502MB          131MB
dind:nozfs      d2c8980d8cf6        489MB          128MB
dind:noc8d      04d2eaebd3c4        440MB          115MB

The embedded containerd also exposes the containerd socket at /run/docker/containerd/containerd.sock (same as before), so this should still work;

docker run -d --rm --name dind --privileged dind:noc8d
docker exec -it dind sh

command -v containerd || echo 'not found'
not found

docker pull -q hello-world
docker.io/library/hello-world:latest

ctr -a /run/docker/containerd/containerd.sock -n moby image ls
REF                                  TYPE                                    DIGEST                                                                  SIZE     PLATFORMS                                                                                                                         LABELS
docker.io/library/hello-world:latest application/vnd.oci.image.index.v1+json sha256:5dd0d3e6e255913fc30f90b9f2b1d359cc2cbdb48090cc4b65f1676e203243cc 15.5 KiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/riscv64,linux/s390x,windows/amd64 -

[thaJeztah]

Pulling directly through containerd .. doesn't work;

ctr -a /run/docker/containerd/containerd.sock -n moby image pull docker.io/library/alpine:latest
ctr: unknown service containerd.services.streaming.v1.Streaming

But with a separate containerd probably wasn't fully functional (when trying to use the image in moby);

ctr -a /run/docker/containerd/containerd.sock -n moby image pull docker.io/library/alpine:latest
docker.io/library/alpine:latest         	fetching image content
docker.io/library/alpine:latest         	fetching image content
└──index (28bd5fe8b56d)                 	complete   	|++++++++++++++++++++++++++++++++++++++|
...

It pulled the image, but docker doesn't have the right data I guess;

docker image ls
IMAGE                ID             DISK USAGE   CONTENT SIZE   EXTRA
alpine:latest        28bd5fe8b56d       13.5MB          4.2MB
hello-world:latest   5dd0d3e6e255       22.6kB         10.3kB

[tianon]

If we do something like this, we can merge this as-is now with no changes, and it will only apply the change on the next release (even if it's an RC):

Suggested change

	# exclude containerd, relying on the "embedded containerd" instead: https://github.com/moby/moby/pull/53388 (makes for a smaller image)
	--exclude 'docker/containerd' \
	; \
	rm docker.tgz; \
	\
	dockerd --version; \
	{{ if .version == "29.8.2" then "" else ( -}}
	# exclude containerd, relying on the "embedded containerd" instead: https://github.com/moby/moby/pull/53388 (makes for a smaller image)
	--exclude 'docker/containerd' \
	{{ ) end -}}
	; \
	rm docker.tgz; \
	\
	dockerd --version; \
	{{ if .version == "29.8.2" then ( -}}
	containerd --version; \
	{{ ) else "" end -}}

[thaJeztah]

Thx! Applied your changes 👍

cc @vvoland FYI

[vvoland]

LGTM