Unable to load docker desktop containerd managed images to cluster

[iamvinov-atlassian]

What happened:
I am attempting to load Docker images into kind cluster: kind load docker-image busybox -n nebulae but getting the following error:
❯ kind load docker-image busybox -n nebulae Image: "busybox" with ID "sha256:5b0f33c83a97f5f7d12698df6732098b0cdb860d377f6307b68efe2c6821296f" not yet present on node "nebulae-control-plane", loading... ERROR: failed to load image: command "docker exec --privileged -i nebulae-control-plane ctr --namespace=k8s.io images import --all-platforms --digests --snapshotter=overlayfs -" failed with error: exit status 1 Command Output: ctr: content digest sha256:83e82a8dd385e27d95f2118c1332d414684aa665552f7f837f86da33674308c4: not found

What you expected to happen:
I expected the image to load successfully. I already have this image pulled locally using docker pull busybox. Upon further investigation, it seems to me that kind or (containerd) expects the image for all platforms to be present on the host for the load command to succeed.

How to reproduce it (as minimally and precisely as possible):

docker pull busybox
kind create cluster --name nebulae
kind load -v 10 docker-image -n nebulae busybox

Anything else we need to know?:
From looking at other answers on the internet, it seems generally this error occurs when the image arch doesn't match the host arch. But this is not the case. I did perform docker images --tree and made sure the images match my host (M3 MacBook Pro) OS.

busybox:latest                                                                         5b0f33c83a97       12.6MB            4MB
├─ linux/arm64/v8                                                                      6ca1ac3927a1       6.02MB         1.85MB
├─ linux/amd64                                                                         a3e1b257b47c       6.56MB         2.16MB
├─ linux/arm/v5                                                                        3076001161ce           0B             0B
├─ linux/arm/v6                                                                        a9fc789b4096           0B             0B
├─ linux/arm/v7                                                                        fb632082f5cb           0B             0B
├─ linux/386                                                                           c0d2f0e7a91f           0B             0B
├─ linux/mips64le                                                                      0e1d386b0b5d           0B             0B
├─ linux/ppc64le                                                                       fc082c5fdd21           0B             0B
├─ linux/riscv64                                                                       d55b3027f77f           0B             0B
└─ linux/s390x                                                                         4bc8b19fe938           0B             0B

Environment:

• kind version: kind v0.25.0 go1.23.3 darwin/arm64
• Runtime info: (use docker info, podman info or nerdctl info):

❯ docker info
Client:
 Version:    27.3.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/vvelu/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.18.0-desktop.2
    Path:     /Users/vvelu/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.30.3-desktop.1
    Path:     /Users/vvelu/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     /Users/vvelu/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     /Users/vvelu/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/vvelu/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/vvelu/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/vvelu/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/vvelu/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/vvelu/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.0
    Path:     /Users/vvelu/.docker/cli-plugins/docker-scout

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 21
 Server Version: 27.3.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.653GiB
 Name: docker-desktop
 ID: 794edb33-e6f7-4749-8c5c-edf7b3d5cf21
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/vvelu/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

• OS: MacOS Sonoma v14.7
• Kubernetes version: (use kubectl version):

❯ kubectl version
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.31.2

• Any proxies or other special environment settings?:

[iamvinov-atlassian]

More information. Unticking: "Use containerd for pulling and storing images" in Docker Desktop actually resolves this.

[BenTheElder]

That error message is coming from ctr when we ask it to import the image saved from docker.

Unfortunately I can't run docker desktop at work, will have to find another way to reproduce this.

Can you look at the same image exported the way kind does with docker save,

kind/pkg/cmd/kind/load/docker-image/docker-image.go

Line 210 in 74acdf7

commandArgs := append([]string{"save", "-o", dest}, images...)

or provide a tarball from that somewhere? That would speed things up (can replicate the rest with kind load image-archive)

this sounds like a containerd/docker bug but we need to confirm how before contacting them. The part kind is doing could be bugged but is pretty straightforward once we decide we need to load the image because it's not already available

[BenTheElder]

Could be a small sample image like busybox if you can confirm the bug still applies to that image and share the containerd vs dockerd mode versions that would speed things up. Otherwise it may be difficult to reproduce due to the licensing of the application and/or my employer's policies, I'll have to see if this is something I can replicate in some other way.

[porridge]

I'm hitting what seems to be the same issue in kuttl's integration tests. Kuttl embeds kind, currently v0.25.0.

Interestingly this works on CI (GHA, ubuntu 20.04 runner) but on my desktop this fails with the same message as for @iamvinov-atlassian

What I've been able to figure out using skopeo inspect --raw docker://docker.io/library/busybox:latest|jq . and docker image save docker.io/library/busybox:latest is that:

• the digest that ctr complains about is claimed to be an attestation-manifest for the linux/amd64 manifest
• the busybox' docker image mentions it in the index image, but the blob itself is nowhere to be found

So it seems like from the PoV of ctr the image is incomplete since it's lacking the attestation blob. FWIW, here is how the integration test fetches and loads the image.

I'm running:

[kuttl]$ docker info
Client:
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.0
    Path:     /home/mowsiany/.docker/cli-plugins/docker-buildx

Server:
 Containers: 20
  Running: 0
  Paused: 0
  Stopped: 20
 Images: 3
 Server Version: 27.3.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: /usr/bin/tini-static
 containerd version: 2.fc41
 runc version: 
 init version: 
 Security Options:
  seccomp
   Profile: builtin
  selinux
  cgroupns
 Kernel Version: 6.11.7-300.fc41.x86_64
 Operating System: Fedora Linux 41 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 62.5GiB
 Name: mowsiany-thinkpadp1gen5.remote.csb
 ID: e8f36c79-610a-4647-8cc3-b734cebd7050
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: porridgerox
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[porridge]

@BenTheElder you might be able to reproduce this with:

git clone https://github.com/kudobuilder/kuttl
cd kuttl
make envtest
KUBEBUILDER_ASSETS=$(./bin/setup-envtest use 1.25.0 --bin-dir `pwd`/bin -p path) go test -tags integration ./pkg/test -v -mod=readonly -test.run TestAddContainers

Unfortunately the actual error is hidden as sigs.k8s.io/kind/pkg/cluster/nodeutils.LoadImageArchive is missing .SetStdout(os.Stdout).SetStderr(os.Stderr) at least in the version we use.