Not getting images from mirror-registry

[eb4x]

What happened:
The cronjobs fetching fedora/centos images time out in an air-gapped cluster.

I0805 13:40:08.423084 1 registry-datasource.go:176] Copying proxy certs
2024/08/05 13:40:08 Ignore common certificate dir: open /proxycerts/: no such file or directory
I0805 13:40:08.423157 1 transport.go:228] Inspecting image from 'docker://quay.io/containerdisks/fedora:latest'
E0805 13:41:08.425189 1 transport.go:78] Could not create image reference: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 34.206.201.197:443: i/o timeout
2024/08/05 13:41:08 Failed to get image digest: Could not create image reference: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 34.206.201.197:443: i/o timeout

What you expected to happen:
Getting the image from a mirror via the /etc/containers/registries.conf.d/99-mirror-registries.conf configuration on each node.

[[registry]]
prefix="quay.io"
location="mirror-registry.openshift-utv.uio.no:8443/mirrors/quay.io"

Additional context:

I have a baremetal OKD 4.14 SCOS in an semi-air-gapped environment. I push images to a minimal mirror-registry (quay) separate from the cluster, and all nodes have a configuration (see above snippet) that forwards the most common registry-urls to a location in mirror-registry.

Environment:

• CDI version (use kubectl get deployments cdi-deployment -o yaml): 1.10.7
• Kubernetes version (use kubectl version): v1.27.1-3507+28ed2d70d51caa-dirty
• DV specification: N/A
• Cloud provider or hardware configuration: OKD/Baremetal
• OS (e.g. from /etc/os-release): CentOS Stream CoreOS 414.9.202401231453-0
• Kernel (e.g. uname -a): 5.14.0-410.el9.x86_64

[aglitke]

@arnongilboa could you take a look at this? It seems we should be supporting this scenario. @eb4x Are you using HCO to deploy the kubevirt components are are you doing a more custom setup? Make sure that you have configured your DataImportCron objects to point to the correct registry. From the error message it looks like you are still trying to fetch from quay.

[arnongilboa]

@aglitke we should definitely support this scenario. @eb4x As mentioned, the DataImportCrons should point urls in the mirror-registry. See here. If you are using OpenShift (with HCO) you may follow this.

[eb4x]

@eb4x Are you using HCO to deploy the kubevirt components are are you doing a more custom setup?

Yep, I'm using the HCO to deploy.

Make sure that you have configured your DataImportCron objects to point to the correct registry. From the error message it looks like you are still trying to fetch from quay.

I'll look into this further, I might have missed something. The idea behind our registry overrides is that we don't have to change where we're pulling from. The worker nodes should know to pull from the mirror-registry.

It works for resources specified in kubernetes. (deployments, daemonsets, pods, etc.) But maybe not from within a running container which is kinda what's happening here?

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten