failed to read expected number of bytes: unexpected EOF

[amir-bialek]

In our k8s on prem we have a local image registry from image docker.io/registry:2.8.3 that is a proxy from our dockerhub (registry mirror).
Sometimes our self-hosted-runners (ARC) get an error when trying to pull an image from it:

ERROR: failed to solve: failed to read expected number of bytes: unexpected EOF

This issue occurs when the manifest in the local registry is corrupted. a quick-fix we do is delete the image from our local registry and re-run the job. this cause the local registry to re-pull it from docker.io

I will like to find a permanent solution, will appreciate your help.

Local registry Env configuration:

      REGISTRY_STORAGE_DELETE_ENABLED: true 
      REGISTRY_PROXY_REMOTEURL: https://registry.hub.docker.com
      REGISTRY_PROXY_USERNAME: some_user
      REGISTRY_PROXY_PASSWORD: some_token

PVC mountPath: /var/lib/registry

We added a registry-ui that connect to the SVC with image joxit/docker-registry-ui:main
ENV Configuration of the ui image:

      SINGLE_REGISTRY: false
      REGISTRY_TITLE: Some_title
      DELETE_IMAGES: true
      SHOW_CONTENT_DIGEST: false
      NGINX_PROXY_PASS_URL: http://Some_svc:Some_port
      SHOW_CATALOG_NB_TAGS: true
      CATALOG_MIN_BRANCHES: 1
      CATALOG_MAX_BRANCHES: 1
      TAGLIST_PAGE_SIZE: 100
      REGISTRY_SECURED: false
      CATALOG_ELEMENTS_LIMIT: 1000

The registry-ui have an ingress, and the local runners are pulling it from that ingress.
They have the folling as configuration for daemon.json:

    {
      "insecure-registries": ["Some_svc:Some_port", "some_ingress" ],
      "registry-mirrors": ["http://Some_svc:Some_port", "https://some_ingress"],
      "features": {
      "containerd-snapshotter": true
      },
      "mtu": 1400
    }

Anyone know what can cause this issue?