Running soci create on an image and pushing doesn't always generate a new image index

[vinfang]

In our current setup, we are using buildctl to produce a tar of our build image, ctr to import it into a store that soci can access, and then running soci create to generate the index. From there , we run ctr push and soci push to push our build image and soci index to our AWS ECR to store the results.

When major code changes occur, a new build image, soci index, and image index are generated, and our ECS task is able to lazily load the image when running. However, we're noticing that if small code changes occur, like maybe a comment change or something small, while a new build image and soci index are generated and pushed, we don't see a new image index show up.

Despite a image index not being newly generated, our ECS task is still able to lazily load the image, and we even tried deleting an existing image index, and it still works. So we wanted to ask if this behavior of not always generating a new image index is correct or not. Or if this is not a soci tool issue but a AWS ECR issue?

I also wanted to ask if this image index is considered the third item in the (https://github.com/awslabs/soci-snapshotter/blob/main/docs/getting-started.md) documentation (the original image, the SOCI index, and manifest)? If it is I would think this is a problem that a new manifest is not being created but so far things still work.

The screenshot below has an image where we generated a new latest image and a new soci index, but no new image index. The one shown is from an older build indicated by the image tag that's pointed to an older latest image.

EDIT: used a different image from another repo that better represents the problem? we're seeing

[Kern--]

Hi, yes, this is expected and you shouldn't see new image indexes going forward because ECR added support for the OCI Image Distirbution Spec 1.1: https://aws.amazon.com/about-aws/whats-new/2024/06/amazon-ecr-oci-image-distribution-version-1-1/

SOCI indexes point one-directionally towards the image that they were made for. When pulling an image with SOCI, we need the reverse mapping from image to SOCI index. In OCI 1.0, that "backreference" is done by the image index tagged with the image digest. In OCI 1.1, that's handled by an API in ECR itself so the image index is no longer needed.