Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

image: multi-manifest index support #176

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

matthewpi
Copy link

@matthewpi matthewpi commented May 2, 2023

Closes #175

The code works and ready to be merged after a review. Any feedback related to the API design is welcome.

@wagoodman wagoodman self-assigned this May 4, 2023
@matthewpi matthewpi force-pushed the issue/175 branch 2 times, most recently from 461d95c to 1860471 Compare May 8, 2023 16:57
@matthewpi matthewpi marked this pull request as ready for review June 14, 2023 20:32
@matthewpi matthewpi changed the title image: rough implementation of multi-manifest index support image: multi-manifest index support Jun 14, 2023
@wagoodman
Copy link
Contributor

Thanks for the contribution (and sorry for the delay)! I'm curious on your thoughts on how this might get wired up in syft/grype? Or maybe this is specifically an API update to be used elsewhere, which is also alright.

And from your issue:

Secondly, there would now another entire code path to support multi-manifest indexes. While this code path does still fully support single-manifest indexes (or just single images if the format doesn't have the concept of multiple manifests), it does mean users of this library will need to implement support for a different API and new users may be confused about using the Index vs non-index functions.

This resonates with me. I think the best way to reconcile this is still allow both code paths, but enhance the GetImageFromSource to take an additional Option which enables automatic resolution to a single image (or error) from an index. This could take additional inputs such as platform into consideration. Passing this "strategy" as an option would at least enable the default path for folks interested in a single image when provided a reference (and also allow folks interested in all of the images behind the index to also have that without polluting the single-image path).

How does this path sound to you? Also would you like help getting this across the finish line?

@matthewpi
Copy link
Author

I'm curious on your thoughts on how this might get wired up in syft/grype?

I'm unsure about this as well. I looked into integrating it into Syft, but the code path the Image type traverses is massive which would require a lot of refactoring or re-thinking of how the integration would work. I know there has been a discussion around how SBOMs would work for multiple images, but I think that's as simple as treating each image in the index independently as while indexes are used and designed for supporting multiple architectures, the images themselves can still be completely different and contain different dependencies (especially if cross-compiled). So I guess being able to run Syft and have it generate SBOMs per image in an index would be the best approach, maybe with the ability to add a reference in the index to the SBOMs (assuming they are pushed to a registry).

Or maybe this is specifically an API update to be used elsewhere, which is also alright.

My main goal with this was to get support added in Syft and Grype for image indexes. Just thought I would start here seeing as it is a prerequisite to be able to add support for image indexes in those consumers.


I think the best way to reconcile this is still allow both code paths, but enhance the GetImageFromSource to take an additional Option which enables automatic resolution to a single image (or error) from an index.

So we would keep the new API as is, but modify GetImageFromSource to call the ProvideIndex API (if supported by the provider), and have it run a filter based off of the OS and architecture, throwing an error if more than one image is matched, but otherwise will just return that one image that matches the filter.

So we could add a WithPlatformSelector option (there is already a WithPlatform option, but it seems designed to set the platform on the image, rather than to be used to filter an index). Re-use the same platform name parsing logic, then use that to filter the images returned by ProvideIndex. Or instead of just adding individual selector options we could do a WithStrategy or WithSelectorStrategy option which uses a struct to allow more advanced filtering down the road, rather than just a single platform.

What do you think about those ideas?


Also would you like help getting this across the finish line?

I'm fine to make the changes in Stereoscope assuming we can agree on a good API design for this. My main problem is figuring out the next steps after this is merged relating to adding support in Syft and/or Grype for this. Some guidance on where to start an integration code-wise, or having a discussion about how the integration should work would be appreciated.

@wagoodman
Copy link
Contributor

Linking the related syft issue here anchore/syft#1683

@JonZeolla
Copy link

👋 found my way here after hitting some syft issues generating a SBOM based on my multiplatform OCI manifest. Interested in the plan here

I agree with @matthewpi that a fanout approach - one SBOM per image:platform - is probably the best short term fix. That's going to be my workaround, if I need to put one in place, and I recently left similar feedback in anchore/syft#1683

@matthewpi
Copy link
Author

I think the best way to reconcile this is still allow both code paths, but enhance the GetImageFromSource to take an additional Option which enables automatic resolution to a single image (or error) from an index. This could take additional inputs such as platform into consideration. Passing this "strategy" as an option would at least enable the default path for folks interested in a single image when provided a reference (and also allow folks interested in all of the images behind the index to also have that without polluting the single-image path).

I added the ability for the OCI providers to take into account a platform selector like a few of the other providers, alongside rebasing on top of the latest main changes. Do you think this satisfies the "strategy" option or should a more generic solution (such as a user-provided filtering function) be used instead?

There are two TODO comments in the latest commit, asking two interesting questions.

  1. In the event of no platform selector and an image index with no identical digests (a recent change that allows indexes referring to the same manifest digest multiple times to be treated as a single image), should we default to the current OS rather than just throwing an error?
  2. Should we throw an error in the edge-case where multiple manifests in an index may be matched by the platform selector?

The other thing that clicked into my head about #2, is a image index can refer to another image index and I believe other media types that are not manifests or images entirely. Should we be ensuring that these get filtered out?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for image indexes with multiple manifests
3 participants