base does no longer accept a tarball

[mering]

In v1, base was accepting a tarball. I assume this is the reason why it sets allow_single_file = True. In v2, this fails with the following error:

ls: unknown io error: '"bazel-out/k8-fastbuild/bin/path/image_tarball/tarball.tar/blobs/*/*"', 'Os { code: 20, kind: NotADirectory, message: "Not a directory" }'

The corresponding code is

rules_oci/oci/private/image.sh

Lines 33 to 48 in c74c2d3

	function base_from() {
	local path="$1"
	# shellcheck disable=SC2045
	for blob in $(coreutils ls -1 -d "$path/blobs/"*/*); do
	local relative_to_blobs="${blob#"$path/blobs"}"
	coreutils mkdir -p "$OUTPUT/blobs/$(coreutils dirname "$relative_to_blobs")"
	if [[ "$USE_TREEARTIFACT_SYMLINKS" == "1" ]]; then
	# Relative path from `output/blobs/sha256/` to `$blob`
	relative="$(coreutils realpath --relative-to="$OUTPUT/blobs/sha256" "$blob" --no-symlinks)"
	coreutils ln -s "$relative" "$OUTPUT/blobs/$relative_to_blobs"
	else
	coreutils cp --no-preserve=mode "$blob" "$OUTPUT/blobs/$relative_to_blobs"
	fi
	done
	coreutils cp --no-preserve=mode "$path/oci-layout" "$OUTPUT/oci-layout"
	jq '.manifests[0].annotations["org.opencontainers.image.ref.name"] = "intermediate"' "$path/index.json" >"$OUTPUT/index.json"

Could you detect if base is a tarball and extract it in this case?

We use this feature to gradually migrate our large code base from rules_docker to rules_oci.

[thesayyn]

oci_image never accepted a tarball as the base, if it did though, probably because of crane we were running at the time. We won't be supporting tarballs as they are usually huge and not cache friendly.

[mering]

@thesayyn I agree that one shouldn't create tarballs as outputs but it might be necessary to support tarballs as inputs to support migrations to rules_oci.

So this is about unpacking a tarball which already exists instead of creating a new one. So this doesn't make things worse as they already are today but instead paves the way to a better future with fewer tarballs.

[thesayyn]

I agree with the sentiment here, however adding this feature back is not something i'd be warm to, i'd be much comfortable with an example showing use of regctl with a genrule to unpack a tarball into an oci-layout which is then used as the base.

[mering]

I agree with the sentiment here, however adding this feature back is not something i'd be warm to, i'd be much comfortable with an example showing use of regctl with a genrule to unpack a tarball into an oci-layout which is then used as the base.

Or maybe providing an explicit oci_import() rule to unpack the tar. Preferably also backporting to 1.x to allow for a smoother upgrade path (using that explicit rule already in 1.x before doing the upgrade).

[mering]

@thesayyn which regctl command would you use to unpack a tarball into an oci-layout?

[thesayyn]

regctl image import path/to/tarball oci-layout://path/to/output

[thesayyn]

Something like genrule would work but genrules no longer support outputting a directory, so you'd have to create a custom rule.

[mering]

regctl image import path/to/tarball oci-layout://path/to/output

Thanks, I couldn't figure out the oci-layout:// prefix when reading their docs.