Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop compressing applehv and hyperv by default #3982

Merged
merged 2 commits into from
Dec 4, 2024

Conversation

jlebon
Copy link
Member

@jlebon jlebon commented Dec 3, 2024

Apple Hypervisor doesn't inherently require images to be compressed with gzip. It's just that when we do compress it, it's the most convenient format to use because gzip is guaranteed to be available on macOS.

Similarly for Windows Hyper-V and ZIP.

Notably, this is different from e.g. GCP, where the platform itself dictates a tar.gz file.

And so for consistency we should have the output from the build step for applehv and hyperv just return the disk image in the format it's intended to be used in, and then cosa compress just compresses them using e.g. gzip or zip. This requires adding a new platform-compressor key in the image.yaml file to allow overridding the default compressor setting for certain platforms.

This allows folks using the same code to build the disk images for those platforms and compress them with the compressor of their choice.

@jlebon
Copy link
Member Author

jlebon commented Dec 3, 2024

Requires: coreos/fedora-coreos-config#3293

(And also requires it to propagate to the other non-production branches. It'll propagate to the production ones via regular promotion.)

@jlebon
Copy link
Member Author

jlebon commented Dec 3, 2024

I guess we also need to tweak the osbuild pipelines for applehv and hyperv to no longer compress. I can try that here (or maybe @dustymabe if you want, you can add on top of this PR), or we can do it in a follow-up.

Comment on lines +124 to 125
elif compressor == 'gzip':
runcmd(['gzip', f'-{gzip_level}', '-c', filepath], stdout=f) # pylint: disable=E0606
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noting here that this gzip-level is not the same as the one used previously from qemuvariants.py.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if we care, just noting the change

@dustymabe
Copy link
Member

It'll propagate to the production ones via regular promotion.

but I think we'd need to forward port coreos/fedora-coreos-config#3293 so the next stable build would pick up the changes if we wanted to build using the latest COSA.

jlebon and others added 2 commits December 3, 2024 15:12
Apple Hypervisor doesn't inherently require images to be compressed with
gzip. It's just that when we _do_ compress it, it's the most convenient
format to use because gzip is guaranteed to be available on macOS.

Similarly for Windows Hyper-V and ZIP.

Notably, this is different from e.g. GCP, where the platform itself
dictates a `tar.gz` file.

And so for consistency we should have the output from the build
step for `applehv` and `hyperv` just return the disk image in the
format it's intended to be used in, and then `cosa compress` just
compresses them using e.g. `gzip` or `zip`. This requires adding a new
`platform-compressor` key in the `image.yaml` file to allow overridding
the default `compressor` setting for certain platforms.

This allows folks using the same code to build the disk images for those
platforms and compress them with the compressor of their choice.
They will now be compressed in the `cosa compress` CoreOS pipeline
stage.
@dustymabe
Copy link
Member

ok - added the OSBuild bits here.

Copy link
Member

@dustymabe dustymabe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jlebon jlebon added the hold waiting on something label Dec 3, 2024
Copy link
Member

@travier travier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not tested but LGTM. I vaguely remember the podman folks looking for zstd compression but maybe that does not matter here as they don't use those images directly?

@travier
Copy link
Member

travier commented Dec 4, 2024

Also, how much will be the impact on S3 storage if we only store uncompressed images?

@travier
Copy link
Member

travier commented Dec 4, 2024

@travier
Copy link
Member

travier commented Dec 4, 2024

ah, my bad, I see in coreos/fedora-coreos-config#3293 that we are compressing with gzip

Copy link
Contributor

@jbtrystram jbtrystram left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

superficial lgtm as I did not test but that looks sane to me.

@dustymabe
Copy link
Member

@travier yep. The podman folks will take the generated .raw and .vhdx images and compress them before distributing using whatever compression they desire.

The changes here, along with coreos/fedora-coreos-config#3293 will yield no changes to the uploaded applehv and hyperv artifacts. It just changes when the compression happens (i.e. in cosa compress versus at the time the image is built).

One minor change here is that the level of gzip compression for the applehv image will be 6 versus 9. We could consider changing the default in cosa compress to 9, but that would also change it for RHCOS. This may be something we still want to do, or not.

@jlebon jlebon merged commit a13cf77 into coreos:main Dec 4, 2024
5 checks passed
@jlebon jlebon removed the hold waiting on something label Dec 4, 2024
@jlebon jlebon deleted the pr/compress-custom branch December 4, 2024 19:00
dustymabe added a commit to dustymabe/custom-coreos-disk-images that referenced this pull request Dec 4, 2024
In COSA we dropped the compression of hyperv and applehv
in the OSBuild invocation [1] so we update here for that.

In COSA upstream we also merged in patches for Live ISO
building but that hasn't landed in RPMS yet so we surgically
remove it here until those new stages land.

[1] coreos/coreos-assembler#3982
dustymabe added a commit to coreos/custom-coreos-disk-images that referenced this pull request Dec 4, 2024
In COSA we dropped the compression of hyperv and applehv
in the OSBuild invocation [1] so we update here for that.

In COSA upstream we also merged in patches for Live ISO
building but that hasn't landed in RPMS yet so we surgically
remove it here until those new stages land.

[1] coreos/coreos-assembler#3982
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants