Fixing hfs-compression.diff

[ryandesign]

Hi, I'm a developer with MacPorts. The hfs-compression.diff patch enabled the use of HFS+ compression on macOS, which is a nice capability to have. As far as I know, despite its name, this transparent compression technology continues to live on in Apple's new APFS filesystem.

Both MacPorts and homebrew used to use the this patch in their rsync packages, but it was removed when updating to 3.1.3 because the patch was deliberately broken in 3a87167 by the addition of this line:

+$$$ERROR$$$ the old patch needs reworking since rsync_xal_get() has totally changed!

Before I dive into this (I've never looked at rsync's code before), do you know if anybody else is already working on reviving this patch?

Do you have any information about the ways in which rsync_xal_get totally changed or in what ways the patch would need to be modified? If not, I understand; just trying to avoid doing unnecessary work if I can.

[WayneD]

No one else has expressed an interest in fixing it. I really should have removed that diff from the patches repo long ago. I don't ever plan to support HFS+ compression in the main rsync and I don't have any way to test the patch myself. Seems like if you manage to get it going as an official MacPorts patch that is the best spot for it to live, since it will be good for those who can use & test it to maintain it.

As for change hints, I have forgotten what the changes were. Sorry.

[ryandesign]

If I get the patch working again, I would want to submit it here for inclusion in this repository since it is of benefit to all Mac users, not just those who get rsync from MacPorts. If you would not accept a fix for this patch, then I don't plan to work on it, because MacPorts does not wish to become the developers of other people's software; we only want to package and distribute existing software. (Homebrew developers have expressed similar sentiments.)

Why don't you want to support HFS+ compression, aside from not being able to test the feature yourself?

[WayneD]

It's a balancing act of how useful a feature is compared to how hard it is to make sure it doesn't break in weird ways. I'll leave it in the patches dir for now for those that might find it useful.

[mascguy]

If I get the patch working again, I would want to submit it here for inclusion in this repository since it is of benefit to all Mac users, not just those who get rsync from MacPorts. If you would not accept a fix for this patch, then I don't plan to work on it, because MacPorts does not wish to become the developers of other people's software; we only want to package and distribute existing software. (Homebrew developers have expressed similar sentiments.)

Ryan, let me know if you're working on this. If not, I'll take a look.

[ryandesign]

Hi @mascguy, no I haven't worked on it. If you want to that'd be great.

[mascguy]

It's a balancing act of how useful a feature is compared to how hard it is to make sure it doesn't break in weird ways. I'll leave it in the patches dir for now for those that might find it useful.

Wayne, I'm currently in the process of understanding the rsync code, and updating the patch for 3.2.3. I'm taking it slow, and being very painstaking about it, given how critical rsync is to everyone.

But if the updated patch includes additional tests and such (assuming those aren't already in the existing one), would that help?

[WayneD]

More tests is always a good thing for those that want to use the patch. It wouldn't help me at all, since I have no way to test it.

[ctipper]

I'm curious does this affect APFS? I was surprised to find the other day that HFS+ is deprecated in macOS Big Sur and Disk Utility no longer supports Core Storage Volumes (encrypted HFS+). It seems difficult to invest in this if support from Apple is going away on the new platform.

[ryandesign]

Support for HFS+ is going away, but APFS supports the same "HFS compression" as HFS+ does.

[mascguy]

More tests is always a good thing for those that want to use the patch. It wouldn't help me at all, since I have no way to test it.

Wayne, if you could be provided access to a Mac VM, where you can build and test rsync... would that be sufficient?

[Haravikk]

Is it actually necessary to have access to a Mac to test this? If HFS compressed data is just an extended attribute then can it not be compared between a sample file and an rsync'd copy to see if it's an exact match with this feature enabled?

More generally; how much (if any) of the code is actually Mac specific? Are there system calls required in extracting the compressed data or can that be retrieved via tools like xattr if you know what to look for?

I'm just wondering if maybe this might not need to be maintained as a Mac/HFS/APFS-specific feature and could instead be made into a more general-purpose feature telling rsync when not to transfer the file content (e.g- --filecontent-xattr com.apple.decmpfs would enable --xattr and tell rsync to transfer the contents of that xattr instead of the normal file content), and would be functionally identical to --hfs-compression or --protect-decmpfs?

[diimdeep]

Can anyone clarify what are implications of not supporting this, does still work, or one can expect data loss ?

For example if compression on HFS+ files was enabled with afsctool -c -l9 /Volumes/1/ and you decided to rsync -alhcuXD --info=progress2,stats /Volumes/1/ /Volumes/2/

[Haravikk]

For example if compression on HFS+ files was enabled with afsctool -c -l9 /Volumes/1/ and you decided to rsync -alhcuXD --info=progress2,stats /Volumes/1/ /Volumes/2/

This command should cause the files to transfer safely, but the files on the receiving side (/Volumes/2) will be uncompressed but otherwise correct.

Basically macOS hides the special com.apple.decmpfs attribute that it uses for compression, as a result rsync isn't aware of it – rsync requests the file data, and macOS invisibly decompresses it. When you use xattr -l /some/file this attribute won't be listed, but I believe you can inspect it with xattr -p com.apple.decmpfs /some/file (could be wrong though, I can't test xattr right now, it might only be the system calls that can do this).

In theory rsync support is just a case of telling it to request that attribute, and send it if it's there (otherwise request file data as normal), but I'm unclear how difficult that might be in practice, or if there are other concerns like the file's size.