Lossy compression

[mdboom]

Consider compression and other alternative encodings of the binary data.

An interesting option is blosc, but lossy compression schemes and simple support for a standard loss-less compression (such as gz or something), may also make sense.

[perrygreenfield]

Are you familiar with the FITS lossy compression schemes that have been developed? These are geared to astronomical data. I'm not aware of any others that do this sort of thing. Most focus on perceptual issues (visual or auditory), but that's not what's important to most astronomers.

[mdboom]

Yes, I should have been explicit -- the lossy compression schemes in FITS do make a lot of sense for astronomy data, and should probably be brought over to FINF. The complaints lodged against them have mainly to do with how support for them is bolted on top of FITS, not the compression schemes themselves.

[perrygreenfield]

Indeed, there is a lot of duct tape in the FITS implementation. I'd argue it's another good example of an ugly solution due to the limited structures FITS supplies.

[embray]

I'll go out on a limb and say it's not even that ugly. Much of how it's done makes some sense. I just don't think they should have used the existing BINTABLE extension and instead defined a new extension type that is explicitly for compressed images. Sure, using BINTABLE meant it was "readable" by FITS readers that don't support the compression schema--but not readable in any useful sense.

[mdboom]

Renamed issue -- this is now for lossy compression methods. Lossless compression is basically a done deal.

[embray]

I guess one could make an argument for tiled lossless compression too, which isn't a done deal. But I think tiling / chunking should be handled independently from compression anyways. So yes, I agree.