Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add blosc compressor #1678

Closed
wants to merge 6 commits into from
Closed

Conversation

eschnett
Copy link

No description provided.

@eschnett eschnett requested a review from a team as a code owner November 12, 2023 17:16
@braingram
Copy link
Contributor

Thanks for working on this. Adding blosc compression to asdf seems like a great addition.

There is support for adding compression algorithms via the Compressor extension api:
https://asdf.readthedocs.io/en/latest/asdf/extending/compressors.html
As the extension api appears to be sufficient for adding blosc support I'd rather add blosc via an extension than add it to the asdf library directly (to keep the core library lightweight but also to make sure the extension api is complete).

I recently started a package asdf-compression: https://github.com/braingram/asdf-compression
which adds blosc support. The blosc compressor in that package could use some of the improvements that you've included in this PR:
https://github.com/braingram/asdf-compression/blob/main/asdf_compression/blsc/compressor.py
asdf-compression will be moved to asdf-format soon and if it's easier I can make the move now and we can work on that package.

@eschnett
Copy link
Author

@braingram For Blosc decompression: Is there a way to convert b"".join(data) to a pointer, i.e. an integer that I can pass to the Blosc library? This would avoid copying data while decompressing.

@eschnett
Copy link
Author

Closing this in favour of braingram/asdf-compression#1

@eschnett eschnett closed this Nov 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants