Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crm bump-recipe should consider using a stream to calculate some SHA-256 hashes #277

Open
schuylermartin45 opened this issue Dec 18, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request source::maintainer

Comments

@schuylermartin45
Copy link
Collaborator

@JeanChristopheMorinPerso discussed this offline.

We're currently downloading source artifacts for two main reasons:

  1. We need to calculate the hashes of source archives when upgrading a recipe file
  2. We eventually want to start scanning source code and doing other sorts of operations to calculate dependencies

Now the current mechanism will scale for the vast majority of conda recipe files/feedstocks.

However, there are some very large artifacts out there that contain massive build artifacts and neural network models, potentially in the gigabytes.

For such large projects, streaming the the archive bytes to the block cipher and ditching the processed bytes makes way more sense.

@schuylermartin45 schuylermartin45 added enhancement New feature or request source::maintainer to sort Needs additional investigation labels Dec 18, 2024
@schuylermartin45 schuylermartin45 self-assigned this Dec 18, 2024
@schuylermartin45 schuylermartin45 removed the to sort Needs additional investigation label Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request source::maintainer
Projects
Status: Backlog
Development

No branches or pull requests

1 participant