Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add block archiving support #339

Open
Tracked by #309
a-saksena opened this issue Nov 13, 2024 · 0 comments
Open
Tracked by #309

feat: add block archiving support #339

a-saksena opened this issue Nov 13, 2024 · 0 comments
Assignees
Labels
Block Node Issues/PR related to the Block Node. Improvement Code changes driven by non business requirements P1 High priority issue. Required to be completed in the assigned milestone.
Milestone

Comments

@a-saksena
Copy link
Contributor

a-saksena commented Nov 13, 2024

Story

AS A Block Node User
I WANT to have my block files archived
SO THAT I will have a reduced storage cost

Tech Notes:

  • DEPENDS ON feat: add compression mode for BlockAsFileWriter #282

  • See Design a directory structure #125 where we have defined the design of the new block-as-file.

  • The scope of this issue is to add archiving support to the block files that will be written to disk.

  • We need to implement a process/thread, that will be going to the fs and will be archiving block files that are already written to disk.

  • The parameters of this process/thread should probably be configurable externally.

  • The archiving process looks something like this:

    1. We zip the block files we need to zip.
    2. We rename the old folder (the one we just zipped) in order for new reads that have happened after we have zipped will go to the zip instead (they will be unable to find the file in it's original path).
    3. We remove the old folder (the one we renamed) after a bit of time, giving a chance to reads to finish reading, thus ensuring that readers will not be interrupted.
@ata-nas ata-nas changed the title Add block archiving support feat: add block archiving support Nov 15, 2024
@ata-nas ata-nas added Improvement Code changes driven by non business requirements P1 High priority issue. Required to be completed in the assigned milestone. Block Node Issues/PR related to the Block Node. labels Nov 15, 2024
@ata-nas ata-nas self-assigned this Nov 15, 2024
@ata-nas ata-nas added this to the 0.3.0 milestone Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Block Node Issues/PR related to the Block Node. Improvement Code changes driven by non business requirements P1 High priority issue. Required to be completed in the assigned milestone.
Projects
None yet
Development

No branches or pull requests

2 participants