feat: added script for searching corrupted (stub) files #16
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The check is performed in 3 steps: initial search, verification and dark object's search. In the fist step a sorted list of objects is read, for every file we get it's size, size of the last object, and also calculate the number of objects that constitute the file. If <file_size> != <object_size>*(<object_count> -1) + <last_obj_size>, the file is considered to be corrupted. In the second step list of obtained in the first step files is read and "full" live check is performed for every file. I.e. every file's object is statted and sizes are compared. This is necessary because some files may be only partially transferred during the dump collection. The final step is collection of orphaned objects. This is done since these objects can not be removed using rados striper.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The check is performed in 3 steps: initial search, verification and dark object's search.
In the fist step a sorted list of objects is read, for every file we get it's size, size of the last object, and also calculate the number of objects that constitute the file.
If <file_size> != <object_size>*(<object_count> -1) + <last_obj_size>, the file is considered to be corrupted.
In the second step list of obtained in the first step files is read and "full" live check is performed for every file. I.e. every file's object is statted and sizes are compared. This is necessary because some files may be only partially transferred during the dump collection.
The final step is collection of orphaned objects. This is done since these objects can not be removed using rados striper.