Remove corrupt/missing sectors

[n8maninger]

Description

Extends hostd's data integrity functionality allowing early detection and notification of corrupt/missing sectors.

Missing sectors are sectors whose volume is not currently loaded. This should be considered "temporary," as the volume may become available later. The missing sector references should be permanently removed if the volume is later forcibly removed. Corrupt sectors are sectors whose data does not match the expected sector root. This shouldn't happen during regular operation but is possible if a disk malfunctions. References to corrupt sectors should be removed as soon as they are detected, enabling the sector to be reuploaded and repaired.

When a volume is removed, sectors are migrated to a new volume. If the sector is missing or corrupt, removal is immediately stopped. However, when a volume is "force" removed, the integrity check is ignored and references to missing or corrupt sectors are migrated within the database to new locations. However, the sectors do not physically exist on disk. Instead, the reference to the sector should be removed.

• remove references to missing and corrupt sectors during forced volume removal
• remove references to corrupt sectors during volume shrink
• remove references to corrupt sectors during downloads
• remove references to corrupt sectors during integrity checks
• show corrupt and missing sectors in the UI

Version

No response

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

No response

Anything else?

No response

[n8maninger]

Added support for removing lost/corrupt sectors during force remove in #318

[timrettop]

Can I make a request associated to this? I can create a separate ticket if you prefer.

I occassionally have concerns about data integrity on certain volumes, it would be nice to be able to check all contracts that have data on that volume. I guess this would not be as useful if there is automated integrity checks, but I'm not sure where development plans are on that.

Oh and thank you so much for this work!