Core, Spark: Rewrite data files with high delete ratio #11825
Conversation
| double deletedRecords = (double) Math.min(file.recordCount(), task.file().recordCount()); | ||
| double deleteRatio = deletedRecords / task.file().recordCount(); |
There was a problem hiding this comment.
[question] wondering if adjusting deleteRecords by taking a min is the right thing for SplitScanTask ? as the recordCount is records in data files.
my understanding was that delete ratio can be always:
file.recordCount() / task.file().recordCount()
IIUC, the following is Maths for this :
long[] splitOffsets = splitOffsets(file);
long splitOffset = splitOffsets != null ? splitOffsets[0] : 0L;
// meaning this fileScan task contains only scannedFileFraction of the file in it
double scannedFileFraction = ((double) length) / (file.fileSizeInBytes() - splitOffset);
// number of deletes associated with this based on uniform distribution of deletes across the splitScan task
long deletedRecords = (scannedFileFraction * deleteFile.recordCount()) ;
long totalRecordWithoutDeletes = (scannedFileFraction * file.recordCount());
double deleteRatio = deletedRecords / totalRecordWithoutDeletes
deleteRatio = (scannedFileFraction * deleteFile.recordCount()) / (scannedFileFraction * file.recordCount())
deleteRatio = (scannedFileFraction * deleteFile.recordCount()) / (scannedFileFraction * file.recordCount())
deleteRatio = deleteFile.recordCount() / file.recordCount()
There was a problem hiding this comment.
I was planning on re-visiting the calculation after the holiday break (sorry forgot to put the PR up as a draft)
There was a problem hiding this comment.
No worries, Happy Holidays @nastra !
No description provided.