[PERF] Split parquet scan tasks into individual row groups #1799
Conversation
src/daft-scan/src/glob.rs
Outdated
| vec![DataFileSource::AnonymousDataFile { | ||
| path: path.to_string(), | ||
| chunk_spec: Some(ChunkSpec::Parquet(vec![rg as i64])), | ||
| size_bytes: Some(rgm.compressed_size() as u64), |
There was a problem hiding this comment.
I interpreted size_bytes as representing the file size, which stores the data compressed, so I'm pretty sure .compressed_size() is the right method, but if it's the size of the data, then we should use .total_byte_size().
|
The function See: https://github.com/Eventual-Inc/Daft/blob/main/src/daft-micropartition/src/micropartition.rs#L221 |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1799 +/- ##
=======================================
Coverage 85.47% 85.47%
=======================================
Files 55 55
Lines 6119 6119
=======================================
Hits 5230 5230
Misses 889 889
|
Conclusion of conversation with @samster25: don't do row group splitting when using python reader, since it is a legacy feature that we don't care about performance for |
|
Also let me know if I should write tests for non-local reads. |
| import daft | ||
|
|
||
| FILES = [ | ||
| "tests/assets/parquet-data/mvp.parquet", |
There was a problem hiding this comment.
We should also add some of the s3 file sources here too. I believe we all these files in s3 as well
There was a problem hiding this comment.
I just removed this test and added a fixture to test_reads_public_data.py
Benchmark results
All times averaged over 5 runs.
Single file read
Read one file in S3 using Ray with 4 worker nodes
Parquet file info:
Multi-file workflow
Read and aggregate (Dataframe.count()) 32 files in S3 using Ray with 4 worker nodes
Parquet file info:
(Averaged over 5 runs)