[FEAT] Add streaming + parallel CSV reader, with decompression support. #1501
Conversation
clarkzinzow
changed the title
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #1501 +/- ##
==========================================
+ Coverage 74.74% 74.76% +0.01%
==========================================
Files 60 60
Lines 6118 6130 +12
==========================================
+ Hits 4573 4583 +10
- Misses 1545 1547 +2
|
clarkzinzow
force-pushed
the
clark/streaming-parallel-csv-read
branch
from
c29cbbe to
46ec9ef
Compare
| // default to Utf8 for conflicting datatypes (e.g bool and int) | ||
| DataType::Utf8 | ||
| } | ||
| } |
There was a problem hiding this comment.
is there any merging we have to do the the temporal types?
There was a problem hiding this comment.
I'm assuming that we'll have a variety of follow-ups there, yes.
| schema, | ||
| // Default buffer size of 512 KiB. | ||
| buffer_size.unwrap_or(512 * 1024), | ||
| // Default chunk size of 64 KiB. |
There was a problem hiding this comment.
Do these constants make sense given its a local buffered file?
There was a problem hiding this comment.
I have a TODO to benchmark locally and tweak these, but I'm assuming that tweaking these won't matter as much for local reads as they do for cloud reads. I can look into tweaking these if you'd like!
This reverts commit 8acf496.
|
@samster25 I reverted the estimated row size piping from schema inference, added a good bit more test coverage, and addressed your primary review comments, PTAL! |
…t are likely to not require reallocation.
This PR adds streaming + parallel CSV reading and parsing, along with support for streaming decompression. In particular, this PR:
The streaming + parallel reading + parsing leads to a 4-8x speed up over the pyarrow reader and the previous non-parallel reader when benchmarking large file (~1 GB) reads, while also resulting in lower memory utilization due to the streaming reading + parsing.
TODOs (follow-up PRs)