[CHORE] Write tpch parquet files one at a time

[colin-ho]

When you specify a num_parts parameter when generating tpch files. It will first generate num_parts CSVs, then read those CSVs and write to parquet using Daft.

However, write_parquet will not respect the input number of files, e.g. even if there are 16 input files there might only be 1 output file.

The fix here is to read and write 1 file at a time.

[codspeed-hq]

CodSpeed Performance Report

Merging #3396 will not alter performance

_{Comparing colin/gen-parquet (85fd788) with main (ec39dc0)}

Summary

✅ 17 untouched benchmarks

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.35%. Comparing base (3394a66) to head (85fd788).
Report is 42 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3396      +/-   ##
==========================================
- Coverage   77.35%   77.35%   -0.01%     
==========================================
  Files         685      685              
  Lines       83631    83637       +6     
==========================================
+ Hits        64695    64697       +2     
- Misses      18936    18940       +4     

see 6 files with indirect coverage changes

[graphite-app]

Graphite Automations

"Request reviewers once CI passes" took an action on this PR • (11/21/24)

1 reviewer was added to this PR based on Andrew Gazelka's automation.