Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] IO Morsel Size #3004

Closed
wants to merge 3 commits into from
Closed

[FEAT] IO Morsel Size #3004

wants to merge 3 commits into from

Conversation

colin-ho
Copy link
Contributor

@colin-ho colin-ho commented Oct 6, 2024

Change morsel size for IO (url download / upload) to be small. Users can still change it up by setting morsel_size in the execution config.

Todos:

  • Ideally I'd like users to be able to configure morsel size at a per-op level, but that will likely be a more involved API change.

@github-actions github-actions bot added the enhancement New feature or request label Oct 6, 2024
Copy link

codspeed-hq bot commented Oct 6, 2024

CodSpeed Performance Report

Merging #3004 will not alter performance

Comparing colin/morsel-size (753ac24) with main (73ff3f3)

Summary

✅ 17 untouched benchmarks

Copy link

codecov bot commented Oct 6, 2024

Codecov Report

Attention: Patch coverage is 90.24390% with 4 lines in your changes missing coverage. Please review.

Project coverage is 78.52%. Comparing base (73ff3f3) to head (753ac24).
Report is 32 commits behind head on main.

Files with missing lines Patch % Lines
src/common/daft-config/src/python.rs 40.00% 3 Missing ⚠️
...-execution/src/intermediate_ops/intermediate_op.rs 87.50% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3004      +/-   ##
==========================================
+ Coverage   78.14%   78.52%   +0.37%     
==========================================
  Files         610      610              
  Lines       72146    71769     -377     
==========================================
- Hits        56381    56354      -27     
+ Misses      15765    15415     -350     
Files with missing lines Coverage Δ
daft/context.py 79.48% <ø> (ø)
src/common/daft-config/src/lib.rs 87.50% <100.00%> (ø)
src/daft-dsl/src/expr/mod.rs 75.66% <100.00%> (+0.10%) ⬆️
src/daft-dsl/src/functions/scalar.rs 94.00% <100.00%> (+0.81%) ⬆️
src/daft-dsl/src/lib.rs 100.00% <ø> (ø)
src/daft-functions/src/uri/download.rs 83.84% <100.00%> (+0.38%) ⬆️
src/daft-functions/src/uri/upload.rs 65.26% <100.00%> (+0.63%) ⬆️
...ft-local-execution/src/intermediate_ops/project.rs 100.00% <100.00%> (ø)
src/daft-local-execution/src/lib.rs 89.74% <100.00%> (ø)
src/daft-local-execution/src/run.rs 89.84% <100.00%> (ø)
... and 2 more

... and 16 files with indirect coverage changes

@colin-ho colin-ho requested a review from samster25 October 7, 2024 01:44
@samster25
Copy link
Member

Hold off until everything else is merged.

We are thinking about updating url_download to it's own logical node rather than keep it as a pure expression in projection.

@samster25 samster25 closed this Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants