Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] Add ability to set global IOConfig #1710

Merged
merged 15 commits into from
Dec 12, 2023
Merged

Conversation

jaychia
Copy link
Contributor

@jaychia jaychia commented Dec 9, 2023

Planning configs:

  • Config flags that are used in LogicalPlan building and construction
  • Includes IOConfigs, and these are eagerly created and put onto the logical plans
  • This exists as a mutable global singleton, and are not propagated to remote workers since all planning work should happen client-side

Execution configs:

  • Config flags that are used in the execution chain (logical-to-physical-translation, physical-to-task-generation, task-scheduling, task-execution)
  • These should be "frozen" per-unique-execution, and used unambiguously throughout that execution

@github-actions github-actions bot added the enhancement New feature or request label Dec 9, 2023
Copy link

codecov bot commented Dec 9, 2023

Codecov Report

Merging #1710 (1d9e0ec) into main (668399b) will decrease coverage by 3.13%.
The diff coverage is 85.96%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1710      +/-   ##
==========================================
- Coverage   85.22%   82.10%   -3.13%     
==========================================
  Files          55       55              
  Lines        5517     5515       -2     
==========================================
- Hits         4702     4528     -174     
- Misses        815      987     +172     
Files Coverage Δ
daft/dataframe/dataframe.py 87.28% <100.00%> (+0.04%) ⬆️
daft/expressions/expressions.py 92.09% <100.00%> (ø)
daft/io/_csv.py 95.23% <100.00%> (+0.50%) ⬆️
daft/io/_json.py 100.00% <100.00%> (ø)
daft/io/_parquet.py 100.00% <100.00%> (ø)
daft/logical/builder.py 87.50% <100.00%> (-1.79%) ⬇️
daft/runners/pyrunner.py 94.18% <100.00%> (-1.73%) ⬇️
daft/runners/ray_runner.py 87.79% <100.00%> (-1.61%) ⬇️
daft/__init__.py 24.24% <0.00%> (-0.76%) ⬇️
daft/io/_iceberg.py 21.27% <0.00%> (ø)
... and 1 more

... and 10 files with indirect coverage changes

@jaychia jaychia changed the title [FEAT] Differentiate between "execution" configs and "planning" configs [FEAT] Add ability to set global IOConfig Dec 9, 2023
@jaychia jaychia force-pushed the jay/global-io-config-3 branch from 4e171db to d376a59 Compare December 9, 2023 04:16
@jaychia jaychia force-pushed the jay/global-io-config-3 branch from 0725739 to 490843a Compare December 12, 2023 03:41
@github-actions github-actions bot added the documentation Improvements or additions to documentation label Dec 12, 2023
@jaychia jaychia merged commit 97753a2 into main Dec 12, 2023
39 of 40 checks passed
@jaychia jaychia deleted the jay/global-io-config-3 branch December 12, 2023 04:49
@jaychia jaychia mentioned this pull request Jan 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant