Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] Add ability to set global IOConfig #1710

Merged
merged 15 commits into from
Dec 12, 2023
Prev Previous commit
Next Next commit
Add top-level imports
Jay Chia committed Dec 12, 2023

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
commit d9aead957d0739cdb1594bb972987b1cb679b13c
1 change: 1 addition & 0 deletions daft/__init__.py
Original file line number Diff line number Diff line change
@@ -54,6 +54,7 @@
# Daft top-level imports
###

from daft.context import set_execution_config, set_planning_config

Check warning on line 57 in daft/__init__.py

Codecov / codecov/patch

daft/__init__.py#L57

Added line #L57 was not covered by tests
from daft.convert import (
from_arrow,
from_dask_dataframe,

Unchanged files with check annotations Beta

or Dataframe operation (e.g. `daft.read_parquet()`).
"""
# Replace values in the DaftPlanningConfig with user-specified overrides
ctx = get_context()
old_daft_planning_config = ctx.daft_planning_config if config is None else config
new_daft_planning_config = old_daft_planning_config.with_config_values(

Check warning on line 196 in daft/context.py

Codecov / codecov/patch

daft/context.py#L194-L196

Added lines #L194 - L196 were not covered by tests
default_io_config=default_io_config,
)
ctx.daft_planning_config = new_daft_planning_config
return ctx

Check warning on line 201 in daft/context.py

Codecov / codecov/patch

daft/context.py#L200-L201

Added lines #L200 - L201 were not covered by tests
def set_execution_config(
) -> DataFrame:
from daft.iceberg.iceberg_scan import IcebergScanOperator
io_config = (

Check warning on line 75 in daft/io/_iceberg.py

Codecov / codecov/patch

daft/io/_iceberg.py#L75

Added line #L75 was not covered by tests
_convert_iceberg_file_io_properties_to_io_config(pyiceberg_table.io.properties)
if io_config is None
else io_config
)
io_config = context.get_context().daft_planning_config.default_io_config if io_config is None else io_config

Check warning on line 80 in daft/io/_iceberg.py

Codecov / codecov/patch

daft/io/_iceberg.py#L80

Added line #L80 was not covered by tests
multithreaded_io = not context.get_context().is_ray_runner
storage_config = StorageConfig.native(NativeStorageConfig(multithreaded_io, io_config))