Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[BUG] [New Executor] Drain channel in limit sink (#2401)
Simple scan + limit currently breaks on new executor. This is because the sink stage runner does not try to drain the receiving channel after the task scheduler has completed execution. Example failure: ``` import daft import tempfile import daft.context with tempfile.TemporaryDirectory() as td: # write to parquet without AQE and native executor because native executor does not support writes yet daft.context.set_execution_config(enable_aqe=False, enable_native_executor=False) df = daft.from_pydict({"a": [1]}) df.write_parquet(td) # read from parquet with AQE and native executor enabled daft.context.set_execution_config(enable_aqe=True, enable_native_executor=True) df = daft.read_parquet(td) df.show() ``` Error: ``` daft.exceptions.DaftCoreException: DaftError::ValueError Need at least 1 MicroPartition to perform concat ``` Todo: - Add better error handling
- Loading branch information