You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current IcebergSplitSource::getNextBatch is synchronous by returning a completed future. I wonder if there're specific reasons why it's implemented this way and whether we can make it asynchronous.
Some background: we made our iceberg use a blocking queue when planning files (in order to reduce memory consumption). Then if multiple-stage query run concurrently in trino, it can run into some kind of dead lock, because the stage scheduler can be blocked at IcebergSplitSource::getNextBatch while the thread pool used in planning files is occupied by other stages.
The text was updated successfully, but these errors were encountered:
The current
IcebergSplitSource::getNextBatch
is synchronous by returning a completed future. I wonder if there're specific reasons why it's implemented this way and whether we can make it asynchronous.Some background: we made our iceberg use a blocking queue when planning files (in order to reduce memory consumption). Then if multiple-stage query run concurrently in trino, it can run into some kind of dead lock, because the stage scheduler can be blocked at
IcebergSplitSource::getNextBatch
while the thread pool used in planning files is occupied by other stages.The text was updated successfully, but these errors were encountered: