Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make IcebergSplitSource generate splits asynchronously #17631

Closed
lirui-apache opened this issue May 25, 2023 · 2 comments
Closed

Make IcebergSplitSource generate splits asynchronously #17631

lirui-apache opened this issue May 25, 2023 · 2 comments
Labels
iceberg Iceberg connector

Comments

@lirui-apache
Copy link

The current IcebergSplitSource::getNextBatch is synchronous by returning a completed future. I wonder if there're specific reasons why it's implemented this way and whether we can make it asynchronous.

Some background: we made our iceberg use a blocking queue when planning files (in order to reduce memory consumption). Then if multiple-stage query run concurrently in trino, it can run into some kind of dead lock, because the stage scheduler can be blocked at IcebergSplitSource::getNextBatch while the thread pool used in planning files is occupied by other stages.

@findinpath
Copy link
Contributor

This request is probably handled by apache/iceberg#10691

@lirui-apache
Copy link
Author

Yes, I think this is not needed anymore, closing it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
iceberg Iceberg connector
Development

Successfully merging a pull request may close this issue.

3 participants