Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] Support for correlated subqueries in SQL (not yet executable) #3304

Merged
merged 5 commits into from
Nov 19, 2024

Conversation

kevinzwang
Copy link
Member

@kevinzwang kevinzwang commented Nov 16, 2024

This PR adds support for converting SQL queries with correlated subqueries into LogicalPlans. It does not add the ability to execute queries with correlated subqueries, but if I am correct, this is the last large piece of support we need on the SQL side for TPC-H questions, and most of the remaining work is plan rewriting, optimization, and translation.

I believe with the new alias_map value in SQLPlanner, we can actually simplify a lot of the logic in plan_aggregate_query and plan_non_agg_query but I will not attempt to do that in this PR.

Relevant for TPC-H questions 4, 17, 20, 21, 22.

Todo:

  • tests

@github-actions github-actions bot added the enhancement New feature or request label Nov 16, 2024
Copy link

codspeed-hq bot commented Nov 16, 2024

CodSpeed Performance Report

Merging #3304 will degrade performances by 20.38%

Comparing kevin/correlated-subqueries (17f815a) with main (1b84250)

Summary

❌ 1 regressions
✅ 16 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark main kevin/correlated-subqueries Change
test_iter_rows_first_row[100 Small Files] 312.1 ms 392 ms -20.38%

@kevinzwang
Copy link
Member Author

I think I broke sql_expr, will try to fix that tomorrow

@kevinzwang kevinzwang marked this pull request as draft November 16, 2024 05:03
Copy link

codecov bot commented Nov 19, 2024

Codecov Report

Attention: Patch coverage is 78.26087% with 35 lines in your changes missing coverage. Please review.

Project coverage is 77.42%. Comparing base (84db665) to head (79d8b03).
Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
src/daft-sql/src/planner.rs 87.60% 15 Missing ⚠️
src/daft-dsl/src/expr/mod.rs 16.66% 10 Missing ⚠️
src/daft-table/src/lib.rs 70.00% 6 Missing ⚠️
src/daft-schema/src/schema.rs 0.00% 3 Missing ⚠️
src/daft-logical-plan/src/partitioning.rs 0.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3304      +/-   ##
==========================================
- Coverage   77.55%   77.42%   -0.14%     
==========================================
  Files         668      676       +8     
  Lines       82268    82660     +392     
==========================================
+ Hits        63807    64003     +196     
- Misses      18461    18657     +196     
Files with missing lines Coverage Δ
src/daft-dsl/src/lib.rs 100.00% <ø> (ø)
src/daft-dsl/src/optimization.rs 98.11% <100.00%> (ø)
src/daft-logical-plan/src/ops/project.rs 62.97% <100.00%> (ø)
src/daft-sql/src/functions.rs 80.39% <ø> (ø)
src/daft-sql/src/lib.rs 100.00% <100.00%> (ø)
src/daft-sql/src/table_provider/mod.rs 53.70% <ø> (ø)
src/daft-logical-plan/src/partitioning.rs 44.28% <0.00%> (ø)
src/daft-schema/src/schema.rs 91.09% <0.00%> (-1.12%) ⬇️
src/daft-table/src/lib.rs 84.26% <70.00%> (-0.38%) ⬇️
src/daft-dsl/src/expr/mod.rs 71.88% <16.66%> (+1.49%) ⬆️
... and 1 more

... and 17 files with indirect coverage changes

---- 🚨 Try these New Features:

@kevinzwang kevinzwang marked this pull request as ready for review November 19, 2024 01:11
Copy link
Contributor

@universalmind303 universalmind303 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just one question, otherwise looks good.

@universalmind303 universalmind303 merged commit 25304eb into main Nov 19, 2024
43 of 44 checks passed
@universalmind303 universalmind303 deleted the kevin/correlated-subqueries branch November 19, 2024 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants