-
Notifications
You must be signed in to change notification settings - Fork 174
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEAT]: expr simplifier #3393
[FEAT]: expr simplifier #3393
Conversation
src/daft-logical-plan/src/optimization/rules/simplify_expressions.rs
Outdated
Show resolved
Hide resolved
CodSpeed Performance ReportMerging #3393 will improve performances by 53.05%Comparing Summary
Benchmarks breakdown
|
src/daft-logical-plan/src/optimization/rules/simplify_expressions.rs
Outdated
Show resolved
Hide resolved
I'm going to split out "add simplification for AND exprs" into a separate PR. |
src/daft-logical-plan/src/optimization/rules/simplify_expressions.rs
Outdated
Show resolved
Hide resolved
src/daft-logical-plan/src/optimization/rules/simplify_expressions.rs
Outdated
Show resolved
Hide resolved
src/daft-logical-plan/src/optimization/rules/simplify_expressions.rs
Outdated
Show resolved
Hide resolved
src/daft-logical-plan/src/optimization/rules/simplify_expressions.rs
Outdated
Show resolved
Hide resolved
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3393 +/- ##
==========================================
+ Coverage 77.42% 77.49% +0.07%
==========================================
Files 702 703 +1
Lines 85210 85641 +431
==========================================
+ Hits 65973 66371 +398
- Misses 19237 19270 +33
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, just have a small comment
// we want to simplify expressions first to make the rest of the rules easier | ||
rule_batches.push(RuleBatch::new( | ||
vec![Box::new(SimplifyExpressionsRule::new())], | ||
RuleExecutionStrategy::FixedPoint(Some(3)), | ||
)); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would put this under the rewrite rules since in my mind, the plan is in an invalid state before the rewrite rules are applied, so the rewrite rules should not rely on any optimizer rules but the optimizer rules can make assumptions about the validity of the plan.
still todo:
- [ ] add simplification forAND
exprsNote for reviewers:
This bypasses the expr simplifier rule if the source is
SQLScanOperator
it was simplifying the queries in a way that weren't working with connectorx. I haven't been able to debug exactly what was going on, but I think it's pretty ok to just skip this optimization for this source for now.Also, wanted to implement a const expr evaluator, but since our
eval_expressions
is associated withdaft_table::Table
, this kind of optimization is not currently possible, and some refactoring is needed.