Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[V1] Address 1.0 partiql-plan feedback and cleanup #1665

Merged
merged 19 commits into from
Dec 6, 2024
Merged

Conversation

RCHowell
Copy link
Member

@RCHowell RCHowell commented Nov 27, 2024

Issue

#1661

Description

This PR address a number of issues to (mostly) complete the 1.0 plan implementation. It includes,

  • interfaces/classes/enums converted to java
  • removes all default methods (uses abcs)
  • adds rewriter
  • cleanup factory methods
  • remove builders until reader to be finalized

Design

This is also included in part of the partiql-plan README

For the rule and strategy patterns to work, we need to model classes whose operands have a stable ordering;
so we have defined an abstract base class for all operators which holds operands and controls the access to them.
We use base implementations for state management and enforcing operands ordering; however we use interfaces for the
top-level of each domain. The abstract bases can be extended, and the operator/rel/rex interfaces can be implemented
directly.

What are operands?

  • An operand is an input to some operator.
  • An operator may have more than one operand e.g. join (left and right).
  • Rel operands are typically called "inputs"
  • Operands unify inputs since PartiQL bridges rel/rex domains.
  • Not all operators are operands e.g. the limit of RelLimit is a rex, but not an operand - it is an "arg"

Why interfaces for top-level domain entities and not abstract base classes?

  • We don’t want to force materialization of operands (consider wrapping a serde class)
  • We don’t want to force holding state (aka having to call super constructors)
  • The operator/rel/rex should be flexible for extension and the interface is the most open-ended / non-prescriptive.

Why abstract base classes for individual operators and not interfaces?

  • Enforce operands ordering is the primary reason; also can memoize operands.
  • Hold state such as mutable tags and computed types.
  • We want the standard operators to be prescriptive but not limiting.

License Information

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@RCHowell RCHowell mentioned this pull request Nov 27, 2024
8 tasks
@RCHowell
Copy link
Member Author

Rel ABCs done, moving on to Rex. Have lots of cleanup for builders and compiler, still draft.

@RCHowell
Copy link
Member Author

need rebase on #1658

Copy link

github-actions bot commented Dec 3, 2024

CROSS-ENGINE-REPORT ❌

BASE (LEGACY-V0.14.8) TARGET (EVAL-BF9E86B) +/-
% Passing 89.67% 94.39% 4.72% ✅
Passing 5287 5565 278 ✅
Failing 609 50 -559 ✅
Ignored 0 281 281 🔶
Total Tests 5896 5896 0 ✅

Testing Details

  • Base Commit: v0.14.8
  • Base Engine: LEGACY
  • Target Commit: bf9e86b
  • Target Engine: EVAL

Result Details

  • ❌ REGRESSION DETECTED. See Now Failing/Ignored Tests. ❌
  • Passing in both: 2643
  • Failing in both: 17
  • Ignored in both: 0
  • PASSING in BASE but now FAILING in TARGET: 3
  • PASSING in BASE but now IGNORED in TARGET: 108
  • FAILING in BASE but now PASSING in TARGET: 180
  • IGNORED in BASE but now PASSING in TARGET: 0

Now FAILING Tests ❌

The following 3 test(s) were previously PASSING in BASE but are now FAILING in TARGET:

Click here to see
  1. undefinedUnqualifiedVariableWithUndefinedVariableBehaviorMissing, compileOption: PERMISSIVE
  2. undefinedUnqualifiedVariableIsNullExprWithUndefinedVariableBehaviorMissing, compileOption: PERMISSIVE
  3. undefinedUnqualifiedVariableIsMissingExprWithUndefinedVariableBehaviorMissing, compileOption: PERMISSIVE

Now IGNORED Tests ❌

The complete list can be found in GitHub CI summary, either from Step Summary or in the Artifact.

Now Passing Tests

180 test(s) were previously failing in BASE (LEGACY-V0.14.8) but now pass in TARGET (EVAL-BF9E86B). Before merging, confirm they are intended to pass.

The complete list can be found in GitHub CI summary, either from Step Summary or in the Artifact.

CROSS-COMMIT-REPORT ✅

BASE (EVAL-443C3EB) TARGET (EVAL-BF9E86B) +/-
% Passing 94.39% 94.39% 0.00% ✅
Passing 5565 5565 0 ✅
Failing 50 50 0 ✅
Ignored 281 281 0 ✅
Total Tests 5896 5896 0 ✅

Testing Details

  • Base Commit: 443c3eb
  • Base Engine: EVAL
  • Target Commit: bf9e86b
  • Target Engine: EVAL

Result Details

  • Passing in both: 5565
  • Failing in both: 50
  • Ignored in both: 281
  • PASSING in BASE but now FAILING in TARGET: 0
  • PASSING in BASE but now IGNORED in TARGET: 0
  • FAILING in BASE but now PASSING in TARGET: 0
  • IGNORED in BASE but now PASSING in TARGET: 0

@RCHowell RCHowell marked this pull request as ready for review December 3, 2024 23:24
@RCHowell RCHowell changed the title [DRAFT] Plan cleanup #1661 working PR [V1] Address 1.0 partiql-plan feedback and cleanup Dec 3, 2024
@RCHowell RCHowell requested a review from johnedquinn December 3, 2024 23:44
Copy link
Member

@johnedquinn johnedquinn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still have some files left to review, but getting the review out quicker to allow you to address feedback sooner.

Comment on lines 58 to 63
@NotNull
@Override
protected final List<Operator> operands() {
Rel c0 = getInput();
return List.of(c0);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I'm trying to figure out here is how this will play with the compilation process of strategies and matches. I know the issue you cut (#1664) mentions groups, but the Measures also have references to Rex's. This is also related to RexStruct, among other scenarios I encountered while reviewing.

Personally, I'm not opposed to passing a reference of the compiler to a strategy. Especially since we might also want to pass the execution mode (since we want to write our own strategies for some of the relations that have special handling for permissive mode). Or, allowing matches to hold general PlanNodes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying to make the distinction between operands and arguments clear. For this example (RelAgggregate) the input relation is the only operand and the measures are an argument. A pattern matches operand trees, but a particular node match can define custom predicate logic on any arguments.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the main reason I bring it up has to do with where getOperands() is used, namely within the whole compilation strategies approach. If someone were to write a strategy for this RelAggregate, they'd have access to the compiled input in the operand tree. But, the Measures are arguments in this scenario, and thus aren't compiled, even though they contain plan Rex's (not compiled). Hence, they can't be used.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing a compiler reference is easy enough

Copy link
Member

@johnedquinn johnedquinn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments/questions remaining. Once addressed, I can approve.

@RCHowell RCHowell merged commit f5c6eff into v1 Dec 6, 2024
14 checks passed
@RCHowell RCHowell deleted the v1-plan-java branch December 6, 2024 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants