[GLUTEN-6920][CORE][VL] New APIs and refactors to allow different backends / components to be registered and used #8143

zhztheplayer · 2024-12-04T08:44:00Z

This is the major API layer change for #6920.

The PR introduces a new concept Component int Gluten and adjusts the role of current concept Backend.

After the change, Backend API will become a special variant of Component API. Details are as below:

Dependency
- Component should have at least one parent. Each parent can either be another Component or Backend.
- Backend doesn't have any parents.
Component and Backend instances are used as exactly the same manner in Gluten. Gluten treats them all as regular entities that inject custom code into Gluten.
Having Component's parents defined, Gluten will figure out a fixed order / priority of using the components. For exmaple:
- A component's parent's loading APIs, namely onDriverStart / onExecutorStart, will be guaranteed to call EARLIER than the same APIs of the child.
- A component's parent's rule injection API, namely injectRules, will be guaranteed to call LATER than the same API of the child.
For legacy reason, only one SubstraitBackend will be allowed to register at the same time. Which means, CH backend and Velox backend are still not allowed to load at the same time. When they are both loaded, error will be thrown. See code.

In subsequent PRs of #6920, we will:

Make iceberg / hudi / delta implement component API, remove the previous rule injection points used by their code.
Make uniffle / celeborn implement component API, remove the previous rule injection points by used their code.
Add an example backend that can work together with Velox backend.

Non-goals:

This doesn't tend to make Velox backend and CH backend work together.

github-actions · 2024-12-04T08:44:17Z

#6920

github-actions · 2024-12-04T08:44:32Z

Run Gluten Clickhouse CI on x86

github-actions · 2024-12-04T09:00:29Z

Run Gluten Clickhouse CI on x86

github-actions · 2024-12-04T09:08:29Z

Run Gluten Clickhouse CI on x86

github-actions · 2024-12-05T01:42:14Z

Run Gluten Clickhouse CI on x86

github-actions · 2024-12-05T01:43:59Z

Run Gluten Clickhouse CI on x86

github-actions · 2024-12-05T01:58:48Z

Run Gluten Clickhouse CI on x86

github-actions · 2024-12-05T03:08:07Z

Run Gluten Clickhouse CI on x86

github-actions · 2024-12-05T03:50:29Z

Run Gluten Clickhouse CI on x86

github-actions · 2024-12-05T06:26:34Z

Run Gluten Clickhouse CI on x86

github-actions · 2024-12-05T06:36:09Z

Run Gluten Clickhouse CI on x86

PHILO-HE · 2024-12-05T08:28:38Z

Non-goals:
This doesn't tend to make Velox backend and CH backend work together.

This is for clarifying the non-goal of this pr or mixed backend work? According to #6920, it seems making different backends co-work at runtime is the long-term goal. Just a bit confused.

zhztheplayer · 2024-12-05T08:43:07Z

Non-goals:
This doesn't tend to make Velox backend and CH backend work together.

This is for clarifying the non-goal of this pr or mixed backend work. According to #6920, it seems making different backends co-work at runtime is the long-term goal. Just a bit confused.

Good question, thanks.

Velox / CH may be possible to work together but it may require for considerable effort on refactoring. I am not sure whether any user is interested with this kind of combination now so I didn't intentionally put it as a goal for this PR (will make a note in issue #6920 too). I think we can choose to have another individual thread for VL+CH working together when it's proved to be worth to have our investment.

PHILO-HE

Just some trivial comments. Thanks!

PHILO-HE · 2024-12-05T09:40:00Z

gluten-core/src/main/scala/org/apache/gluten/backend/Component.scala

+
+/**
+ * The base API to inject user-defined logic to Gluten. To register a component, its implementations
+ * should be placed to Gluten's classpath with a Java service file. Gluten will discover all the


So the implementation of external component may depend on this trait in its codebase? Maybe, better to place such interface code in a new module, then it can be introduced into external project as a dependency, instead of unnecessarily depending on gluten-core.

If it makes sense, it's ok to do this in the future, considering we are still in the early stage to support external component.

Maybe, better to place such interface code in a new module

If you think the APIs can be isolated from concrete implementations, this is actually what I have been doing for long time on the module gluten-core. You can refer to gluten-core's latest code structure here to see if we are aligned already.

gluten-core will become the thin bridge between Gluten backends and Spark's plugin API, with all the predefined arch level features including the enhanced query optimizer and other runtime essentials.

@zhztheplayer, I see. We are on the same page.

PHILO-HE · 2024-12-05T09:49:49Z

gluten-core/src/main/scala/org/apache/gluten/backend/Component.scala

+     * requirement from component A to component B.
+     */
+    // format: on
+    def sorted(): Seq[Component] = synchronized {


Does cyclic dependency possibly happen? It would be better to leave some comments to clarify it, e.g., cyclic dependency is not allowed.

Just note error will be reported for cyclic case. Please document it at above.

Thanks and I've moved/added documents to the public API Component.sorted. This one is internal.

PHILO-HE · 2024-12-05T09:55:20Z

gluten-core/src/main/scala/org/apache/gluten/backend/Component.scala

+  private class Graph {
+    import Graph._
+    private val registry: Registry = new Registry()
+    private val requirements: mutable.Buffer[(Int, Class[_ <: Component])] = mutable.Buffer()


Use depend/dependency to describe the requirement concept? Not quite sure which one is logically clearer.

Do parent / requirement sound really unclear? If so I'd make them dependency which is also OK to me.

github-actions · 2024-12-06T01:34:26Z

Run Gluten Clickhouse CI on x86

github-actions · 2024-12-06T02:04:43Z

Run Gluten Clickhouse CI on x86

PHILO-HE · 2024-12-06T03:53:11Z

gluten-core/src/main/scala/org/apache/gluten/backend/Component.scala

+   *
+   *   1. [component-B, component-A, component-C]
+   *   2. [component-C, component-B, component-A]
+   *   3. [component-B, component-C, component-A]


Just some questions need clarification from my superficial understanding.

The class loading order potentially decides the occurrence of any one of the above ordered legal combination?

If A depends on both B & C and there is no dependency between B and C, I assume the above 2nd/3rd combinations are acceptable. Is it possible that they produce different behaviors? For example, both B and C override some rule in A. The order of B and C will determine which one actually overrides that rule?

The class loading order potentially decides the occurrence of any one of the above ordered legal combination?

That's possible so far. We could introduce some kind of configurations to determine on a fixed load order but that's messing things up so we should only do that when really necessary.

For example, both B and C override some rule in A. The order of B and C will determine which one actually overrides that rule?

If A depends on B & C, then it means A could override rules in B or C, not the opposite. Say if rule in A tends to create AScanExec while rules in B / C tend to create BScanExec / CScanExec, then A's rule will be applied in higher precedence so AScanExec will be eventually used.

This is the case in heuristic query planner. If RAS planner is used, the rules are loaded with exactly the same importance. Using which one is subject to the cost model.

PHILO-HE

Thanks!

jackylee-ch · 2024-12-06T07:48:01Z

gluten-arrow/src/main/java/org/apache/gluten/init/NativeBackendInitializer.java

 import java.util.concurrent.atomic.AtomicBoolean;

 import scala.runtime.BoxedUnit;

 // Initialize native backend before calling any native methods from Java side.
 public final class NativeBackendInitializer {
  private static final Logger LOG = LoggerFactory.getLogger(NativeBackendInitializer.class);
-  private static final AtomicBoolean initialized = new AtomicBoolean(false);
+  private static final Map<String, NativeBackendInitializer> instances = new ConcurrentHashMap<>();


Since This doesn't tend to make Velox backend and CH backend work together, why we need a backend map here?

This change is not tended actually. It's just because the module gluten-arrow can no longer get a precise backend name because it's decoupled with gluten-substrait so it may see multiple backends loaded. There is another option to move this class to backends-velox so no such change is needed.

zhztheplayer · 2024-12-06T08:29:00Z

I am merging this. Let me know if any other questions @jackylee-ch

Thanks for you guys' review!

github-actions bot added CORE works for Gluten Core VELOX RSS CLICKHOUSE labels Dec 4, 2024

zhztheplayer changed the title ~~[GLUTEN-6920][CORE][VL] Allow different backends / components to be registered and used~~ [GLUTEN-6920][CORE][VL] APIs to allow different backends / components to be registered and used Dec 4, 2024

zhztheplayer changed the title ~~[GLUTEN-6920][CORE][VL] APIs to allow different backends / components to be registered and used~~ [GLUTEN-6920][CORE][VL] New APIs and refactors to allow different backends / components to be registered and used Dec 4, 2024

zhztheplayer marked this pull request as ready for review December 5, 2024 01:59

zhztheplayer mentioned this pull request Dec 5, 2024

[CORE] Mixed backend(s) #6920

Open

PHILO-HE reviewed Dec 5, 2024

View reviewed changes

zhztheplayer added 8 commits December 6, 2024 09:32

fixup

359fbfa

fixup

fc74d49

fixup

aa10d69

fixup

9ec300f

fixup

0401172

fixup

4d653b7

fixup

114576d

fixup

f5c72fb

zhztheplayer added 18 commits December 6, 2024 09:32

fixup

68dc6db

fixup

fddf1a5

fixup

ec7666c

fixup

3aabfbd

fixup

fe42efa

fixup

9660086

fixup

c264925

fixup

41249b2

fixup

b4ded02

fixup

4d6d558

fixup

8385e11

fixup

7942f0c

fixup

e6a1e56

fixup

1b2ea63

fixup

3d6eb5c

fixup

de1326c

fixup

0094249

fixup

e83d7f4

zhztheplayer force-pushed the wip-component branch from 2394166 to e83d7f4 Compare December 6, 2024 01:33

zhztheplayer added 2 commits December 6, 2024 10:02

fixup

93de387

fixup

3c272ac

PHILO-HE reviewed Dec 6, 2024

View reviewed changes

PHILO-HE approved these changes Dec 6, 2024

View reviewed changes

jackylee-ch reviewed Dec 6, 2024

View reviewed changes

zhztheplayer merged commit 232ce55 into apache:main Dec 6, 2024
47 of 48 checks passed

This was referenced Dec 9, 2024

[CORE] Query planner: Simplify validator FallbackByNativeValidation #8177

Merged

[CORE][VL][CH] Make Iceberg code implement component API #8192

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GLUTEN-6920][CORE][VL] New APIs and refactors to allow different backends / components to be registered and used #8143

[GLUTEN-6920][CORE][VL] New APIs and refactors to allow different backends / components to be registered and used #8143

zhztheplayer commented Dec 4, 2024 •

edited

Loading

github-actions bot commented Dec 4, 2024

github-actions bot commented Dec 4, 2024

github-actions bot commented Dec 4, 2024

github-actions bot commented Dec 4, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

PHILO-HE commented Dec 5, 2024 •

edited

Loading

zhztheplayer commented Dec 5, 2024

PHILO-HE left a comment

PHILO-HE Dec 5, 2024 •

edited

Loading

zhztheplayer Dec 6, 2024

PHILO-HE Dec 6, 2024

PHILO-HE Dec 5, 2024

PHILO-HE Dec 5, 2024

zhztheplayer Dec 6, 2024 •

edited

Loading

PHILO-HE Dec 5, 2024

zhztheplayer Dec 6, 2024

github-actions bot commented Dec 6, 2024

github-actions bot commented Dec 6, 2024

PHILO-HE Dec 6, 2024

zhztheplayer Dec 6, 2024

PHILO-HE left a comment

jackylee-ch Dec 6, 2024

zhztheplayer Dec 6, 2024

zhztheplayer commented Dec 6, 2024

[GLUTEN-6920][CORE][VL] New APIs and refactors to allow different backends / components to be registered and used #8143

[GLUTEN-6920][CORE][VL] New APIs and refactors to allow different backends / components to be registered and used #8143

Conversation

zhztheplayer commented Dec 4, 2024 • edited Loading

github-actions bot commented Dec 4, 2024

github-actions bot commented Dec 4, 2024

github-actions bot commented Dec 4, 2024

github-actions bot commented Dec 4, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

PHILO-HE commented Dec 5, 2024 • edited Loading

zhztheplayer commented Dec 5, 2024

PHILO-HE left a comment

Choose a reason for hiding this comment

PHILO-HE Dec 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhztheplayer Dec 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Dec 6, 2024

github-actions bot commented Dec 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PHILO-HE left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhztheplayer commented Dec 6, 2024

zhztheplayer commented Dec 4, 2024 •

edited

Loading

PHILO-HE commented Dec 5, 2024 •

edited

Loading

PHILO-HE Dec 5, 2024 •

edited

Loading

zhztheplayer Dec 6, 2024 •

edited

Loading