Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose "hints" to the planner #39

Open
ryannedolan opened this issue Jun 28, 2023 · 1 comment
Open

Expose "hints" to the planner #39

ryannedolan opened this issue Jun 28, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@ryannedolan
Copy link
Collaborator

#37 adds "subscription hints", which are passed to the control plane via templates. This is useful for things like kafka.numPartitions which are specific to the control plane.

For more advanced use-cases, we need the planner (i.e. PipelineRel etc) to have access to "hints" as well. This would enable users to influence the output (Flink) SQL generated for a pipeline. For example, hints could be used to dynamically enable/disable specific planner rules.

Calcite has a concept of "contexts" which might be useful here. See: https://calcite.apache.org/javadocAggregate/org/apache/calcite/plan/Context.html

@ryannedolan ryannedolan added the enhancement New feature or request label Jun 28, 2023
@ryannedolan
Copy link
Collaborator Author

In the DDL, we should support Calcite hints (e.g. /*+OPTIONS(foo=bar)*/). This seems to require forking Calcite, or at least copy-pasting a substantial chunk of it, because Calcite's JDBC driver does not provide a mechanism for registering custom hints (or really any hints). Alternatively, we could wrap our planner in a Calcite Framework, which does support custom hints.

Flink SQL also supports Calcite hints, tho I don't think we need to leverage them here. Our hints should get passed through as connector configuration via WITH (foo = bar).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant