Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-957] Refactor incremental materialization, using new incremental_strategy foundations #411

Open
Tracked by #9290
jtcohen6 opened this issue Jul 29, 2022 · 2 comments

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented Jul 29, 2022

Follow up to #402, which implemented the minimal version to keep the current code working

See dbt-labs/dbt-snowflake#196 for reference implementation:

  • Define valid_incremental_strategies
  • Add to materialization code:
    {#-- Get the incremental_strategy, the macro to use for the strategy, and build the sql --#}
    {% set incremental_strategy = config.get('incremental_strategy') or 'default' %}
    {% set incremental_predicates = config.get('incremental_predicates', none) %}
    {% set strategy_sql_macro_func = adapter.get_incremental_strategy_macro(context, incremental_strategy) %}
    {% set strategy_arg_dict = ({'target_relation': target_relation, 'temp_relation': tmp_relation, 'unique_key': unique_key, 'dest_columns': dest_columns, 'predicates': incremental_predicates }) %}
    {% set build_sql = strategy_sql_macro_func(strategy_arg_dict) %}
  • Take stock of remaining discrepancies between default + spark incremental materializations, identify opportunities for further refactor + eventual deduplication
@github-actions github-actions bot changed the title Refactor incremental materialization, using new incremental_strategy foundations [CT-957] Refactor incremental materialization, using new incremental_strategy foundations Jul 29, 2022
@github-actions
Copy link
Contributor

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

@dbeatty10
Copy link
Contributor

As-of v1.7, dbt-spark currently only supports these incremental strategies:

  • append
  • merge
  • insert_overwrite

So it currently does not support this one:

  • delete+insert

https://docs.getdbt.com/docs/build/incremental-models#supported-incremental-strategies-by-adapter

image

Since dbt-core dbt-adapters has default definitions of append and delete+insert, it should be trivial for dbt-spark to support both of those during implementation of this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants