Fix merge fails when partition is required

[tnk-ysk]

resolves #792
docs dbt-labs/docs.getdbt.com/#

Problem

Trying to migrate a BQ stored procedure to DBT where the partition on the target table is required. I receive this error on the table the requires a partition:

BadRequest("
Cannot query over table 'test' without a filter over column(s) 'date' that can be used for partition elimination")

Solution

Avoid require partition filter by adding partition is null or partition is not null to the merge condition.
This condition does not change the target record.

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

[cla-bot]

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @tnk-ysk

[cla-bot]

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @tnk-ysk

[tnk-ysk]

I'm wondering if I should add the partition_filld parameter to get_merge_sql and modify default__get_merge_sql instead of adding bigquery__get_merge_sql.
Please let us know what you think.

[github-christophe-oudar]

Great contribution @tnk-ysk!
Could it be possible to extract

{% if partition_config and config.get('require_partition_filter') -%}
            {% set avoid_require_partition %}
                DBT_INTERNAL_DEST.{{ partition_config.field }} IS NULL OR DBT_INTERNAL_DEST.{{ partition_config.field }} IS NOT NULL
            {% endset %}
            {% do predicates.append(avoid_require_partition) %}
        {%- endif -%}

And add that condition to the predicates so that you can combine it with https://github.com/dbt-labs/dbt-bigquery/pull/994/files?

[tnk-ysk]

@github-christophe-oudar
Thank you for your review.
I extracted the process and made it into a macro, so please check it.

I will to fix #994 after this PR is merged.

[github-christophe-oudar]

@tnk-ysk it looks better!
Then wouldn't you be able to keep the default macro and pass the predicates array as the incremental_predicates param from the get_merge_sql macro?
My point is that if someone has overridden the get_merge_sql macro inside its dbt project for any reason, he won't benefit from the change if we redefine it that way.

[tnk-ysk]

@github-christophe-oudar
Thank you for your wonderful comment.
I fixed it. The predicate is also added if unique_key is not set, but there is no negative effect.

Also made the macro return the predicate instead of making destructive change.

[github-christophe-oudar]

LGTM, I'll let dbt Labs team takes over the review as I don't have any rights.