[ADAP-492] [Feature] Support partition_by and cluster_by on python models when supplied in model configurations

[kalanyuz]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt-bigquery functionality, rather than a Big Idea better suited to a discussion

Describe the feature

This issue was previous raised before in #247 but is now marked as stale and closed.

Currently, Python models do not support table creation with partition_by and cluster_by when supplied to the model configuration, despite the spark-bigquery-connector having support for it with indirect save mode.

I believe we can support for both configurations by modifying materializations/table.sql

dbt-bigquery/dbt/include/bigquery/macros/materializations/table.sql

Line 109 in 1972d72

df.write \

to something like

df.write \
  .mode("overwrite") \
  .format("bigquery") \
  .option("writeMethod", "indirect").option("writeDisposition", 'WRITE_TRUNCATE') \
  {%- if partition_config.data_type | lower in ('date','timestamp','datetime') %}
  .option("partitionField", "{{- partition_config.field -}}") \
  .option("partitonType", "{{- partition_config.granularity -}}") \
  {%- endif %}
  {%- if raw_cluster_by is not none %}
  .option("clusteredFields", "{{- raw_cluster_by|join(',') -}}") \
  {%- endif %}
  .save("{{target_relation}}")
{% endmacro %}

Describe alternatives you've considered

No response

Who will this benefit?

All BigQuery users who are looking to create complex models using dataproc

Are you interested in contributing this feature?

yes

Anything else?

No response

[dbeatty10]

This would be good to have @kalanyuz 🤩

Would you be interested in opening up a PR with your solution?

[kalanyuz]

Definitely. What do I need?On Apr 25, 2023, at 23:35, Doug Beatty ***@***.***> wrote: This would be good to have @kalanyuz 🤩 Would you be interested in opening up a PR with your solution? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[dbeatty10]

Awesome @kalanyuz!

Here's an overview of the pull request process. After reading that, I'd suggest reading the entire contributing file top-to-bottom.

There's a lot of content in there, so please reach out if you get stuck and need any help!

[kalanyuz]

@dbeatty10 I've submitted a PR here

[jtcohen6]

(Much older issue for this: #247)