[ADAP-492] Support partition_by and cluster_by on python models (#680)

[kalanyuz]

resolves #680
resolves #984

Description

Currently, Python models do not support table creation with partition_by and cluster_by when supplied to the model configuration, despite the spark-bigquery-connector having support for it with indirect save mode.

For more details, see #680

Checklist

• I have read the contributing guide and understand what's expected of me
• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
• I have run changie new to create a changelog entry

[kalanyuz]

@dbeatty10 Bump in case this is forgotten 🙇🏼

[dbeatty10]

Not forgotten @kalanyuz, but thank you for reaching out! ❤️

Next step is that this will be assigned to one of our engineers for review in an upcoming sprint.

[007vasy]

wen merge?

[kalanyuz]

@dbeatty10 Sorry for reaching out again on this one but our team is really looking forward to this patch being merged. Any update on this one? We are very close to patching this and link a local package in our pipeline at this point since it's been quite a while 😢

[mikealfare]

Thanks for your contribution @kalanyuz! I just left one comment on the functional code. Could you please also provide a test, or tests, for this new functionality? That would demonstrate this functionality is working as expected and help us ensure it doesn't break with future changes.

[kalanyuz]

@mikealfare I don't think the e2e tests that connects with BigQuery is being exposed to contributors and will be handled by the dbt-bigquery team? Otherwise I can take a shot when I have time.

[mikealfare]

There's a template test environment file here: https://github.com/dbt-labs/dbt-bigquery/blob/main/test.env.example. If you copy it to a file called test.env and supply the contents for a BQ instance to which you have access, the tests will run against that BQ instance. For more information, please refer to the Testing section of our contributor's guide here: https://github.com/dbt-labs/dbt-bigquery/blob/main/CONTRIBUTING.md.

[kalanyuz]

@mikealfare
Finally had the time to come back to this!
Added an e2e test for partitioned python models.

Have you considered breaking out the partition and cluster options as their own variables prior to inserting them into the df.write flow? I think it would make it more readable, especially given that we can't indent on python models.

Unfortunately these parameters (partitionField, partitionType, and clusteredFields) , cannot be set separately via spark.conf like bucket option. Or do you have some other ideas in mind?