[CT-821] Allow passing through Cluster Configuration parameters via profiles.yml when using the thrift connection method

[VShkaberda]

Description

Allow passing through Cluster Configuration parameters via profiles.yml when using the thrift connection method.
This allows configuration of values that would need to be set via SET to be run in the Spark session and take effect on DBT models. Analogous feature for odbc has been implemented in this request.

Alternatives

Alternative is to use pre_hook (taken example) but this method doesn't work quite well and has disadvantages.

{{ config(
    materialized='table',
    pre_hook=['SET spark.sql.shuffle.partitions=1200'],
    ...
)}}

This feature can be implemented through passing Configuration parameters as configuration parameter of hive.connect() of pyHive.

Benefits

1. Parameters can be located in one place.
2. It allows to set parameters which can't be set through the pre-hook.
3. Sensitive parameters that don't suppose to be in code could be used (such as spark.hadoop.fs.s3a.access.key, spark.hadoop.fs.s3a.secret.key).

Our team may be contributing this feature. One issue is the naming in the profiles.yml: should it be server_side_parameters to make it matching used in odbc or configuration to make it matching used in hive.connect(). Or both name should be allowed?

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[hanna-liashchuk]

Hi,
I've prepared a PR for this #577

[Fleid]

Hey @hanna-liashchuk, could you please check this discussion, I'm trying to regroup all the threads on that topic in one place.
Please let me know if the plan works for you, before we can decide to move forward with this specific issue.

[JCZuurmond]

Hi @VShkaberda , have you considered the SQL hints. For the repartition example in the issue I recommend the hints over setting the configuration as you might want to change this configuration parameter for different models.

[VShkaberda]

@JCZuurmond I appreciate your hint. My case is more common, e.g. passing spark.hadoop.fs.s3a.access.key, spark.hadoop.fs.s3a.secret.key, spark.executor.memory etc.