[Feature] Python Model use Databricks Serverless Compute

[PhilippLange]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

I couldn't find any documentation on using Databricks Serverless Compute when executing a DBT Python model.

As the Databricks Serverless Compute feature is pretty new, it isn't well documented within the Databricks Docs. However the Databricks Terraform Provider supports creating Serverless Job Tasks. Within the Terraform Provider Docs I could find the following note:
If no job_cluster_key, existing_cluster_id, or new_cluster were specified in task definition, then task will executed using serverless compute.

Using this as a starting point, I tried fiddling around with submission_method all_purpose_cluster and job_cluster. In addition I did not set the cluster_id / jub_cluster_config (or setting them to None / empty Strings).

However DBT prevents me to submit a job like that, as I receive the following Error Messages:

Databricks `http_path` or `cluster_id` of an all-purpose cluster is required for the `all_purpose_cluster` submission method.

 `job_cluster_config` is required for the `job_cluster` submission method.

Therefore I think some DBT enhancement is necessary to support Databricks Serverless Compute.

This feature would greatly benefit our job execution times and I'd appreciate someone looking into this.

Describe alternatives you've considered

I tried to fiddle with the existing configuration possibilities in order to get dbt to submit a serverless run, which did not work.

Who will this benefit?

Anyone using DBT Python models with a Databricks backend due to reduced startup times.

Are you interested in contributing this feature?

I'm willing to contribute, if someone guides me in the right direction.

Anything else?

No response