Dataproc Job options in model config

[gbmarc1]

Problem

For a given model, it was required to further configure the pyspark job. We are adding config properties to do so.

Solution

In the _submit_dataproc_job method of the ClusterDataprocHelper we are reading optional dataproc_pyspark_job field. This field is an object with the following schema.

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

[cla-bot]

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @gbmarc1

[nathaniel-may]

Hi @gbmarc1, thanks for opening this PR. The next steps are to create a feature request issue for this and link them so we can make any design considerations first.

[gbmarc1]

Thank you @nathaniel-may,
I created this issue.

[gbmarc1]

@nathaniel-may Just pinging

[Fleid]

Hey @gbmarc1, could you please add tests to the PR?

[gbmarc1]

@Fleid I would be happy to add test about the added functionality. But I cannot find any test for class ClusterDataprocHelper or the python_submission file. Could you point me to them if I am not missing anything?

[Fleid]

@dataders would you happen to know that? :)