Feature/add quota project option #1345
Conversation
colin-rogers-dbt
added
user docs
|
@colin-rogers-dbt thank you for taking a look at this, do you know what the next steps with might look like? Am i right in thinking that if this change is merged it will be in the next release? Do you happen to know when that release is likely to be? |
|
@jcarpenter12 Sorry for the radio silence here, this is a great PR! Ideally we would like to add a functional test (likely similar to how we test location config to make sure we don't break this in the future. If that's something you're comfortable doing, go for it, otherwise someone on our team can take this over and add it. |
apologies for not getting back to you sooner @colin-rogers-dbt , didn't get a notification for some reason. I will take a look and check my understanding and get back to you. Thanks for your help! |
@colin-rogers-dbt i've had a crack at adding a functional test. I've tested it locally and it works with the temp GCP projects I setup to mirror what your CI jobs are doing. It has two use cases, one for if no quota project is set which is the default and another for whether an alternate quota project is set. I have tested locally and it passes for those two use cases but understand if this is not quite what you're after, happy to update it if so |
There was a problem hiding this comment.
Thanks for a thorough PR @jcarpenter12! I appreciate testing both the case with and without the new config, the latter is often missed. And thank you for the docs PR as well!
thanks @mikealfare I can see that the test is failing on the CI as the service account doesn't have the service usage consumer role on the |
## What are you changing in this pull request and why? <!-- Describe your changes and why you're making them. If related to an open issue or a pull request on dbt Core or another repository, then link to them here! To learn more about the writing conventions used in the dbt Labs docs, see the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md). --> References dbt-labs/dbt-bigquery#1343 dbt-labs/dbt-bigquery#1344 Adding docs to detail changes in PR dbt-labs/dbt-bigquery#1345 This adds detail about how to override the `quota_project` through the dbt profile. It also updates the information regarding the execution project. I believe that the execution project just sets where the BQ job is created and doesn't actually impact where it's billed from (I could be wrong on this, but from my testing I can see that the quota project is set from the environment not from the execution project override currently detailed in the docs). ## Checklist - [x] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." <!-- PRE-RELEASE VERSION OF dbt (if so, uncomment): - [ ] Add a note to the prerelease version [Migration Guide](https://github.com/dbt-labs/docs.getdbt.com/tree/current/website/docs/docs/dbt-versions/core-upgrade) --> <!-- ADDING OR REMOVING PAGES (if so, uncomment): - [ ] Add/remove page in `website/sidebars.js` - [ ] Provide a unique filename for new pages - [ ] Add an entry for deleted pages in `website/vercel.json` - [ ] Run link testing locally with `npm run build` to update the links that point to deleted pages --> --------- Co-authored-by: Leona B. Campbell <[email protected]> Co-authored-by: Matt Shaver <[email protected]>
resolves #1343
resolves #1344
resolves #1347
Docs PR dbt-labs/docs.getdbt.com#6054
Problem
Currently it is not possible to set a quota project in the
profiles.ymlconfiguration for BigQuery. This means that the quota project defaults to the quota project of the credentials. This can cause issues if using an account to auth that does not have the BQ API enabled (see bug here #1347 ). It also means that users can not run their models under a specific quota project if they want to.Solution
Add an optional
quota_projectparam in the profiles.yml that will allow a user to set a quota project if they want to. If they do not set it the behaviour will not change. The only override being applied is aconfig_optionfor the quota project that defaults to None anyway.Checklist