From f68f723700ad43d318c68c9846bf5d09fe83e126 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 1 Oct 2024 17:47:39 -0400 Subject: [PATCH 01/16] on run start behavior change --- website/docs/reference/global-configs/behavior-changes.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index 8e9e93b8488..040998873f5 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -68,6 +68,7 @@ When we use dbt Cloud in the following table, we're referring to accounts that h | require_resource_names_without_spaces | 2024.05.146 | TBD* | 1.8.0 | 1.9.0 | | source_freshness_run_project_hooks | 2024.03.61 | TBD* | 1.8.0 | 1.9.0 | | [Redshift] [restrict_direct_pg_catalog_access](#redshift-restrict_direct_pg_catalog_access) | 2024.09.242 | TBD* | dbt-redshift v1.9.0 | 1.9.0 | +| skip_nodes_if_on_run_start_fails | | TBD* | 1.9.0 | TBD* | When the dbt Cloud Maturity is "TBD," it means we have not yet determined the exact date when these flags' default values will change. Affected users will see deprecation warnings in the meantime, and they will receive emails providing advance warning ahead of the maturity date. In the meantime, if you are seeing a deprecation warning, you can either: - Migrate your project to support the new behavior, and then set the flag to `True` to stop seeing the warnings. @@ -129,6 +130,12 @@ on-run-start: ``` +### On-run-start hook + +Set the `skip_nodes_if_on_run_start_fails` flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook. + +The flag is `False` by default. + ## Adapter-specific behavior changes Some adapters may show behavior changes when certain flags are enabled. Refer to the following sections for each respective adapter. ### [Redshift] restrict_direct_pg_catalog_access From 66f612f83587a74735088807d49c6626dd44af0a Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 1 Oct 2024 18:38:30 -0400 Subject: [PATCH 02/16] adding state:modified --- .../docs/reference/global-configs/behavior-changes.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index 040998873f5..e1852ef73f5 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -69,6 +69,7 @@ When we use dbt Cloud in the following table, we're referring to accounts that h | source_freshness_run_project_hooks | 2024.03.61 | TBD* | 1.8.0 | 1.9.0 | | [Redshift] [restrict_direct_pg_catalog_access](#redshift-restrict_direct_pg_catalog_access) | 2024.09.242 | TBD* | dbt-redshift v1.9.0 | 1.9.0 | | skip_nodes_if_on_run_start_fails | | TBD* | 1.9.0 | TBD* | +| state_modified_compare_more_unrendered | | TBD* | 1.9.0 | TBD* | When the dbt Cloud Maturity is "TBD," it means we have not yet determined the exact date when these flags' default values will change. Affected users will see deprecation warnings in the meantime, and they will receive emails providing advance warning ahead of the maturity date. In the meantime, if you are seeing a deprecation warning, you can either: - Migrate your project to support the new behavior, and then set the flag to `True` to stop seeing the warnings. @@ -132,9 +133,17 @@ on-run-start: ### On-run-start hook +The flag is `False` by default. + Set the `skip_nodes_if_on_run_start_fails` flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook. -The flag is `False` by default. +### Source definitions and variables for state:modified + +The flags are `False` by default. + +Set `state_modified_compare_more_unrendered` to `True` to start persisting `unrendered_database` and `unrendered_schema` configs during source parsing, and do comparison on unrendered values during `state:modified` checks. Setting the flag to `True` reduces false positives during `state:modified` checks when `prod` and `dev` environments have different configs. + +Set the `state_modified_compare_vars` to `True` if a model uses a `var` or `env_var` in its definition. dbt will be able to identify its lineage to include the model in `state:modified` because the `var` or `env_var` value has changed. ## Adapter-specific behavior changes Some adapters may show behavior changes when certain flags are enabled. Refer to the following sections for each respective adapter. From adb5fa2b476a90bf2e727d2ed9130b7147d15a5d Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 1 Oct 2024 19:50:08 -0400 Subject: [PATCH 03/16] Adding parameter to table --- website/docs/reference/global-configs/behavior-changes.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index e1852ef73f5..36420f0a1f4 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -70,6 +70,7 @@ When we use dbt Cloud in the following table, we're referring to accounts that h | [Redshift] [restrict_direct_pg_catalog_access](#redshift-restrict_direct_pg_catalog_access) | 2024.09.242 | TBD* | dbt-redshift v1.9.0 | 1.9.0 | | skip_nodes_if_on_run_start_fails | | TBD* | 1.9.0 | TBD* | | state_modified_compare_more_unrendered | | TBD* | 1.9.0 | TBD* | +| state_modified_compare_vars | | TBD* | 1.9.0 | TBD* | When the dbt Cloud Maturity is "TBD," it means we have not yet determined the exact date when these flags' default values will change. Affected users will see deprecation warnings in the meantime, and they will receive emails providing advance warning ahead of the maturity date. In the meantime, if you are seeing a deprecation warning, you can either: - Migrate your project to support the new behavior, and then set the flag to `True` to stop seeing the warnings. From 0b2f1dbf3644a86b14f2ddfe7f95148710676aa8 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 2 Oct 2024 19:11:04 -0400 Subject: [PATCH 04/16] New updates --- .../cloud/git/import-a-project-by-git-url.md | 16 +++++++++------ .../docs/docs/dbt-versions/release-notes.md | 1 + .../global-configs/behavior-changes.md | 20 ++++++------------- 3 files changed, 17 insertions(+), 20 deletions(-) diff --git a/website/docs/docs/cloud/git/import-a-project-by-git-url.md b/website/docs/docs/cloud/git/import-a-project-by-git-url.md index 90c54dbb1b1..3b5e7279a77 100644 --- a/website/docs/docs/cloud/git/import-a-project-by-git-url.md +++ b/website/docs/docs/cloud/git/import-a-project-by-git-url.md @@ -5,9 +5,19 @@ pagination_next: "docs/cloud/git/connect-github" pagination_prev: null --- +:::note limited integration + +Some features of dbt Cloud require a tight integration with your git host, for example, updating GitHub pull requests with dbt Cloud run statuses. Importing your project by a URL prevents you from using these features. Once you give dbt Cloud access to your repository, you can continue to set up your project by adding a connection and creating and running your first dbt Cloud job. + +Don't see your git provider on this page? Please [contact dbt Support](mailto:support@getdbt.com) - we're happy to help you set up dbt Cloud with any supported git provider. + +::: + In dbt Cloud, you can import a git repository from any valid git URL that points to a dbt project. There are some important considerations to keep in mind when doing this. + ## Git protocols + You must use the `git@...` or `ssh:..`. version of your git URL, not the `https://...` version. dbt Cloud uses the SSH protocol to clone repositories, so dbt Cloud will be unable to clone repos supplied with the HTTP protocol. ## Managing Deploy Keys @@ -140,12 +150,6 @@ If you use Azure DevOps and you are on the dbt Cloud Enterprise plan, you can im -## Other git providers - -Don't see your git provider here? Please [contact dbt Support](mailto:support@getdbt.com) - we're happy to help you set up dbt Cloud with any supported git provider. - -## Limited integration -Some features of dbt Cloud require a tight integration with your git host, for example, updating GitHub pull requests with dbt Cloud run statuses. Importing your project by a URL prevents you from using these features. Once you give dbt Cloud access to your repository, you can continue to set up your project by adding a connection and creating and running your first dbt Cloud job. ## FAQs diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index f98cd199ad7..d8c5894b9ce 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -20,6 +20,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo ## October 2024 +- **Behavior change:** New behavior change flags for [`state:modified`](/reference/global-configs/behavior-changes#source-definitions-and-variables-for-state) to reduce false positives/negatives and enhanced behavior for [`on-run-start` hooks](/reference/global-configs/behavior-changes#on-run-start-hook) to skip nodes on failure. - **Enhancement**: In dbt Cloud Versionless, snapshots defined in SQL files can now use `config` defined in `schema.yml` YAML files. This update resolves the previous limitation that required snapshot properties to be defined exclusively in `dbt_project.yml` and/or a `config()` block within the SQL file. This enhancement will be included in the upcoming dbt Core v1.9 release. - **New:** dbt Explorer now includes trust signal icons, which is currently available as a [Preview](/docs/dbt-versions/product-lifecycles#dbt-cloud). Trust signals offer a quick, at-a-glance view of data health when browsing your dbt models in Explorer. These icons indicate whether a model is **Healthy**, **Caution**, **Degraded**, or **Unknown**. For accurate health data, ensure the resource is up-to-date and has had a recent job run. Refer to [Trust signals](/docs/collaborate/explore-projects#trust-signals-for-resources) for more information. - **New:** Auto exposures are now available in Preview in dbt Cloud. Auto-exposures helps users understand how their models are used in downstream analytics tools to inform investments and reduce incidents. It imports and auto-generates exposures based on Tableau dashboards, with user-defined curation. To learn more, refer to [Auto exposures](/docs/collaborate/auto-exposures). diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index 36420f0a1f4..346dc663535 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -64,13 +64,13 @@ When we use dbt Cloud in the following table, we're referring to accounts that h | Flag | dbt Cloud: Intro | dbt Cloud: Maturity | dbt Core: Intro | dbt Core: Maturity | |-----------------------------------------------------------------|------------------|---------------------|-----------------|--------------------| -| require_explicit_package_overrides_for_builtin_materializations | 2024.04.141 | 2024.06.192 | 1.6.14, 1.7.14 | 1.8.0 | -| require_resource_names_without_spaces | 2024.05.146 | TBD* | 1.8.0 | 1.9.0 | -| source_freshness_run_project_hooks | 2024.03.61 | TBD* | 1.8.0 | 1.9.0 | +| require_explicit_package_overrides_for_builtin_materializations | 2024.04 | 2024.06 | 1.6.14, 1.7.14 | 1.8.0 | +| require_resource_names_without_spaces | 2024.05 | TBD* | 1.8.0 | 1.9.0 | +| source_freshness_run_project_hooks | 2024.03 | TBD* | 1.8.0 | 1.9.0 | | [Redshift] [restrict_direct_pg_catalog_access](#redshift-restrict_direct_pg_catalog_access) | 2024.09.242 | TBD* | dbt-redshift v1.9.0 | 1.9.0 | -| skip_nodes_if_on_run_start_fails | | TBD* | 1.9.0 | TBD* | -| state_modified_compare_more_unrendered | | TBD* | 1.9.0 | TBD* | -| state_modified_compare_vars | | TBD* | 1.9.0 | TBD* | +| skip_nodes_if_on_run_start_fails | 2024.10 | TBD* | 1.9.0 | TBD* | +| state_modified_compare_more_unrendered | 2024.10 | TBD* | 1.9.0 | TBD* | +| state_modified_compare_vars | 2024.10 | TBD* | 1.9.0 | TBD* | When the dbt Cloud Maturity is "TBD," it means we have not yet determined the exact date when these flags' default values will change. Affected users will see deprecation warnings in the meantime, and they will receive emails providing advance warning ahead of the maturity date. In the meantime, if you are seeing a deprecation warning, you can either: - Migrate your project to support the new behavior, and then set the flag to `True` to stop seeing the warnings. @@ -145,11 +145,3 @@ The flags are `False` by default. Set `state_modified_compare_more_unrendered` to `True` to start persisting `unrendered_database` and `unrendered_schema` configs during source parsing, and do comparison on unrendered values during `state:modified` checks. Setting the flag to `True` reduces false positives during `state:modified` checks when `prod` and `dev` environments have different configs. Set the `state_modified_compare_vars` to `True` if a model uses a `var` or `env_var` in its definition. dbt will be able to identify its lineage to include the model in `state:modified` because the `var` or `env_var` value has changed. - -## Adapter-specific behavior changes -Some adapters may show behavior changes when certain flags are enabled. Refer to the following sections for each respective adapter. -### [Redshift] restrict_direct_pg_catalog_access - -Originally, the `dbt-redshift` adapter was built on top of the `dbt-postgres` adapter and used Postgres tables for metadata access. With this flag enabled, the adapter will use the Redshift API (through the Python client) if available, or query Redshift's `information_schema` tables instead of using `pg_` tables. - -While we don't expect any user-noticeable behavior changes due to this change, out of caution we are gating it behind a behavior-change flag and encouraging users to test it before it becomes the default for everyone. From 257bc4a576031f4d9bd1944ac798132cb7a11f7a Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 2 Oct 2024 19:15:12 -0400 Subject: [PATCH 05/16] Undoing change --- .../cloud/git/import-a-project-by-git-url.md | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/website/docs/docs/cloud/git/import-a-project-by-git-url.md b/website/docs/docs/cloud/git/import-a-project-by-git-url.md index 3b5e7279a77..90c54dbb1b1 100644 --- a/website/docs/docs/cloud/git/import-a-project-by-git-url.md +++ b/website/docs/docs/cloud/git/import-a-project-by-git-url.md @@ -5,19 +5,9 @@ pagination_next: "docs/cloud/git/connect-github" pagination_prev: null --- -:::note limited integration - -Some features of dbt Cloud require a tight integration with your git host, for example, updating GitHub pull requests with dbt Cloud run statuses. Importing your project by a URL prevents you from using these features. Once you give dbt Cloud access to your repository, you can continue to set up your project by adding a connection and creating and running your first dbt Cloud job. - -Don't see your git provider on this page? Please [contact dbt Support](mailto:support@getdbt.com) - we're happy to help you set up dbt Cloud with any supported git provider. - -::: - In dbt Cloud, you can import a git repository from any valid git URL that points to a dbt project. There are some important considerations to keep in mind when doing this. - ## Git protocols - You must use the `git@...` or `ssh:..`. version of your git URL, not the `https://...` version. dbt Cloud uses the SSH protocol to clone repositories, so dbt Cloud will be unable to clone repos supplied with the HTTP protocol. ## Managing Deploy Keys @@ -150,6 +140,12 @@ If you use Azure DevOps and you are on the dbt Cloud Enterprise plan, you can im +## Other git providers + +Don't see your git provider here? Please [contact dbt Support](mailto:support@getdbt.com) - we're happy to help you set up dbt Cloud with any supported git provider. + +## Limited integration +Some features of dbt Cloud require a tight integration with your git host, for example, updating GitHub pull requests with dbt Cloud run statuses. Importing your project by a URL prevents you from using these features. Once you give dbt Cloud access to your repository, you can continue to set up your project by adding a connection and creating and running your first dbt Cloud job. ## FAQs From 53b4d1556f46c5fe349368a8a6c68add6d63b1d8 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 2 Oct 2024 19:24:00 -0400 Subject: [PATCH 06/16] Removing content --- website/docs/docs/dbt-versions/release-notes.md | 2 +- website/docs/reference/global-configs/behavior-changes.md | 6 ++---- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index d8c5894b9ce..db0ac2807a4 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -20,7 +20,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo ## October 2024 -- **Behavior change:** New behavior change flags for [`state:modified`](/reference/global-configs/behavior-changes#source-definitions-and-variables-for-state) to reduce false positives/negatives and enhanced behavior for [`on-run-start` hooks](/reference/global-configs/behavior-changes#on-run-start-hook) to skip nodes on failure. +- **Behavior change:** New behavior change flag for [`state:modified`](/reference/global-configs/behavior-changes#source-definitions-for-state) to reduce false positives/negatives and enhanced behavior for [`on-run-start` hooks](/reference/global-configs/behavior-changes#on-run-start-hook) to skip nodes on failure. - **Enhancement**: In dbt Cloud Versionless, snapshots defined in SQL files can now use `config` defined in `schema.yml` YAML files. This update resolves the previous limitation that required snapshot properties to be defined exclusively in `dbt_project.yml` and/or a `config()` block within the SQL file. This enhancement will be included in the upcoming dbt Core v1.9 release. - **New:** dbt Explorer now includes trust signal icons, which is currently available as a [Preview](/docs/dbt-versions/product-lifecycles#dbt-cloud). Trust signals offer a quick, at-a-glance view of data health when browsing your dbt models in Explorer. These icons indicate whether a model is **Healthy**, **Caution**, **Degraded**, or **Unknown**. For accurate health data, ensure the resource is up-to-date and has had a recent job run. Refer to [Trust signals](/docs/collaborate/explore-projects#trust-signals-for-resources) for more information. - **New:** Auto exposures are now available in Preview in dbt Cloud. Auto-exposures helps users understand how their models are used in downstream analytics tools to inform investments and reduce incidents. It imports and auto-generates exposures based on Tableau dashboards, with user-defined curation. To learn more, refer to [Auto exposures](/docs/collaborate/auto-exposures). diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index 346dc663535..65e6541fcdc 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -138,10 +138,8 @@ The flag is `False` by default. Set the `skip_nodes_if_on_run_start_fails` flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook. -### Source definitions and variables for state:modified +### Source definitions for state:modified -The flags are `False` by default. +The flags is `False` by default. Set `state_modified_compare_more_unrendered` to `True` to start persisting `unrendered_database` and `unrendered_schema` configs during source parsing, and do comparison on unrendered values during `state:modified` checks. Setting the flag to `True` reduces false positives during `state:modified` checks when `prod` and `dev` environments have different configs. - -Set the `state_modified_compare_vars` to `True` if a model uses a `var` or `env_var` in its definition. dbt will be able to identify its lineage to include the model in `state:modified` because the `var` or `env_var` value has changed. From b493b5c895ec6bdbc4bdfa3862b7148b1501108b Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 2 Oct 2024 19:43:06 -0400 Subject: [PATCH 07/16] Splitting release note --- website/docs/docs/dbt-versions/release-notes.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index db0ac2807a4..f52d74f1ac5 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -20,7 +20,8 @@ Release notes are grouped by month for both multi-tenant and virtual private clo ## October 2024 -- **Behavior change:** New behavior change flag for [`state:modified`](/reference/global-configs/behavior-changes#source-definitions-for-state) to reduce false positives/negatives and enhanced behavior for [`on-run-start` hooks](/reference/global-configs/behavior-changes#on-run-start-hook) to skip nodes on failure. +- **Behavior change:** Set [`state_modified_compare_more_unrendered`](reference/global-configs/behavior-changes#source-definitions-for-state) to true to reduce false positives for `state:modified` when configs differ between `dev` and `prod` environments. +- **Behavior change:** Set the [`skip_nodes_if_on_run_start_fails`](/reference/global-configs/behavior-changes#on-run-start-hook) flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook. - **Enhancement**: In dbt Cloud Versionless, snapshots defined in SQL files can now use `config` defined in `schema.yml` YAML files. This update resolves the previous limitation that required snapshot properties to be defined exclusively in `dbt_project.yml` and/or a `config()` block within the SQL file. This enhancement will be included in the upcoming dbt Core v1.9 release. - **New:** dbt Explorer now includes trust signal icons, which is currently available as a [Preview](/docs/dbt-versions/product-lifecycles#dbt-cloud). Trust signals offer a quick, at-a-glance view of data health when browsing your dbt models in Explorer. These icons indicate whether a model is **Healthy**, **Caution**, **Degraded**, or **Unknown**. For accurate health data, ensure the resource is up-to-date and has had a recent job run. Refer to [Trust signals](/docs/collaborate/explore-projects#trust-signals-for-resources) for more information. - **New:** Auto exposures are now available in Preview in dbt Cloud. Auto-exposures helps users understand how their models are used in downstream analytics tools to inform investments and reduce incidents. It imports and auto-generates exposures based on Tableau dashboards, with user-defined curation. To learn more, refer to [Auto exposures](/docs/collaborate/auto-exposures). From 40543841e7b55ace7786e6a035801bc30371fa7a Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 2 Oct 2024 19:44:47 -0400 Subject: [PATCH 08/16] Moving new content to top --- .../global-configs/behavior-changes.md | 25 ++++++++++--------- 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index 65e6541fcdc..2c20a41bdd8 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -76,6 +76,19 @@ When the dbt Cloud Maturity is "TBD," it means we have not yet determined the ex - Migrate your project to support the new behavior, and then set the flag to `True` to stop seeing the warnings. - Set the flag to `False`. You will continue to see warnings, and you will retain the legacy behavior even after the maturity date (when the default value changes). +### On-run-start hook + +The flag is `False` by default. + +Set the `skip_nodes_if_on_run_start_fails` flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook. + +### Source definitions for state:modified + +The flags is `False` by default. + +Set `state_modified_compare_more_unrendered` to `True` to start persisting `unrendered_database` and `unrendered_schema` configs during source parsing, and do comparison on unrendered values during `state:modified` checks. Setting the flag to `True` reduces false positives during `state:modified` checks when `prod` and `dev` environments have different configs. + + ### Package override for built-in materialization Setting the `require_explicit_package_overrides_for_builtin_materializations` flag to `True` prevents this automatic override. @@ -131,15 +144,3 @@ on-run-start: - '{{ ... if flags.WHICH != 'freshness' }}' ``` - -### On-run-start hook - -The flag is `False` by default. - -Set the `skip_nodes_if_on_run_start_fails` flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook. - -### Source definitions for state:modified - -The flags is `False` by default. - -Set `state_modified_compare_more_unrendered` to `True` to start persisting `unrendered_database` and `unrendered_schema` configs during source parsing, and do comparison on unrendered values during `state:modified` checks. Setting the flag to `True` reduces false positives during `state:modified` checks when `prod` and `dev` environments have different configs. From 789e4e5f56cc6e4a9b8c2a3e832b3b5bb9b59cca Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 2 Oct 2024 19:51:18 -0400 Subject: [PATCH 09/16] Update website/docs/docs/dbt-versions/release-notes.md --- website/docs/docs/dbt-versions/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index b1c2badc8af..c2a24c1373b 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -20,7 +20,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo ## October 2024 -- **Behavior change:** Set [`state_modified_compare_more_unrendered`](reference/global-configs/behavior-changes#source-definitions-for-state) to true to reduce false positives for `state:modified` when configs differ between `dev` and `prod` environments. +- **Behavior change:** Set [`state_modified_compare_more_unrendered`](/reference/global-configs/behavior-changes#source-definitions-for-state) to true to reduce false positives for `state:modified` when configs differ between `dev` and `prod` environments. - **Behavior change:** Set the [`skip_nodes_if_on_run_start_fails`](/reference/global-configs/behavior-changes#on-run-start-hook) flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook. - **Enhancement**: In dbt Cloud Versionless, snapshots defined in SQL files can now use `config` defined in `schema.yml` YAML files. This update resolves the previous limitation that required snapshot properties to be defined exclusively in `dbt_project.yml` and/or a `config()` block within the SQL file. This enhancement will be included in the upcoming dbt Core v1.9 release. - **Enhancement**: In May 2024, dbt Cloud versionless began inferring a model's `primary_key` based on configured data tests and/or constraints within `manifest.json`. The inferred `primary_key` is visible in dbt Explorer and utilized by the dbt Cloud [compare changes](/docs/deploy/run-visibility#compare-tab) feature. This will also be released in dbt Core 1.9. From 44dd0280cf7592c7caa885e599389cbf1d89eac6 Mon Sep 17 00:00:00 2001 From: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Date: Thu, 3 Oct 2024 09:22:59 -0600 Subject: [PATCH 10/16] Removing until `state_modified_compare_vars` is shipped and available Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- website/docs/reference/global-configs/behavior-changes.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index 2c20a41bdd8..2b4651ecbe2 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -70,7 +70,6 @@ When we use dbt Cloud in the following table, we're referring to accounts that h | [Redshift] [restrict_direct_pg_catalog_access](#redshift-restrict_direct_pg_catalog_access) | 2024.09.242 | TBD* | dbt-redshift v1.9.0 | 1.9.0 | | skip_nodes_if_on_run_start_fails | 2024.10 | TBD* | 1.9.0 | TBD* | | state_modified_compare_more_unrendered | 2024.10 | TBD* | 1.9.0 | TBD* | -| state_modified_compare_vars | 2024.10 | TBD* | 1.9.0 | TBD* | When the dbt Cloud Maturity is "TBD," it means we have not yet determined the exact date when these flags' default values will change. Affected users will see deprecation warnings in the meantime, and they will receive emails providing advance warning ahead of the maturity date. In the meantime, if you are seeing a deprecation warning, you can either: - Migrate your project to support the new behavior, and then set the flag to `True` to stop seeing the warnings. From 3940c63577162e261a1ddbabbf9980f309066edb Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 3 Oct 2024 11:56:31 -0400 Subject: [PATCH 11/16] Update website/docs/reference/global-configs/behavior-changes.md Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- website/docs/reference/global-configs/behavior-changes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index 2b4651ecbe2..c747200c96c 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -83,7 +83,7 @@ Set the `skip_nodes_if_on_run_start_fails` flag to `True` to skip all selected r ### Source definitions for state:modified -The flags is `False` by default. +The flag is `False` by default. Set `state_modified_compare_more_unrendered` to `True` to start persisting `unrendered_database` and `unrendered_schema` configs during source parsing, and do comparison on unrendered values during `state:modified` checks. Setting the flag to `True` reduces false positives during `state:modified` checks when `prod` and `dev` environments have different configs. From b1ad4e0e2f34d3bce19c23560b6f06d77011ddd6 Mon Sep 17 00:00:00 2001 From: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Date: Thu, 3 Oct 2024 10:01:06 -0600 Subject: [PATCH 12/16] Keep the `restrict_direct_pg_catalog_access` row as-is --- website/docs/reference/global-configs/behavior-changes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index c747200c96c..84b8f69cf88 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -67,7 +67,7 @@ When we use dbt Cloud in the following table, we're referring to accounts that h | require_explicit_package_overrides_for_builtin_materializations | 2024.04 | 2024.06 | 1.6.14, 1.7.14 | 1.8.0 | | require_resource_names_without_spaces | 2024.05 | TBD* | 1.8.0 | 1.9.0 | | source_freshness_run_project_hooks | 2024.03 | TBD* | 1.8.0 | 1.9.0 | -| [Redshift] [restrict_direct_pg_catalog_access](#redshift-restrict_direct_pg_catalog_access) | 2024.09.242 | TBD* | dbt-redshift v1.9.0 | 1.9.0 | +| [Redshift] [restrict_direct_pg_catalog_access](#redshift-restrict_direct_pg_catalog_access) | 2024.09 | TBD* | dbt-redshift v1.9.0 | 1.9.0 | | skip_nodes_if_on_run_start_fails | 2024.10 | TBD* | 1.9.0 | TBD* | | state_modified_compare_more_unrendered | 2024.10 | TBD* | 1.9.0 | TBD* | From 2cfa8f2ce0fb30930d2627d2d6ed71e61e826828 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 3 Oct 2024 12:57:01 -0400 Subject: [PATCH 13/16] Update website/docs/reference/global-configs/behavior-changes.md Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- website/docs/reference/global-configs/behavior-changes.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index 84b8f69cf88..0d6719ac1ee 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -85,7 +85,9 @@ Set the `skip_nodes_if_on_run_start_fails` flag to `True` to skip all selected r The flag is `False` by default. -Set `state_modified_compare_more_unrendered` to `True` to start persisting `unrendered_database` and `unrendered_schema` configs during source parsing, and do comparison on unrendered values during `state:modified` checks. Setting the flag to `True` reduces false positives during `state:modified` checks when `prod` and `dev` environments have different configs. +Set `state_modified_compare_more_unrendered` to `True` to reduce false positives during `state:modified` checks (especially when configs differ by target environment like `prod` vs. `dev`). + +Setting the flag to `True` changes the `state:modified` comparison from using rendered values to unrendered values instead. It accomplishes this by persisting `unrendered_config` during parsing of models and `unrendered_database` and `unrendered_schema` configs during parsing of sources. ### Package override for built-in materialization From 027e6d270a7596e20b29541352da27217f9c1d4a Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 3 Oct 2024 12:57:10 -0400 Subject: [PATCH 14/16] Apply suggestions from code review Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- website/docs/docs/dbt-versions/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index c2a24c1373b..0f8867edc5a 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -21,7 +21,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo ## October 2024 - **Behavior change:** Set [`state_modified_compare_more_unrendered`](/reference/global-configs/behavior-changes#source-definitions-for-state) to true to reduce false positives for `state:modified` when configs differ between `dev` and `prod` environments. -- **Behavior change:** Set the [`skip_nodes_if_on_run_start_fails`](/reference/global-configs/behavior-changes#on-run-start-hook) flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook. +- **Behavior change:** Set the [`skip_nodes_if_on_run_start_fails`](/reference/global-configs/behavior-changes#failures-in-on-run-start-hooks) flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook. - **Enhancement**: In dbt Cloud Versionless, snapshots defined in SQL files can now use `config` defined in `schema.yml` YAML files. This update resolves the previous limitation that required snapshot properties to be defined exclusively in `dbt_project.yml` and/or a `config()` block within the SQL file. This enhancement will be included in the upcoming dbt Core v1.9 release. - **Enhancement**: In May 2024, dbt Cloud versionless began inferring a model's `primary_key` based on configured data tests and/or constraints within `manifest.json`. The inferred `primary_key` is visible in dbt Explorer and utilized by the dbt Cloud [compare changes](/docs/deploy/run-visibility#compare-tab) feature. This will also be released in dbt Core 1.9. Read about the [order dbt infers columns can be used as primary key of a model](https://github.com/dbt-labs/dbt-core/blob/7940ad5c7858ff11ef100260a372f2f06a86e71f/core/dbt/contracts/graph/nodes.py#L534-L541). From b1f29f0413df8c8a7a4449756f4e159929b3719e Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 3 Oct 2024 12:57:29 -0400 Subject: [PATCH 15/16] Update website/docs/reference/global-configs/behavior-changes.md Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- website/docs/reference/global-configs/behavior-changes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index 0d6719ac1ee..2e258449a06 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -75,7 +75,7 @@ When the dbt Cloud Maturity is "TBD," it means we have not yet determined the ex - Migrate your project to support the new behavior, and then set the flag to `True` to stop seeing the warnings. - Set the flag to `False`. You will continue to see warnings, and you will retain the legacy behavior even after the maturity date (when the default value changes). -### On-run-start hook +### Failures in on-run-start hooks The flag is `False` by default. From 07c74a6fdcf51847c18260c0f077b9956a56597e Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 3 Oct 2024 12:58:03 -0400 Subject: [PATCH 16/16] Update website/docs/reference/global-configs/behavior-changes.md --- website/docs/reference/global-configs/behavior-changes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index 2e258449a06..01b71aa6976 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -87,7 +87,7 @@ The flag is `False` by default. Set `state_modified_compare_more_unrendered` to `True` to reduce false positives during `state:modified` checks (especially when configs differ by target environment like `prod` vs. `dev`). -Setting the flag to `True` changes the `state:modified` comparison from using rendered values to unrendered values instead. It accomplishes this by persisting `unrendered_config` during parsing of models and `unrendered_database` and `unrendered_schema` configs during parsing of sources. +Setting the flag to `True` changes the `state:modified` comparison from using rendered values to unrendered values instead. It accomplishes this by persisting `unrendered_config` during model parsing and `unrendered_database` and `unrendered_schema` configs during source parsing. ### Package override for built-in materialization