Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New behavior changes: state_modified_compare_more_unrendered and skip_nodes_if_on_run_start_fails #6204

Merged
merged 22 commits into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
f68f723
on run start behavior change
matthewshaver Oct 1, 2024
66f612f
adding state:modified
matthewshaver Oct 1, 2024
adb5fa2
Adding parameter to table
matthewshaver Oct 1, 2024
316b53b
Merge branch 'current' into hook-change
runleonarun Oct 1, 2024
488e1f5
Merge branch 'current' into hook-change
mirnawong1 Oct 2, 2024
97430a2
Merge branch 'current' into hook-change
runleonarun Oct 2, 2024
0b2f1db
New updates
matthewshaver Oct 2, 2024
257bc4a
Undoing change
matthewshaver Oct 2, 2024
53b4d15
Removing content
matthewshaver Oct 2, 2024
ebe1bb1
Merge branch 'current' into hook-change
matthewshaver Oct 2, 2024
b493b5c
Splitting release note
matthewshaver Oct 2, 2024
1b34dbe
Merge branch 'hook-change' of https://github.com/dbt-labs/docs.getdbt…
matthewshaver Oct 2, 2024
4054384
Moving new content to top
matthewshaver Oct 2, 2024
789e4e5
Update website/docs/docs/dbt-versions/release-notes.md
matthewshaver Oct 2, 2024
44dd028
Removing until `state_modified_compare_vars` is shipped and available
dbeatty10 Oct 3, 2024
3940c63
Update website/docs/reference/global-configs/behavior-changes.md
matthewshaver Oct 3, 2024
5307d86
Merge branch 'current' into hook-change
matthewshaver Oct 3, 2024
b1ad4e0
Keep the `restrict_direct_pg_catalog_access` row as-is
dbeatty10 Oct 3, 2024
2cfa8f2
Update website/docs/reference/global-configs/behavior-changes.md
matthewshaver Oct 3, 2024
027e6d2
Apply suggestions from code review
matthewshaver Oct 3, 2024
b1f29f0
Update website/docs/reference/global-configs/behavior-changes.md
matthewshaver Oct 3, 2024
07c74a6
Update website/docs/reference/global-configs/behavior-changes.md
matthewshaver Oct 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo

## October 2024

- **Behavior change:** New behavior change flags for [`state:modified`](/reference/global-configs/behavior-changes#source-definitions-and-variables-for-state) to reduce false positives/negatives and enhanced behavior for [`on-run-start` hooks](/reference/global-configs/behavior-changes#on-run-start-hook) to skip nodes on failure.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
- **Enhancement**: In dbt Cloud Versionless, snapshots defined in SQL files can now use `config` defined in `schema.yml` YAML files. This update resolves the previous limitation that required snapshot properties to be defined exclusively in `dbt_project.yml` and/or a `config()` block within the SQL file. This enhancement will be included in the upcoming dbt Core v1.9 release.
- **New:** dbt Explorer now includes trust signal icons, which is currently available as a [Preview](/docs/dbt-versions/product-lifecycles#dbt-cloud). Trust signals offer a quick, at-a-glance view of data health when browsing your dbt models in Explorer. These icons indicate whether a model is **Healthy**, **Caution**, **Degraded**, or **Unknown**. For accurate health data, ensure the resource is up-to-date and has had a recent job run. Refer to [Trust signals](/docs/collaborate/explore-projects#trust-signals-for-resources) for more information.
- **New:** Auto exposures are now available in Preview in dbt Cloud. Auto-exposures helps users understand how their models are used in downstream analytics tools to inform investments and reduce incidents. It imports and auto-generates exposures based on Tableau dashboards, with user-defined curation. To learn more, refer to [Auto exposures](/docs/collaborate/auto-exposures).
Expand Down
25 changes: 17 additions & 8 deletions website/docs/reference/global-configs/behavior-changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,13 @@ When we use dbt Cloud in the following table, we're referring to accounts that h

| Flag | dbt Cloud: Intro | dbt Cloud: Maturity | dbt Core: Intro | dbt Core: Maturity |
|-----------------------------------------------------------------|------------------|---------------------|-----------------|--------------------|
| require_explicit_package_overrides_for_builtin_materializations | 2024.04.141 | 2024.06.192 | 1.6.14, 1.7.14 | 1.8.0 |
| require_resource_names_without_spaces | 2024.05.146 | TBD* | 1.8.0 | 1.9.0 |
| source_freshness_run_project_hooks | 2024.03.61 | TBD* | 1.8.0 | 1.9.0 |
| require_explicit_package_overrides_for_builtin_materializations | 2024.04 | 2024.06 | 1.6.14, 1.7.14 | 1.8.0 |
| require_resource_names_without_spaces | 2024.05 | TBD* | 1.8.0 | 1.9.0 |
| source_freshness_run_project_hooks | 2024.03 | TBD* | 1.8.0 | 1.9.0 |
| [Redshift] [restrict_direct_pg_catalog_access](#redshift-restrict_direct_pg_catalog_access) | 2024.09.242 | TBD* | dbt-redshift v1.9.0 | 1.9.0 |
| skip_nodes_if_on_run_start_fails | 2024.10 | TBD* | 1.9.0 | TBD* |
| state_modified_compare_more_unrendered | 2024.10 | TBD* | 1.9.0 | TBD* |
| state_modified_compare_vars | 2024.10 | TBD* | 1.9.0 | TBD* |
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved

When the dbt Cloud Maturity is "TBD," it means we have not yet determined the exact date when these flags' default values will change. Affected users will see deprecation warnings in the meantime, and they will receive emails providing advance warning ahead of the maturity date. In the meantime, if you are seeing a deprecation warning, you can either:
- Migrate your project to support the new behavior, and then set the flag to `True` to stop seeing the warnings.
Expand Down Expand Up @@ -129,10 +132,16 @@ on-run-start:
```
</File>

## Adapter-specific behavior changes
Some adapters may show behavior changes when certain flags are enabled. Refer to the following sections for each respective adapter.
### [Redshift] restrict_direct_pg_catalog_access
### On-run-start hook

Originally, the `dbt-redshift` adapter was built on top of the `dbt-postgres` adapter and used Postgres tables for metadata access. With this flag enabled, the adapter will use the Redshift API (through the Python client) if available, or query Redshift's `information_schema` tables instead of using `pg_` tables.
The flag is `False` by default.

While we don't expect any user-noticeable behavior changes due to this change, out of caution we are gating it behind a behavior-change flag and encouraging users to test it before it becomes the default for everyone.
Set the `skip_nodes_if_on_run_start_fails` flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook.

### Source definitions and variables for state:modified

The flags are `False` by default.

Set `state_modified_compare_more_unrendered` to `True` to start persisting `unrendered_database` and `unrendered_schema` configs during source parsing, and do comparison on unrendered values during `state:modified` checks. Setting the flag to `True` reduces false positives during `state:modified` checks when `prod` and `dev` environments have different configs.

Set the `state_modified_compare_vars` to `True` if a model uses a `var` or `env_var` in its definition. dbt will be able to identify its lineage to include the model in `state:modified` because the `var` or `env_var` value has changed.
Loading