Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add new data health signal page #6621

Merged
merged 30 commits into from
Dec 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
4912a29
new trust signal page
mirnawong1 Dec 9, 2024
610e2ff
Update website/docs/docs/collaborate/trust-signals.md
mirnawong1 Dec 9, 2024
c09b89d
Update website/docs/docs/collaborate/trust-signals.md
mirnawong1 Dec 9, 2024
fe5478f
Update trust-signals.md
mirnawong1 Dec 9, 2024
6343f73
add links
mirnawong1 Dec 9, 2024
95495e6
Merge branch 'update-trust-signals' of github.com:dbt-labs/docs.getdb…
mirnawong1 Dec 9, 2024
c40f5c2
Update website/docs/docs/collaborate/trust-signals.md
mirnawong1 Dec 9, 2024
26a6754
Merge branch 'current' into update-trust-signals
mirnawong1 Dec 9, 2024
ac14f56
Merge branch 'current' into update-trust-signals
mirnawong1 Dec 10, 2024
2554c4f
roxi's feedback
mirnawong1 Dec 10, 2024
70e385d
Merge branch 'current' into update-trust-signals
mirnawong1 Dec 10, 2024
3eebbb0
Merge branch 'current' into update-trust-signals
mirnawong1 Dec 10, 2024
bedd6dd
Update website/docs/docs/dbt-versions/release-notes.md
mirnawong1 Dec 10, 2024
8425efb
Merge branch 'current' into update-trust-signals
mirnawong1 Dec 11, 2024
68e9084
Update website/docs/docs/collaborate/data-health-signals.md
mirnawong1 Dec 11, 2024
b2eca9e
Merge branch 'current' into update-trust-signals
mirnawong1 Dec 11, 2024
cbe4c6a
jarrod's feedback
mirnawong1 Dec 11, 2024
b930af1
fix
mirnawong1 Dec 11, 2024
e350bc9
consistent wording
mirnawong1 Dec 11, 2024
6a1fc0f
Merge branch 'current' into update-trust-signals
mirnawong1 Dec 13, 2024
53233ca
Merge branch 'current' into update-trust-signals
mirnawong1 Dec 16, 2024
bde23b3
Update data-health-signals.md
mirnawong1 Dec 16, 2024
0d1bfde
Merge branch 'current' into update-trust-signals
mirnawong1 Dec 16, 2024
8e25daf
Merge branch 'current' into update-trust-signals
mirnawong1 Dec 16, 2024
da2c934
remove and/or
mirnawong1 Dec 17, 2024
7246ab4
add freshness
mirnawong1 Dec 17, 2024
4e51b03
Merge branch 'current' into update-trust-signals
mirnawong1 Dec 17, 2024
483c64f
Update data-health-signals.md
mirnawong1 Dec 17, 2024
6ce21f9
Update website/docs/docs/collaborate/data-health-signals.md
runleonarun Dec 18, 2024
8a4ceea
Merge branch 'current' into update-trust-signals
runleonarun Dec 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 88 additions & 0 deletions website/docs/docs/collaborate/data-health-signals.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
---
title: "Data health signals"
sidebar_label: "Data health signals"
id: data-health-signals
description: "Learn how data health signals offer a quick, at-a-glance view of data health when browsing your resources in dbt Explorer."
image: /img/docs/collaborate/dbt-explorer/data-health-signal.jpg
---

# Data health signals <Lifecycle status="preview" />
Data health signals offer a quick, at-a-glance view of data health when browsing your resources in dbt Explorer. They keep you informed on the status of your resource's health using the indicators **Healthy**, **Caution**, **Degraded**, or **Unknown**.

- Supported resources are [models](/docs/build/models), [sources](/docs/build/sources), and [exposures](/docs/build/exposures).
- For accurate health data, ensure the resource is up-to-date and had a recent job run.
- Each data health signal reflects key data health components, such as test success status, missing resource descriptions, missing tests, absence of builds in 30-day windows, [and more](#data-health-signal-criteria)

<Lightbox src="/img/docs/collaborate/dbt-explorer/data-health-signal.jpg" width="55%" title="View data health signals for your models."/>

## Access data health signals

Access data health signals in the following places:
Copy link
Member

@jcserv jcserv Dec 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For exposures, trust signals are also present in embedded health tiles which is an iframe that users can put into their BI dashboards to view the health of their exposures

Screenshot 2024-12-11 at 8 57 38 AM

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's a great question and i asked @rpourzand the same in the comments earlier! this is what she said:

Honestly, trust signal is more of a blanket statement for a number of things, including this.. I'd just call this - "Data Health Signals".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jcserv - do you see 'trust signal' and 'trust signal icons' mentioned in the docs? i thought i removed references to it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh i see, you mean we say trust signal in the data health tile doc -- good flag!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

per Mirna's Slack note:

will add source collection and exposure update after meta 3973/3971 and 4025 are complete in a follow up pr

- In the [search function](/docs/collaborate/explore-projects#search-resources) or under **Models**, **Sources**, or **Exposures** in the **Resource** tab.
- For sources, the data health signal also indicates the [source freshness](/docs/deploy/source-freshness) status.
- In the **Health** column on [each resource's details page](/docs/collaborate/explore-projects#view-resource-details). Hover over or click the signal to view detailed information.
- In the **Health** column of public models tables.
- In the [DAG lineage graph](/docs/collaborate/explore-projects#project-lineage). Click any node to open the node details panel where you can view it and its details.
- In [Data health tiles](/docs/collaborate/data-tile) through an embeddable iFrame and visible in your BI dashboard.

<Lightbox src="/img/docs/collaborate/dbt-explorer/data-health-signal.gif" width="95%" title="Access data health signals in multiple places in dbt Explorer."/>

## Data health signal criteria

Each resource has a health state that is determined by specific set of criteria. Select the following tabs to view the criteria for that resource type.
<Tabs>
<TabItem value="models" label="Models">

The health state of a model is determined by the following criteria:
<!-- TODO: remove the 'tbd' lines in the table once meta 4025 is done -->
| **Health state** | **Criteria** |
|-------------------|---------------|
| ✅ **Healthy** | All of the following must be true:<br /><br /> - Built successfully in the last run<br />- Built in the last 30 days<br />- Model has tests configured<br />- All tests passed<br />- All upstream [sources are fresh](/docs/build/sources#source-data-freshness) or freshness is not applicable (set to `null`)<br />- Has a description |
| 🟡 **Caution** | One of the following must be true: <br /><br />- Not built in the last 30 days<br />- Tests are not configured<br />- Tests return warnings<br />- One or more upstream sources are stale:<br />&nbsp;&nbsp;&nbsp;&nbsp;- Has a freshness check configured<br />&nbsp;&nbsp;&nbsp;&nbsp;- Freshness check ran in the past 30 days<br />&nbsp;&nbsp;&nbsp;&nbsp;- Freshness check returned a warning<br />- Missing a description |
| 🔴 **Degraded** | One of the following must be true: <br /><br />- Model failed to build<br />- Model has failing tests<br />- One or more upstream sources are stale:<br />&nbsp;&nbsp;&nbsp;&nbsp;- Freshness check hasn’t run in the past 30 days<br />&nbsp;&nbsp;&nbsp;&nbsp;- Freshness check returned an error |
| ⚪ **Unknown** | - Unable to determine health of resource; no job runs have processed the resource. |

</TabItem>

<TabItem value="sources" label="Sources">

The health state of a source is determined by the following criteria:

| **Health state** | **Criteria** |
|-------------------|---------------|
| ✅ Healthy | All of the following must be true: <br /><br />- Freshness check configured<br />- Freshness check passed<br />- Freshness check ran in the past 30 days<br />- Has a description |
| 🟡 Caution | One of the following must be true: <br /><br />- Freshness check returned a warning<br />- Freshness check not configured<br />- Freshness check not run in the past 30 days<br />- Missing a description |
| 🔴 Degraded | - Freshness check returned an error |
| ⚪ Unknown | Unable to determine health of resource; no job runs have processed the resource. |

</TabItem>

<TabItem value="exposures" label="Exposures">

The health state of an exposure is determined by the following criteria:

| **Health state** | **Criteria** |
|-------------------|---------------|
| ✅ Healthy | All of the following must be true: <br /><br />- Underlying sources are fresh<br />- Underlying models built successfully<br />- Underlying models’ tests passing<br /><!-- - Freshness must be applicable <br /> - (TBD) Underlying models built in the last 30 days --> |
| 🟡 Caution | One of the following must be true: <br /><br />- At least one underlying source’s freshness checks returned a warning<br />- At least one underlying model was skipped<br />- At least one underlying model’s tests returned a warning<br /><!-- - (TBD) At least one model not built in the last 30 days --> |
| 🔴 Degraded | One of the following must be true: <br /><br />- At least one underlying source’s freshness checks returned an error<br />- At least one underlying model did not build successfully<br />- At least one model’s tests returned an error |

</TabItem>

<!-- TODO: Add source collection health once META-3973/3971 are completed
<TabItem value="source-collection" label="Source collection health">

The health state of a source collection is determined by the following criteria:

Functions as an aggregate of underlying sources

| **Health state** | **Criteria** |
|-------------------|---------------|
| ✅ Healthy | - All underlying sources have freshness checks configured OR<br />- All passed their freshness checks OR<br />- All freshness checks ran in the past 30 days OR<br /> - All sources have a description |
| 🟡 Caution | - One or more sources lack freshness checks OR<br />- One or more freshness checks returned a warning OR<br />- One or more freshness checks not run in the past 30 days OR<br />- One or more sources missing a description |
| 🔴 Degraded | - One or more underlying sources’ freshness checks returned error |

</TabItem>
-->

</Tabs>
10 changes: 5 additions & 5 deletions website/docs/docs/collaborate/data-tile.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,24 @@
title: "Data health tile"
id: "data-tile"
sidebar_label: "Data health tile"
description: "Embed data health tiles in your dashboards to distill trust signals for data consumers."
description: "Embed data health tiles in your dashboards to distill data health signals for data consumers."
image: /img/docs/collaborate/dbt-explorer/data-tile-pass.jpg
---

With data health tiles, stakeholders will get an at-a-glance confirmation on whether the data they’re looking at is stale or degraded. This trust signal allows teams to immediately go back into Explorer to see more details and investigate issues.
With data health tiles, stakeholders will get an at-a-glance confirmation on whether the data they’re looking at is stale or degraded. It allows teams to immediately go back into Explorer to see more details and investigate issues.

The data health tile:

- Distills trust signals for data consumers.
- Distills [data health signals](/docs/collaborate/data-health-signals) for data consumers.
- Deep links you into dbt Explorer where you can further dive into upstream data issues.
- Provides richer information and makes it easier to debug.
- Revamps the existing, [job-based tiles](#job-based-data-health).

Data health tiles rely on [exposures](/docs/build/exposures) to surface trust signals in your dashboards. When you configure exposures in your dbt project, you are explicitly defining how specific outputs—like dashboards or reports—depend on your data models.
Data health tiles rely on [exposures](/docs/build/exposures) to surface data health signals in your dashboards. When you configure exposures in your dbt project, you are explicitly defining how specific outputs—like dashboards or reports—depend on your data models.

<DocCarousel slidesPerView={1}>
<Lightbox src="/img/docs/collaborate/dbt-explorer/data-tile-pass.jpg" width="60%" title="Example of passing Data health tile in your dashboard." />
<Lightbox src="/img/docs/collaborate/dbt-explorer/data-tiles.png" width="60%" title="Embed data health tiles in your dashboards to distill trust signals for data consumers." />
<Lightbox src="/img/docs/collaborate/dbt-explorer/data-tiles.png" width="60%" title="Embed data health tiles in your dashboards to distill data health signals for data consumers." />
</DocCarousel>

## Prerequisites
Expand Down
64 changes: 8 additions & 56 deletions website/docs/docs/collaborate/explore-projects.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: "Discover data with dbt Explorer"
sidebar_label: "Discover data with dbt Explorer"
description: "Learn about dbt Explorer and how to interact with it to understand, improve, and leverage your dbt projects."
image: /img/docs/collaborate/dbt-explorer/example-project-lineage-graph.png
pagination_next: "docs/collaborate/access-from-dbt-cloud"
pagination_next: "docs/collaborate/data-health-signals"
pagination_prev: null
---

Expand Down Expand Up @@ -31,6 +31,7 @@ Navigate the dbt Explorer overview page to access your project's resources and m
- **Marts and public models** &mdash; View the [marts](/best-practices/how-we-structure/1-guide-overview#guide-structure-overview) and [public models](/docs/collaborate/govern/model-access#access-modifiers) in your project. You can also navigate to all public models in your account through this view.
- **Model query history** &mdash; Use [model query history](/docs/collaborate/model-query-history) to track consumption queries on your models for deeper insights.
- **Auto-exposures** &mdash; [Set up and view auto-exposures](/docs/collaborate/auto-exposures) to automatically expose relevant data models from Tableau to enhance visibility.
- **Data health signals** &mdash; View the [data-health-signals](/docs/collaborate/data-health-signals) for each resource to understand its health and performance.

<Lightbox src="/img/docs/collaborate/dbt-explorer/explorer-main-page.gif" width="100%" title="Access dbt Explorer from dbt Cloud by clicking Explore in the navigation."/>

Expand Down Expand Up @@ -162,61 +163,8 @@ Under the the **Models** option, you can filter on model properties (access or m

</Expandable>

<Expandable alt_header="Trust signals for resources" lifecycle="preview">

Trust signal icons offer a quick, at-a-glance view of data health when browsing your resources in dbt Explorer. These icons keep you informed on the status of your resource's health using the indicators **Healthy**, **Caution**, **Degraded**, and **Unknown**. For accurate health data, ensure the resource is up-to-date and has had a recent job run. Supported resources are models, sources, and exposures.

Each trust signal icon reflects key data health components, such as test success status, missing resource descriptions, absence of builds in 30-day windows, and more.

To access trust signals:
- Use the search function or click on **Models**, **Sources** or **Exposures** under the **Resource** tab.
- View the icons under the **Health** column.
- Hover over or click the trust signal to see detailed information.
- For sources, the trust signal also indicates the source freshness status.

<Lightbox src="/img/docs/collaborate/dbt-explorer/trust-signal-health.jpg" width="60%" title="View trust signals for your models."/>

<!-- commenting out until further iteration on the content

<Tabs>
<TabItem value="healthy" label="Healthy">
A resource is considered Healthy if it meets the following criteria:
- Ran all tests successfully
- Successfully built in its last run
- Has a description and tests configured
- Has been built in the past 30 days
<Lightbox src="/img/docs/collaborate/dbt-explorer/trust-signal-healthy.png" title="Healthy trust signal icon"/>
</TabItem>

<TabItem value="caution" label="Caution">
Any of the following conditions can trigger a caution status:
- Successfully built in its last run
- Is missing a description
- Does not have any tests configured
- Has not been built in the past 30 day
<Lightbox src="/img/docs/collaborate/dbt-explorer/trust-signal-caution.png" title="Caution trust signal icon"/>
</TabItem>

<TabItem value="degraded" label="Degraded">
A resource is considered Degraded if it meets any of the following criteria:
- Last run failed or source freshness error
<Lightbox src="/img/docs/collaborate/dbt-explorer/trust-signals-degraded.jpg" width="35%" title="Degraded trust signal icon"/>
</TabItem>

<TabItem value="unknown" label="Unknown">
The Unknown trust signal icon indicates that:
- Health information is unknown because the resource hasn't been run recently.
<Lightbox src="/img/docs/collaborate/dbt-explorer/trust-signal-unknown.png" title="Unknown trust signal icon"/>

</TabItem>
</Tabs>

-->

</Expandable>

### Example of keyword search
Example of results from searching on the keyword `customers` and applying the filters models, description, and code. Trust signals are visible to the right of the model name in the search results.
Example of results from searching on the keyword `customers` and applying the filters models, description, and code. [Data health signals](/docs/collaborate/data-health-signals) are visible to the right of the model name in the search results.

<Lightbox src="/img/docs/collaborate/dbt-explorer/example-keyword-search.png" width="100%" title="Example of keyword search" />

Expand All @@ -225,6 +173,7 @@ Example of results from searching on the keyword `customers` and applying the fi
From the sidebar, you can browse your project's resources, its file tree, and the database.

- **Resources** tab &mdash; All resources in the project organized by type. Select any resource type in the list and all those resources in the project will display as a table in the main section of the page. For a description on the different resource types (like models, metrics, and so on), refer to [About dbt projects](/docs/build/projects).
- [Data health signals](/docs/collaborate/data-health-signals) are visible to the right of the resource name under the **Health** column.
- **File Tree** tab &mdash; All resources in the project organized by the file in which they are defined. This mirrors the file tree in your dbt project repository.
- **Database** tab &mdash; All resources in the project organized by the database and schema in which they are built. This mirrors your data platform's structure that represents the [applied state](/docs/dbt-cloud-apis/project-state) of your project.

Expand Down Expand Up @@ -253,7 +202,7 @@ In the upper right corner of the resource details page, you can:

<Expandable alt_header="What details are available for a model?">

- Trust signal icon &mdash; Icons offering a quick, at-a-glance view of data health. These icons indicate whether a model is Healthy, Caution, Degraded, or Unknown. Hover over an icon to view detailed information about the model's health.
- **Data health signals** &mdash; [Data health signals](/docs/collaborate/data-health-signals) offer a quick, at-a-glance view of data health. These icons indicate whether a model is Healthy, Caution, Degraded, or Unknown. Hover over an icon to view detailed information about the model's health.
- **Status bar** (below the page title) &mdash; Information on the last time the model ran, whether the run was successful, how the data is materialized, number of rows, and the size of the model.
- **General** tab includes:
- **Lineage** graph &mdash; The model’s lineage graph that you can interact with. The graph includes one upstream node and one downstream node from the model. Click the Expand icon in the graph's upper right corner to view the model in full lineage graph mode.
Expand All @@ -270,6 +219,7 @@ In the upper right corner of the resource details page, you can:
<Expandable alt_header="What details are available for an exposure?">

- **Status bar** (below the page title) &mdash; Information on the last time the exposure was updated.
- **Data health signals** &mdash; [Data health signals](/docs/collaborate/data-health-signals) offer a quick, at-a-glance view of data health. These icons indicate whether a resource is Healthy, Caution, or Degraded. Hover over an icon to view detailed information about the exposure's health.
- **General** tab includes:
- **Data health** &mdash; The status on data freshness and data quality.
- **Status** section &mdash; The status on data freshness and data quality.
Expand Down Expand Up @@ -303,6 +253,7 @@ Example of the Tests view:
<Expandable alt_header="What details are available for each source table within a source collection?">

- **Status bar** (below the page title) &mdash; Information on the last time the source was updated and the number of tables the source uses.
- **Data health signals** &mdash; [Data health signals](/docs/collaborate/data-health-signals) offer a quick, at-a-glance view of data health. These icons indicate whether a resource is Healthy, Caution, or Degraded. Hover over an icon to view detailed information about the source's health.
- **General** tab includes:
- **Lineage** graph &mdash; The source’s lineage graph that you can interact with. The graph includes one upstream node and one downstream node from the source. Click the Expand icon in the graph's upper right corner to view the source in full lineage graph mode.
- **Description** section &mdash; A description of the source.
Expand All @@ -323,6 +274,7 @@ Example of the details view for the model `customers`:<br /> <Lightbox src="/img

</DocCarousel>


## Staging environment

dbt Explorer supports views for [staging deployment environments](/docs/deploy/deploy-environments#staging-environment), in addition to the production environment. This gives you a unique view into your pre-production data workflows, with the same tools available in production, while providing an extra layer of scrutiny.
Expand Down
Loading
Loading