Skip to content

Commit

Permalink
Merge branch 'current' into add-snapshot-yaml
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Dec 19, 2024
2 parents 265546b + 9953c37 commit 6c431d7
Show file tree
Hide file tree
Showing 26 changed files with 347 additions and 108 deletions.
1 change: 1 addition & 0 deletions website/dbt-versions.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ exports.versions = [
},
{
version: "1.9",
customDisplay: "1.9 (Compatible)",
EOLDate: "2025-12-08",
},
{
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/build/metrics-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -271,9 +271,9 @@ metrics:
measure:
name: cancellations_usd # Specify the measure you are creating a proxy for.
fill_nulls_with: 0
join_to_timespine: true
filter: |
{{ Dimension('order__value')}} > 100 and {{Dimension('user__acquisition')}} is not null
join_to_timespine: true
```
</File>

Expand Down
63 changes: 59 additions & 4 deletions website/docs/docs/build/packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,9 +157,57 @@ packages:

Where `name: 'dbt_utils'` specifies the subfolder of `dbt_packages` that's created for the package source code to be installed within.

### Private packages
## Private packages

#### SSH Key Method (Command Line only)
### Native private packages <Lifecycle status='beta'/>

dbt Cloud supports private packages from [supported](#prerequisites) Git repos leveraging an exisiting [configuration](/docs/cloud/git/git-configuration-in-dbt-cloud) in your environment. Previously, you had to configure a [token](#git-token-method) to retrieve packages from your private repos.

#### Prerequisites

To use native private packages, you must have one of the following Git providers configured in the **Integrations** section of your **Account settings**:
- [GitHub](/docs/cloud/git/connect-github)
- [Azure DevOps](/docs/cloud/git/connect-azure-devops)
- Support for GitLab is coming soon.


#### Configuration

Use the `private` key in your `packages.yml` or `dependencies.yml` to clone package repos using your existing dbt Cloud Git integration without having to provision an access token or create a dbt Cloud environment variable:

<File name="packages.yml">

```yaml
packages:
- private: dbt-labs/awesome_repo
- package: normal packages
[...]
```

</File>

You can pin private packages similar to regular dbt packages:

```yaml
packages:
- private: dbt-labs/awesome_repo
revision: "0.9.5" # Pin to a tag, branch, or complete 40-character commit hash
```

If you are using multiple Git integrations, disambiguate by adding the provider key:

```yaml
packages:
- private: dbt-labs/awesome_repo
provider: "github" # GitHub and Azure are currently supported. GitLab is coming soon.
```

With this method, you can retrieve private packages from an integrated Git provider without any additional steps to connect.

### SSH key method (command line only)
If you're using the Command Line, private packages can be cloned via SSH and an SSH key.

When you use SSH keys to authenticate to your git remote server, you don’t need to supply your username and password each time. Read more about SSH keys, how to generate them, and how to add them to your git provider here: [Github](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh) and [GitLab](https://docs.gitlab.com/ee/user/ssh.html).
Expand All @@ -177,7 +225,14 @@ packages:
If you're using dbt Cloud, the SSH key method will not work, but you can use the [HTTPS Git Token Method](https://docs.getdbt.com/docs/build/packages#git-token-method).


#### Git token method
### Git token method

:::note

dbt Cloud has [native support](#native-private-packages) for Git hosted private packages with GitHub and Azure DevOps (GitLab coming soon). If you are using a supported [integrated Git environment](/docs/cloud/git/git-configuration-in-dbt-cloud), you no longer need to configure Git tokens to retrieve private packages.

:::

This method allows the user to clone via HTTPS by passing in a git token via an environment variable. Be careful of the expiration date of any token you use, as an expired token could cause a scheduled run to fail. Additionally, user tokens can create a challenge if the user ever loses access to a specific repo.


Expand Down Expand Up @@ -246,7 +301,7 @@ Read more about creating a Personal Access Token [here](https://confluence.atlas



#### Configure subdirectory for packaged projects
## Configure subdirectory for packaged projects

In general, dbt expects `dbt_project.yml` to be located as a top-level file in a package. If the packaged project is instead nested in a subdirectory—perhaps within a much larger mono repo—you can optionally specify the folder path as `subdirectory`. dbt will attempt a [sparse checkout](https://git-scm.com/docs/git-sparse-checkout) of just the files located within that subdirectory. Note that you must be using a recent version of `git` (`>=2.26.0`).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: "Import and auto-generate exposures from dashboards and understand
image: /img/docs/cloud-integrations/auto-exposures/explorer-lineage2.jpg
---

# Configure auto-exposures <Lifecycle status="preview,enterprise" />
# Configure auto-exposures <Lifecycle status="enterprise" />

As a data team, it’s critical that you have context into the downstream use cases and users of your data products. [Auto-exposures](/docs/collaborate/auto-exposures) integrates natively with Tableau and [auto-generates downstream lineage](/docs/collaborate/auto-exposures#view-auto-exposures-in-dbt-explorer) in dbt Explorer for a richer experience.

Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/cloud-integrations/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Many data applications integrate with dbt Cloud, enabling you to leverage the po
<div className="grid--3-col">

<Card
title="Configure auto-exposures (preview)"
title="Configure auto-exposures"
body="Import and auto-generate exposures from dashboards to understand how models are used in downstream tools for a richer downstream lineage."
link="/docs/cloud-integrations/configure-auto-exposures"
icon="dbt-bit"/>
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/collaborate/auto-exposures.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ pagination_next: "docs/collaborate/data-tile"
image: /img/docs/cloud-integrations/auto-exposures/explorer-lineage.jpg
---

# Auto-exposures <Lifecycle status="preview,enterprise" />
# Auto-exposures <Lifecycle status="enterprise" />

As a data team, it’s critical that you have context into the downstream use cases and users of your data products. Auto-exposures integrate natively with Tableau (Power BI coming soon) and auto-generate downstream lineage in dbt Explorer for a richer experience.

Expand Down
88 changes: 88 additions & 0 deletions website/docs/docs/collaborate/data-health-signals.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
---
title: "Data health signals"
sidebar_label: "Data health signals"
id: data-health-signals
description: "Learn how data health signals offer a quick, at-a-glance view of data health when browsing your resources in dbt Explorer."
image: /img/docs/collaborate/dbt-explorer/data-health-signal.jpg
---

# Data health signals <Lifecycle status="preview" />
Data health signals offer a quick, at-a-glance view of data health when browsing your resources in dbt Explorer. They keep you informed on the status of your resource's health using the indicators **Healthy**, **Caution**, **Degraded**, or **Unknown**.

- Supported resources are [models](/docs/build/models), [sources](/docs/build/sources), and [exposures](/docs/build/exposures).
- For accurate health data, ensure the resource is up-to-date and had a recent job run.
- Each data health signal reflects key data health components, such as test success status, missing resource descriptions, missing tests, absence of builds in 30-day windows, [and more](#data-health-signal-criteria)

<Lightbox src="/img/docs/collaborate/dbt-explorer/data-health-signal.jpg" width="55%" title="View data health signals for your models."/>

## Access data health signals

Access data health signals in the following places:
- In the [search function](/docs/collaborate/explore-projects#search-resources) or under **Models**, **Sources**, or **Exposures** in the **Resource** tab.
- For sources, the data health signal also indicates the [source freshness](/docs/deploy/source-freshness) status.
- In the **Health** column on [each resource's details page](/docs/collaborate/explore-projects#view-resource-details). Hover over or click the signal to view detailed information.
- In the **Health** column of public models tables.
- In the [DAG lineage graph](/docs/collaborate/explore-projects#project-lineage). Click any node to open the node details panel where you can view it and its details.
- In [Data health tiles](/docs/collaborate/data-tile) through an embeddable iFrame and visible in your BI dashboard.

<Lightbox src="/img/docs/collaborate/dbt-explorer/data-health-signal.gif" width="95%" title="Access data health signals in multiple places in dbt Explorer."/>

## Data health signal criteria

Each resource has a health state that is determined by specific set of criteria. Select the following tabs to view the criteria for that resource type.
<Tabs>
<TabItem value="models" label="Models">

The health state of a model is determined by the following criteria:
<!-- TODO: remove the 'tbd' lines in the table once meta 4025 is done -->
| **Health state** | **Criteria** |
|-------------------|---------------|
|**Healthy** | All of the following must be true:<br /><br /> - Built successfully in the last run<br />- Built in the last 30 days<br />- Model has tests configured<br />- All tests passed<br />- All upstream [sources are fresh](/docs/build/sources#source-data-freshness) or freshness is not applicable (set to `null`)<br />- Has a description |
| 🟡 **Caution** | One of the following must be true: <br /><br />- Not built in the last 30 days<br />- Tests are not configured<br />- Tests return warnings<br />- One or more upstream sources are stale:<br />&nbsp;&nbsp;&nbsp;&nbsp;- Has a freshness check configured<br />&nbsp;&nbsp;&nbsp;&nbsp;- Freshness check ran in the past 30 days<br />&nbsp;&nbsp;&nbsp;&nbsp;- Freshness check returned a warning<br />- Missing a description |
| 🔴 **Degraded** | One of the following must be true: <br /><br />- Model failed to build<br />- Model has failing tests<br />- One or more upstream sources are stale:<br />&nbsp;&nbsp;&nbsp;&nbsp;- Freshness check hasn’t run in the past 30 days<br />&nbsp;&nbsp;&nbsp;&nbsp;- Freshness check returned an error |
|**Unknown** | - Unable to determine health of resource; no job runs have processed the resource. |

</TabItem>

<TabItem value="sources" label="Sources">

The health state of a source is determined by the following criteria:

| **Health state** | **Criteria** |
|-------------------|---------------|
| ✅ Healthy | All of the following must be true: <br /><br />- Freshness check configured<br />- Freshness check passed<br />- Freshness check ran in the past 30 days<br />- Has a description |
| 🟡 Caution | One of the following must be true: <br /><br />- Freshness check returned a warning<br />- Freshness check not configured<br />- Freshness check not run in the past 30 days<br />- Missing a description |
| 🔴 Degraded | - Freshness check returned an error |
| ⚪ Unknown | Unable to determine health of resource; no job runs have processed the resource. |

</TabItem>

<TabItem value="exposures" label="Exposures">

The health state of an exposure is determined by the following criteria:

| **Health state** | **Criteria** |
|-------------------|---------------|
| ✅ Healthy | All of the following must be true: <br /><br />- Underlying sources are fresh<br />- Underlying models built successfully<br />- Underlying models’ tests passing<br /><!-- - Freshness must be applicable <br /> - (TBD) Underlying models built in the last 30 days --> |
| 🟡 Caution | One of the following must be true: <br /><br />- At least one underlying source’s freshness checks returned a warning<br />- At least one underlying model was skipped<br />- At least one underlying model’s tests returned a warning<br /><!-- - (TBD) At least one model not built in the last 30 days --> |
| 🔴 Degraded | One of the following must be true: <br /><br />- At least one underlying source’s freshness checks returned an error<br />- At least one underlying model did not build successfully<br />- At least one model’s tests returned an error |

</TabItem>

<!-- TODO: Add source collection health once META-3973/3971 are completed
<TabItem value="source-collection" label="Source collection health">
The health state of a source collection is determined by the following criteria:
Functions as an aggregate of underlying sources
| **Health state** | **Criteria** |
|-------------------|---------------|
| ✅ Healthy | - All underlying sources have freshness checks configured OR<br />- All passed their freshness checks OR<br />- All freshness checks ran in the past 30 days OR<br /> - All sources have a description |
| 🟡 Caution | - One or more sources lack freshness checks OR<br />- One or more freshness checks returned a warning OR<br />- One or more freshness checks not run in the past 30 days OR<br />- One or more sources missing a description |
| 🔴 Degraded | - One or more underlying sources’ freshness checks returned error |
</TabItem>
-->

</Tabs>
10 changes: 5 additions & 5 deletions website/docs/docs/collaborate/data-tile.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,24 @@
title: "Data health tile"
id: "data-tile"
sidebar_label: "Data health tile"
description: "Embed data health tiles in your dashboards to distill trust signals for data consumers."
description: "Embed data health tiles in your dashboards to distill data health signals for data consumers."
image: /img/docs/collaborate/dbt-explorer/data-tile-pass.jpg
---

With data health tiles, stakeholders will get an at-a-glance confirmation on whether the data they’re looking at is stale or degraded. This trust signal allows teams to immediately go back into Explorer to see more details and investigate issues.
With data health tiles, stakeholders will get an at-a-glance confirmation on whether the data they’re looking at is stale or degraded. It allows teams to immediately go back into Explorer to see more details and investigate issues.

The data health tile:

- Distills trust signals for data consumers.
- Distills [data health signals](/docs/collaborate/data-health-signals) for data consumers.
- Deep links you into dbt Explorer where you can further dive into upstream data issues.
- Provides richer information and makes it easier to debug.
- Revamps the existing, [job-based tiles](#job-based-data-health).

Data health tiles rely on [exposures](/docs/build/exposures) to surface trust signals in your dashboards. When you configure exposures in your dbt project, you are explicitly defining how specific outputs—like dashboards or reports—depend on your data models.
Data health tiles rely on [exposures](/docs/build/exposures) to surface data health signals in your dashboards. When you configure exposures in your dbt project, you are explicitly defining how specific outputs—like dashboards or reports—depend on your data models.

<DocCarousel slidesPerView={1}>
<Lightbox src="/img/docs/collaborate/dbt-explorer/data-tile-pass.jpg" width="60%" title="Example of passing Data health tile in your dashboard." />
<Lightbox src="/img/docs/collaborate/dbt-explorer/data-tiles.png" width="60%" title="Embed data health tiles in your dashboards to distill trust signals for data consumers." />
<Lightbox src="/img/docs/collaborate/dbt-explorer/data-tiles.png" width="60%" title="Embed data health tiles in your dashboards to distill data health signals for data consumers." />
</DocCarousel>

## Prerequisites
Expand Down
Loading

0 comments on commit 6c431d7

Please sign in to comment.