diff --git a/latest/getting_started/index.html b/latest/getting_started/index.html index 01513d44..381d0fe4 100644 --- a/latest/getting_started/index.html +++ b/latest/getting_started/index.html @@ -735,17 +735,31 @@
Run dbt-bouncer
as part of your CI pipeline:
-
steps:
- ...
-
- - uses: godatadriven/dbt-bouncer@vX.X
- with:
- config-file: ./<PATH_TO_CONFIG_FILE>
- output-file: results.json # optional, default does not save a results file
- send-pr-comment: true # optional, defaults to true
- verbose: false # optional, defaults to false
-
- ...
+name: CI pipeline
+
+on:
+ pull_request:
+ branches:
+ - main
+
+jobs:
+ run-dbt-bouncer:
+ permissions:
+ pull-requests: write # Required to write a comment on the PR
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout
+ uses: actions/checkout@v4
+
+ - name: Generate or fetch dbt artifacts
+ run: ...
+
+ - uses: godatadriven/dbt-bouncer@vX.X
+ with:
+ config-file: ./<PATH_TO_CONFIG_FILE>
+ output-file: results.json # optional, default does not save a results file
+ send-pr-comment: true # optional, defaults to true
+ verbose: false # optional, defaults to false
We recommend pinning both a major and minor version number.
Docker#
@@ -857,7 +871,7 @@ How to add a custom check to <
September 8, 2024
+ September 18, 2024
-
diff --git a/latest/search/search_index.json b/latest/search/search_index.json
index eb88db42..dadc0964 100644
--- a/latest/search/search_index.json
+++ b/latest/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":""},{"location":"#welcome-to-dbt-bouncer","title":"Welcome to dbt-bouncer","text":"dbt-bouncer
is an open-source tool that allows you to configure and enforce conventions for your dbt project. The conventions are run against dbt's artifact files (think ./target/manifest.json
) resulting in speedy tests. Conventions can be specified in a .yml
file, allowing maximum customisation to the conventions you wish to follow (or create \ud83d\ude00).
Check out our Getting Started
guide.
"},{"location":"#terminology","title":"Terminology","text":" - Check: A check is a rule run against a dbt artifact.
- Config file: A
.yml
file that specifies which checks to run along with any parameters. - dbt artifacts directory: The directory that contains the dbt artifacts (
manifest.json
, etc.), generally this is ./target
.
"},{"location":"#aims","title":"Aims","text":"dbt-bouncer
aims to:
- Provide a 100% configurable way to enforce conventions in a dbt project.
- Be as fast as possible, running checks against dbt artifacts.
- Be as easy as possible to use, with a simple config file written in
YML
. - Be as flexible as possible, allowing checks to be written in python.
- Provide immediate feedback when run as part of a CI pipeline.
"},{"location":"#about","title":"About","text":"dbt-bouncer
is free software, released under the MIT license. It originated at Xebia Data in Amsterdam, Netherlands. Source code is available on GitHub.
All contributions, in the form of bug reports, pull requests, feedback or discussion are welcome. See the contributing guide for more information.
"},{"location":"CONTRIBUTING/","title":"Contributing to dbt-bouncer
","text":"dbt-bouncer
is open source software. Whether you are a seasoned open source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project.
"},{"location":"CONTRIBUTING/#about-this-document","title":"About this document","text":"There are many ways to contribute to the ongoing development of dbt-bouncer
, such as by participating in discussions and issues.
The rest of this document serves as a more granular guide for contributing code changes to dbt-bouncer
(this repository). It is not intended as a guide for using dbt-bouncer
, and some pieces assume a level of familiarity with Python development (virtualenvs, Poetry
, etc). Specific code snippets in this guide assume you are using macOS or Linux and are comfortable with the command line.
If you get stuck, we're happy to help! Just open an issue or draft PR and we'll do our best to help out.
"},{"location":"CONTRIBUTING/#note","title":"Note","text":" - Branches: All pull requests from community contributors should target the
main
branch (default).
"},{"location":"CONTRIBUTING/#getting-the-code","title":"Getting the code","text":""},{"location":"CONTRIBUTING/#installing-git","title":"Installing git","text":"You will need git
in order to download and modify the dbt-bouncer
source code. On macOS, the best way to download git is to just install Xcode.
"},{"location":"CONTRIBUTING/#contributors","title":"Contributors","text":"You can contribute to dbt-bouncer
by forking the dbt-bouncer
repository. For a detailed overview on forking, check out the GitHub docs on forking. In short, you will need to:
- Fork the
dbt-bouncer
repository. - Clone your fork locally.
- Check out a new branch for your proposed changes.
- Push changes to your fork.
- Open a pull request against
godatadriven/dbt-bouncer
from your forked repository.
"},{"location":"CONTRIBUTING/#setting-up-an-environment","title":"Setting up an environment","text":"There are some tools that will be helpful to you in developing locally. While this is the list relevant for dbt-bouncer
development, many of these tools are used commonly across open-source python projects.
"},{"location":"CONTRIBUTING/#tools","title":"Tools","text":"These are the tools used in dbt-bouncer
development and testing:
click
to create our CLI interface. - GitHub Actions for automating tests and checks, once a PR is pushed to the
dbt-bouncer
repository. make
to run multiple setup or test steps in combination. mypy
for static type checking. Poetry
to manage our python virtual environment. pre-commit
to easily run those checks. Pydantic
to validate our configuration file. pytest
to define, discover, and run tests. Ruff
to lint and format python code.
A deep understanding of these tools in not required to effectively contribute to dbt-bouncer
, but we recommend checking out the attached documentation if you're interested in learning more about each one.
"},{"location":"CONTRIBUTING/#virtual-environments","title":"Virtual environments","text":"We strongly recommend using virtual environments when developing code in dbt-bouncer
. We recommend creating this virtualenv in the root of the dbt-bouncer
repository. To create a new virtualenv, run:
poetry shell\n
This will create a new Python virtual environment.
"},{"location":"CONTRIBUTING/#setting-environment-variables","title":"Setting environment variables","text":"Set required environment variables by copying .env.example
to .env
and updating the values.
"},{"location":"CONTRIBUTING/#running-dbt-bouncer-in-development","title":"Running dbt-bouncer
in development","text":""},{"location":"CONTRIBUTING/#installation","title":"Installation","text":"First make sure that you set up your virtualenv
as described in Setting up an environment. Next, install dbt-bouncer
, its dependencies and pre-commit
:
poetry install\npoetry run pre-commit install\n
When installed in this way, any changes you make to your local copy of the source code will be reflected immediately in your next dbt-bouncer
run.
"},{"location":"CONTRIBUTING/#running-dbt-bouncer","title":"Running dbt-bouncer
","text":"With your virtualenv activated, the dbt-bouncer
script should point back to the source code you've cloned on your machine. You can verify this by running which dbt-bouncer
. This command should show you a path to an executable in your virtualenv. You can run dbt-bouncer
using the provided example configuration file via:
poetry run dbt-bouncer --config-file dbt-bouncer-example.yml\n
"},{"location":"CONTRIBUTING/#testing","title":"Testing","text":"Once you're able to manually test that your code change is working as expected, it's important to run existing automated tests, as well as adding some new ones. These tests will ensure that: - Your code changes do not unexpectedly break other established functionality - Your code changes can handle all known edge cases - The functionality you're adding will keep working in the future
"},{"location":"CONTRIBUTING/#note_1","title":"Note","text":" - Generating dbt artifacts: If you change the configuration of the dbt project located in
dbt_project
then you will need to re-generate the dbt artifacts used in testing. To do so, run:
make build-artifacts\n
"},{"location":"CONTRIBUTING/#test-commands","title":"Test commands","text":"There are a few methods for running tests locally.
"},{"location":"CONTRIBUTING/#makefile","title":"makefile
","text":"There are multiple targets in the makefile
to run common test suites, most notably:
# Runs unit tests\nmake test-unit\n\n# Runs integration tests\nmake test-integration\n\n# Runs all tests\nmake test\n
"},{"location":"CONTRIBUTING/#pre-commit","title":"pre-commit
","text":"pre-commit
takes care of running all code-checks for formatting and linting. Run poetry run pre-commit install
to install pre-commit
in your local environment. Once this is done you can use the git pre-commit hooks to ensure proper formatting and linting.
"},{"location":"CONTRIBUTING/#pytest","title":"pytest
","text":"Finally, you can also run a specific test or group of tests using pytest
directly. With a virtualenv active and dev dependencies installed you can do things like:
# run all unit tests in a file\npoetry run pytest ./tests/unit/checks/catalog/test_columns.py\n\n# run a specific unit test\npoetry run pytest ./tests/unit/checks/catalog/test_columns.py::test_check_columns_are_documented_in_public_models\n
See pytest usage docs for an overview of useful command-line options.
"},{"location":"CONTRIBUTING/#assorted-development-tips","title":"Assorted development tips","text":" - Append
# type: ignore
to the end of a line if you need to disable mypy
on that line, preferably with the specific rule to ignore such as # type: ignore[union-attr]
.
"},{"location":"CONTRIBUTING/#adding-a-new-check","title":"Adding a new check","text":"To add a new check follow the below steps:
- In
./src/dbt_bouncer/checks
choose the appropriate directory for your check. For example, if your check only requires the manifest.json
then use the manifest
directory, if your check requires the catalog.json
then use the catalog
directory. - Within the chosen directory assess if a suitable file already exists. For example, if your check applies to a model then
manifest/check_models.py
is a suitable location. -
Within the chosen file, add a Pydantic model, this object must meet the following criteria:
- Start with \"Check\".
- Inherit from
dbt_bouncer.check_base.BaseCheck
. - Have a
name
attribute that is a string whose value is the snake case equivalent of the class name. - A
default
value provided for optional input arguments and arguments that are received at execution time. - Have a doc string that includes a description of the check, a list of possible input parameters and at least one example.
- A clear message in the event of a failure.
-
After the check is added, add the check to dbt-bouncer-example.yml
and run dbt-bouncer --config-file dbt-bouncer-example.yml
to ensure the check succeeds.
- (Optional) If the dbt project located in
./dbt_project
needs to be updated then do so and also run make build-artifacts
to generate the new test artifacts. - Add at least one happy path and one unhappy path test to
./tests
. The appropriate test file will be the one matching the directory of the check. For example, if the check is in ./src/dbt_bouncer/checks/catalog/check_columns.py
then the test file will be ./tests/unit/checks/catalog/test_columns.py
. - Run
make test
to ensure the tests pass. - Open a PR \ud83c\udf89!
"},{"location":"CONTRIBUTING/#submitting-a-pull-request","title":"Submitting a Pull Request","text":"Code can be merged into the current development branch main
by opening a pull request. If the proposal looks like it's on the right track, then a dbt-bouncer
maintainer will review the PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code. Once merged, your contribution will be available for the next release of dbt-bouncer
.
Automated tests run via GitHub Actions. If you're a first-time contributor, all tests will require a maintainer to approve.
Once all tests are passing and your PR has been approved, a dbt-bouncer
maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada:
"},{"location":"cli/","title":"CLI","text":"This page provides documentation for the dbt-bouncer
CLI.
"},{"location":"cli/#dbt-bouncer","title":"dbt-bouncer","text":"Entrypoint for dbt-bouncer.
Raises: RuntimeError: If output file has an invalid extension.
Usage:
dbt-bouncer [OPTIONS]\n
Options:
--config-file PUREPATH Location of the YML config file.\n --output-file PATH Location of the json file where check metadata will\n be saved.\n -v, --verbosity Verbosity.\n --version Show the version and exit.\n --help Show this message and exit.\n
"},{"location":"cli/#exit-codes","title":"Exit codes","text":"dbt-bouncer
returns the following exit codes:
-
0
: All checks have succeeded.
-
1
:
- At least one check has failed. Check the logs for more information.
- A fatal error occurred during check setup or check execution. Check the logs for more information.
"},{"location":"config_file/","title":"Config file","text":"dbt-bouncer
requires a config file which determines what checks are run. The following options are available, in order of priority:
- A file passed via the
--config-file
CLI flag. - A file named
dbt-bouncer.yml
in the current working directory. - A
[tool.dbt-bouncer]
section in pyproject.toml
.
Here is an example config file in yaml
:
# [Optional] Directory where the dbt artifacts exists, generally the `target` directory inside a dbt project. Defaults to `./target`.\ndbt_artifacts_dir: target\n\nmanifest_checks:\n - name: check_macro_name_matches_file_name\n - name: check_model_names\n include: ^models/staging\n model_name_pattern: ^stg_\n
And the same config in toml
:
[tool.dbt-bouncer]\n# [Optional] Directory where the dbt artifacts exists, generally the `target` directory inside a dbt project. Defaults to `./target`.\ndbt_artifacts_dir = \"target\"\n\n[[tool.dbt-bouncer.manifest_checks]]\nname = \"check_macro_name_matches_file_name\"\n\n[[tool.dbt-bouncer.manifest_checks]]\nname = \"check_model_names\"\ninclude = \"^models/staging\"\nmodel_name_pattern = \"^stg_\"\n
For more example config files, see here.
"},{"location":"config_file/#common-arguments","title":"Common arguments","text":""},{"location":"config_file/#exclude-and-include","title":"Exclude and Include","text":"Most (but not all) checks accept the following optional arguments:
exclude
: Regexp to match which original file paths to exclude. include
: Regexp to match which original file paths to include.
Example per resource type:
Exposures
: The original file path to the properties file where the source is defined, e.g. ^models/marts/finance
will match exposures defined in ./models/marts/finance/_exposures.yml
. Macros
: The original file path to the macro file, e.g. ^macros/system
will match files like ./macros/system/generate_schema_name.sql
. Models
: The original file path to the model file, e.g. ^marts
will match files like ./models/marts/customers.sql
. Run results
: The original file path to the file associated with the resource, e.g. ^seeds/finance
will match seeds in ./seeds/finance
, ^models/staging
will match models and tests in ./models/staging
. Semantic models
: The original file path to the properties file where the semantic model is defined, e.g. ^models/marts/finance
will match semantic models defined in ./models/marts/finance/_finance__semantic_models.yml
. Sources
: The original file path to the properties file where the source is defined, e.g. ^models/staging/crm
will match sources defined in ./models/staging/crm/_crm__sources.yml
. Unit tests
: The original file path to the properties file where the unit test is defined, e.g. ^models/staging/crm
will match unit tests defined in ^staging/crm/_stg_crm__unit_tests.yml
.
To determine if a check accepts these arguments view the Checks page.
Note
exclude
and include
can be specified at both the check level and the global level. Should both levels be specified, then the check level is applied. All the below examples result in the check_model_names
check being run on all models in ./models/staging
:
# Specify `include` at the check level only\nmanifest_checks:\n - name: check_model_names\n include: ^models/staging\n model_name_pattern: ^stg_\n
# Specify `include` at the check and global levels\ninclude: ^models/marts\nmanifest_checks:\n - name: check_model_names\n include: ^models/staging\n model_name_pattern: ^stg_\n
# Specify `include` at the global level only\ninclude: ^models/staging\nmanifest_checks:\n - name: check_model_names\n model_name_pattern: ^stg_\n
Note
When compiled on Windows machines, keys such as original_file_path
, patch_path
and path
take the form:
models\\\\staging\\\\crm\\\\model_1.sql\n
When compiled on Linux and Mac machines, these same keys take the form:
models/staging/crm/model_1.sql\n
dbt-bouncer
converts all of these paths to the Linux/Mac form, hence when you are supplying values to exclude
and include
you should use the Linux/Mac form.
"},{"location":"config_file/#severity","title":"Severity","text":"All checks accept a severity
argument, valid values are:
error
: If the check fails then dbt-bouncer
will return a non-zero exit code. warn
: If the check fails then dbt-bouncer
will return a non-zero exit code.
severity
can also be specified globally, this is useful when applying dbt-bouncer
to a pre-existing dbt project. It allows you to run dbt-bouncer
, identify the checks that fail and address the failures in your own time without receiving non-zero exit codes:
# Specify `severity` at the global levels: all checks will have a `warn` severity, avoiding non-zero exit codes.\nseverity: warn\n\nmanifest_checks:\n - name: check_exposure_based_on_view\n ...\n
Note
severity
can be specified at both the check level and the global level. Should both levels be specified, then the global level is applied.
# No `severity` specified: check will have an `error` severity.\nmanifest_checks:\n - name: check_exposure_based_on_view\n
# Specify `severity` at the check level only: check will have a `warn` severity.\nmanifest_checks:\n - name: check_exposure_based_on_view\n severity: warn\n
# Specify `severity` at the check and global levels: check will have a `warn` severity.\nseverity: warn\nmanifest_checks:\n - name: check_exposure_based_on_view\n severity: error\n
# Specify `severity` at the global level only: check will have a `warn` severity.\nseverity: warn\nmanifest_checks:\n - name: check_exposure_based_on_view\n
"},{"location":"faq/","title":"Frequently Asked Questions","text":""},{"location":"faq/#can-other-tools-perform-the-same-checks-as-dbt-bouncer","title":"Can other tools perform the same checks as dbt-bouncer
?","text":"There are several other tools that perform similar tasks as dbt-bouncer
.
- dbt-checkpoint: A collection of
pre-commit
hooks for dbt projects. Tests are written in python. Configuration is performed via .pre-commit-config.yaml
. Provided the dbt artifacts have already been generated, dbt-checkpoint
does not need access to the underlying database. The hooks execute when a new commit is made, as such dbt-checkpoint
is designed to be run only as part of pre-commit
. - dbt-project-evaluator: This is a dbt package from dbt Labs. Tests are written in
.sql
files using a combination of Jinja and SQL. Configuration is performed via dbt_project.yml
and seed files (i.e. csv files). Requires a connection to underlying database. Designed to be run both in a CI pipeline and also during active development. - dbt-score: This is a python package installable via
pip
. A collection of tests that apply only to dbt models. Tests can be executed from the command line. Tests are written in python. Configuration is performed via a pyproject.toml
file. Provided the dbt artifacts have already been generated, dbt-score
does not need access to the underlying database. Designed to be run during development.
While the above tools inhabit the same space as dbt-bouncer
they do not provide what we consider to be the optimum experience that dbt-bouncer
provides:
- Designed to run both locally and in a CI pipeline.
- Configurable via a file format,
YML
, that dbt developers are already familiar with. - Does not require database access.
- Can run tests against any of dbt's artifacts.
- Allows tests to be written in python.
As such we consider dbt-bouncer
to be the best tool to enforce conventions in a dbt project.
Tip
dbt-bouncer
can perform all the tests currently included in dbt-checkpoint
, dbt-project-evaluator
and dbt-score
. If you see an existing test that is not possible with dbt-bouncer
, open an issue and we'll add it!
"},{"location":"faq/#how-to-set-up-dbt-bouncer-in-a-monorepo","title":"How to set up dbt-bouncer
in a monorepo?","text":"A monorepo may consist of one directory with a dbt project and other directories with unrelated code. It may be desired for dbt-bouncer
to be configured from the root directory. Sample directory tree:
.\n\u251c\u2500\u2500 dbt-bouncer.yml\n\u251c\u2500\u2500 README.md\n\u251c\u2500\u2500 dbt-project\n\u2502 \u251c\u2500\u2500 models\n\u2502 \u251c\u2500\u2500 dbt_project.yml\n\u2502 \u2514\u2500\u2500 profiles.yml\n\u2514\u2500\u2500 package-a\n \u251c\u2500\u2500 src\n \u251c\u2500\u2500 tests\n \u2514\u2500\u2500 package.json\n
To ease configuration you can use exclude
or include
at the global level (see Config File for more details). For the above example dbt-bouncer.yml
could be configured as:
dbt_artifacts_dir: dbt-project/target\ninclude: ^dbt-project\n\nmanifest_checks:\n - name: check_exposure_based_on_non_public_models\n
dbt-bouncer
can now be run from the root directory.
"},{"location":"getting_started/","title":"Getting Started","text":""},{"location":"getting_started/#how-to-run-dbt-bouncer","title":"How to run dbt-bouncer
","text":" -
Generate dbt artifacts by running a dbt command:
dbt parse
to generate a manifest.json
artifact. dbt docs generate
to generate a catalog.json
artifact (necessary if you are using catalog checks). dbt run
(or any other command that implies it e.g. dbt build
) to generate a run_results.json
artifact (necessary if you are using run results checks).
-
Create a dbt-bouncer.yml
config file, details here.
-
Run dbt-bouncer
to validate that your conventions are being maintained.
"},{"location":"getting_started/#installing-with-python","title":"Installing with Python","text":"Install from pypi.org:
pip install dbt-bouncer # or via any other package manager\n
Run:
dbt-bouncer --config-file <PATH_TO_CONFIG_FILE>\n
Running dbt-bouncer (X.X.X)...\nLoaded config from dbt-bouncer-example.yml...\nValidating conf...\n
dbt-bouncer
also supports a verbose mode, run:
dbt-bouncer --config-file <PATH_TO_CONFIG_FILE> -v\n
Running dbt-bouncer (X.X.X)...\nconfig_file=PosixPath('dbt-bouncer-example.yml')\nconfig_file_source='COMMANDLINE'\nConfig file passed via command line: dbt-bouncer-example.yml\nLoading config from /home/pslattery/repos/dbt-bouncer/dbt-bouncer-example.yml...\nLoading config from dbt-bouncer-example.yml...\nLoaded config from dbt-bouncer-example.yml...\nconf={'dbt_artifacts_dir': 'dbt_project/target', 'catalog_checks': [{'name': 'check_column_name_complies_to_column_type', 'column_name_pattern': '^is_.*', 'exclude': '^staging', 'types': ['BOOLEAN']}]}\nValidating conf...\n
"},{"location":"getting_started/#running-as-an-executable-using-uv","title":"Running as an executable using uv","text":"Run dbt-bouncer
as a standalone Python executable using uv
:
uvx dbt-bouncer --config-file <PATH_TO_CONFIG_FILE>\n
"},{"location":"getting_started/#github-actions","title":"GitHub Actions","text":"Run dbt-bouncer
as part of your CI pipeline:
steps:\n ...\n\n - uses: godatadriven/dbt-bouncer@vX.X\n with:\n config-file: ./<PATH_TO_CONFIG_FILE>\n output-file: results.json # optional, default does not save a results file\n send-pr-comment: true # optional, defaults to true\n verbose: false # optional, defaults to false\n\n ...\n
We recommend pinning both a major and minor version number.
"},{"location":"getting_started/#docker","title":"Docker","text":"Run dbt-bouncer
via Docker:
docker run --rm \\\n --volume \"$PWD\":/app \\\n ghcr.io/godatadriven/dbt-bouncer:vX.X.X \\\n --config-file /app/<PATH_TO_CONFIG_FILE>\n
"},{"location":"getting_started/#pex","title":"Pex","text":"You can also run the .pex
(Python EXecutable) artifact directly once you have a python executable (3.8 -> 3.12) installed:
wget https://github.com/godatadriven/dbt-bouncer/releases/download/vX.X.X/dbt-bouncer.pex -O dbt-bouncer.pex\n\npython dbt-bouncer.pex --config-file $PWD/<PATH_TO_CONFIG_FILE>\n
"},{"location":"getting_started/#how-to-contribute-a-check-to-dbt-bouncer","title":"How to contribute a check to dbt-bouncer
","text":"See Adding a new check.
"},{"location":"getting_started/#how-to-add-a-custom-check-to-dbt-bouncer","title":"How to add a custom check to dbt-bouncer
","text":"In addition to the checks built into dbt-bouncer
, the ability to add custom checks is supported. This allows users to write checks that are specific to the conventions of their projects. To add a custom check:
- Create an empty directory and add a
custom_checks_dir
key to your config file. The value of this key should be the path to the directory you just created, relative to where the config file is located. - In this directory create an empty
__init__.py
file. - In this directory create a subdirectory named
catalog
, manifest
or run_results
depending on the type of artifact you want to check. -
In this subdirectory create a python file that defines a check. The check must meet the following criteria:
- Start with \"Check\".
- Inherit from dbt_bouncer.check_base.BaseCheck.
- Have a name attribute that is a string whose value is the snake case equivalent of the class name.
- A default value provided for optional input arguments and arguments that are received at execution time.
- Have a doc string that includes a description of the check, a list of possible input parameters and at least one example.
- A clear message in the event of a failure.
-
In your config file, add the name of the check and any desired arguments.
- Run
dbt-bouncer
, your custom check will be executed.
An example:
-
Directory tree:
.\n\u251c\u2500\u2500 dbt-bouncer.yml\n\u251c\u2500\u2500 dbt_project.yml\n\u251c\u2500\u2500 my_custom_checks\n| \u251c\u2500\u2500 __init__.py\n| \u2514\u2500\u2500 manifest\n| \u2514\u2500\u2500 check_custom_to_me.py\n\u2514\u2500\u2500 target\n \u2514\u2500\u2500 manifest.json\n
-
Contents of check_custom_to_me.py
:
from typing import TYPE_CHECKING, Literal\n\nfrom pydantic import Field\n\nfrom dbt_bouncer.check_base import BaseCheck\n\nif TYPE_CHECKING:\n import warnings\n\n with warnings.catch_warnings():\n warnings.filterwarnings(\"ignore\", category=UserWarning)\n from dbt_bouncer.parsers import DbtBouncerModelBase\n\n\nclass CheckModelDepcrecationDate(BaseCheck):\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_deprecation_date\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n\n assert self.model.deprecation_date is not None, f\"`{self.model.name}` requires a `deprecation_date` to be set.\"\n
-
Contents of dbt-bouncer.yml
:
custom_checks_dir: my_custom_checks\n\nmanifest_checks:\n - name: check_model_deprecation_date\n include: ^models/staging/legacy_erp\n
"},{"location":"checks/","title":"Checks","text":"dbt-bouncer
runs checks against artifacts from dbt. These checks fall into three categories:
- Catalog checks:
- Catalog Sources
- Columns
- Manifest checks:
- Exposures
- Lineage
- Macros
- Metadata
- Models
- Semantic Models
- Sources
- Unit Tests
- Run Results checks:
- Run Results
"},{"location":"checks/catalog/check_catalog_sources/","title":"Catalog Checks: Catalog Sources","text":"Note
The below checks require both catalog.json
and manifest.json
to be present.
"},{"location":"checks/catalog/check_catalog_sources/#catalog.check_catalog_sources.CheckSourceColumnsAreAllDocumented","title":"CheckSourceColumnsAreAllDocumented
","text":"All columns in a source should be included in the source's properties file, i.e. .yml
file.
Receives at execution time:
Name Type Description catalog_source
CatalogTable
The CatalogTable object to check.
sources
List[DbtBouncerSourceBase]
List of DbtBouncerSourceBase objects parsed from catalog.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
catalog_checks:\n - name: check_source_columns_are_all_documented\n
Source code in src/dbt_bouncer/checks/catalog/check_catalog_sources.py
class CheckSourceColumnsAreAllDocumented(BaseCheck):\n \"\"\"All columns in a source should be included in the source's properties file, i.e. `.yml` file.\n\n Receives:\n catalog_source (CatalogTable): The CatalogTable object to check.\n sources (List[DbtBouncerSourceBase]): List of DbtBouncerSourceBase objects parsed from `catalog.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n catalog_checks:\n - name: check_source_columns_are_all_documented\n ```\n\n \"\"\"\n\n catalog_source: \"CatalogTable\" = Field(default=None)\n name: Literal[\"check_source_columns_are_all_documented\"]\n sources: List[\"DbtBouncerSourceBase\"] = Field(default=[])\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n source = next(\n s for s in self.sources if s.unique_id == self.catalog_source.unique_id\n )\n undocumented_columns = [\n v.name\n for _, v in self.catalog_source.columns.items()\n if v.name not in source.columns\n ]\n assert not undocumented_columns, f\"`{self.catalog_source.unique_id}` has columns that are not included in the sources properties file: {undocumented_columns}\"\n
"},{"location":"checks/catalog/check_columns/","title":"Catalog Checks: Columns","text":"Note
The below checks require both catalog.json
and manifest.json
to be present.
"},{"location":"checks/catalog/check_columns/#catalog.check_columns.CheckColumnDescriptionPopulated","title":"CheckColumnDescriptionPopulated
","text":"Columns must have a populated description.
Receives at execution time:
Name Type Description catalog_node
CatalogTable
The CatalogTable object to check.
models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_column_description_populated\n include: ^models/marts\n
Source code in src/dbt_bouncer/checks/catalog/check_columns.py
class CheckColumnDescriptionPopulated(BaseCheck):\n \"\"\"Columns must have a populated description.\n\n Receives:\n catalog_node (CatalogTable): The CatalogTable object to check.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_column_description_populated\n include: ^models/marts\n ```\n\n \"\"\"\n\n catalog_node: \"CatalogTable\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_column_description_populated\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if self.catalog_node.unique_id.split(\".\")[0] == \"model\":\n model = next(\n m for m in self.models if m.unique_id == self.catalog_node.unique_id\n )\n non_complying_columns = []\n for _, v in self.catalog_node.columns.items():\n if (\n model.columns.get(v.name) is None\n or len(model.columns[v.name].description.strip()) <= 4\n ):\n non_complying_columns.append(v.name)\n\n assert not non_complying_columns, f\"`{self.catalog_node.unique_id.split('.')[-1]}` has columns that do not have a populated description: {non_complying_columns}\"\n
"},{"location":"checks/catalog/check_columns/#catalog.check_columns.CheckColumnNameCompliesToColumnType","title":"CheckColumnNameCompliesToColumnType
","text":"Columns with specified data types must comply to the specified regexp naming pattern.
Parameters:
Name Type Description Default column_name_pattern
str
Regex pattern to match the model name.
required types
List[str]
List of data types to check.
required Receives at execution time:
Name Type Description catalog_node
CatalogTable
The CatalogTable object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
catalog_checks:\n # DATE columns must end with \"_date\"\n - name: check_column_name_complies_to_column_type\n column_name_pattern: .*_date$\n types:\n - DATE\n
catalog_checks:\n # BOOLEAN columns must start with \"is_\"\n - name: check_column_name_complies_to_column_type\n column_name_pattern: ^is_.*\n types:\n - BOOLEAN\n
catalog_checks:\n # Columns of all types must consist of lowercase letters and underscores. Note that the specified types depend on the underlying database.\n - name: check_column_name_complies_to_column_type\n column_name_pattern: ^[a-z_]*$\n types:\n - BIGINT\n - BOOLEAN\n - DATE\n - DOUBLE\n - INTEGER\n - VARCHAR\n
Source code in src/dbt_bouncer/checks/catalog/check_columns.py
class CheckColumnNameCompliesToColumnType(BaseCheck):\n \"\"\"Columns with specified data types must comply to the specified regexp naming pattern.\n\n Parameters:\n column_name_pattern (str): Regex pattern to match the model name.\n types (List[str]): List of data types to check.\n\n Receives:\n catalog_node (CatalogTable): The CatalogTable object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n catalog_checks:\n # DATE columns must end with \"_date\"\n - name: check_column_name_complies_to_column_type\n column_name_pattern: .*_date$\n types:\n - DATE\n ```\n ```yaml\n catalog_checks:\n # BOOLEAN columns must start with \"is_\"\n - name: check_column_name_complies_to_column_type\n column_name_pattern: ^is_.*\n types:\n - BOOLEAN\n ```\n ```yaml\n catalog_checks:\n # Columns of all types must consist of lowercase letters and underscores. Note that the specified types depend on the underlying database.\n - name: check_column_name_complies_to_column_type\n column_name_pattern: ^[a-z_]*$\n types:\n - BIGINT\n - BOOLEAN\n - DATE\n - DOUBLE\n - INTEGER\n - VARCHAR\n ```\n\n \"\"\"\n\n catalog_node: \"CatalogTable\" = Field(default=None)\n column_name_pattern: str\n name: Literal[\"check_column_name_complies_to_column_type\"]\n types: List[str]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n non_complying_columns = [\n v.name\n for _, v in self.catalog_node.columns.items()\n if v.type in self.types\n and re.compile(self.column_name_pattern.strip()).match(v.name) is None\n ]\n\n assert not non_complying_columns, f\"`{self.catalog_node.unique_id.split('.')[-1]}` has columns that don't comply with the specified regexp pattern (`{self.column_name_pattern}`): {non_complying_columns}\"\n
"},{"location":"checks/catalog/check_columns/#catalog.check_columns.CheckColumnsAreAllDocumented","title":"CheckColumnsAreAllDocumented
","text":"All columns in a model should be included in the model's properties file, i.e. .yml
file.
Receives at execution time:
Name Type Description catalog_node
CatalogTable
The CatalogTable object to check.
models
List[DbtBouncerModel]
List of DbtBouncerModel objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
catalog_checks:\n - name: check_columns_are_all_documented\n
Source code in src/dbt_bouncer/checks/catalog/check_columns.py
class CheckColumnsAreAllDocumented(BaseCheck):\n \"\"\"All columns in a model should be included in the model's properties file, i.e. `.yml` file.\n\n Receives:\n catalog_node (CatalogTable): The CatalogTable object to check.\n models (List[DbtBouncerModel]): List of DbtBouncerModel objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n catalog_checks:\n - name: check_columns_are_all_documented\n ```\n\n \"\"\"\n\n catalog_node: \"CatalogTable\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_columns_are_all_documented\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if self.catalog_node.unique_id.split(\".\")[0] == \"model\":\n model = next(\n m for m in self.models if m.unique_id == self.catalog_node.unique_id\n )\n undocumented_columns = [\n v.name\n for _, v in self.catalog_node.columns.items()\n if v.name not in model.columns\n ]\n assert not undocumented_columns, f\"`{self.catalog_node.unique_id.split('.')[-1]}` has columns that are not included in the models properties file: {undocumented_columns}\"\n
"},{"location":"checks/catalog/check_columns/#catalog.check_columns.CheckColumnsAreDocumentedInPublicModels","title":"CheckColumnsAreDocumentedInPublicModels
","text":"Columns should have a populated description in public models.
Receives at execution time:
Name Type Description catalog_node
CatalogTable
The CatalogTable object to check.
models
List[DbtBouncerModel]
List of DbtBouncerModel objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
catalog_checks:\n - name: check_columns_are_documented_in_public_models\n
Source code in src/dbt_bouncer/checks/catalog/check_columns.py
class CheckColumnsAreDocumentedInPublicModels(BaseCheck):\n \"\"\"Columns should have a populated description in public models.\n\n Receives:\n catalog_node (CatalogTable): The CatalogTable object to check.\n models (List[DbtBouncerModel]): List of DbtBouncerModel objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n catalog_checks:\n - name: check_columns_are_documented_in_public_models\n ```\n\n \"\"\"\n\n catalog_node: \"CatalogTable\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_columns_are_documented_in_public_models\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if self.catalog_node.unique_id.split(\".\")[0] == \"model\":\n model = next(\n m for m in self.models if m.unique_id == self.catalog_node.unique_id\n )\n non_complying_columns = []\n for _, v in self.catalog_node.columns.items():\n if model.access.value == \"public\":\n column_config = model.columns.get(v.name)\n if (\n column_config is None\n or len(column_config.description.strip()) < 4\n ):\n non_complying_columns.append(v.name)\n\n assert not non_complying_columns, f\"`{self.catalog_node.unique_id.split('.')[-1]}` is a public model but has columns that don't have a populated description: {non_complying_columns}\"\n
"},{"location":"checks/catalog/check_columns/#catalog.check_columns.CheckColumnHasSpecifiedTest","title":"CheckColumnHasSpecifiedTest
","text":"Columns that match the specified regexp pattern must have a specified test.
Parameters:
Name Type Description Default column_name_pattern
str
Regex pattern to match the column name.
required test_name
str
Name of the test to check for.
required Receives at execution time:
Name Type Description catalog_node
CatalogTable
The CatalogTable object to check.
tests
List[DbtBouncerTestBase]
List of DbtBouncerTestBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
catalog_checks:\n - name: check_column_has_specified_test\n column_name_pattern: ^is_.*\n test_name: not_null\n
Source code in src/dbt_bouncer/checks/catalog/check_columns.py
class CheckColumnHasSpecifiedTest(BaseCheck):\n \"\"\"Columns that match the specified regexp pattern must have a specified test.\n\n Parameters:\n column_name_pattern (str): Regex pattern to match the column name.\n test_name (str): Name of the test to check for.\n\n Receives:\n catalog_node (CatalogTable): The CatalogTable object to check.\n tests (List[DbtBouncerTestBase]): List of DbtBouncerTestBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n catalog_checks:\n - name: check_column_has_specified_test\n column_name_pattern: ^is_.*\n test_name: not_null\n ```\n\n \"\"\"\n\n catalog_node: \"CatalogTable\" = Field(default=None)\n column_name_pattern: str\n name: Literal[\"check_column_has_specified_test\"]\n test_name: str\n tests: List[\"DbtBouncerTestBase\"] = Field(default=[])\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n columns_to_check = [\n v.name\n for _, v in self.catalog_node.columns.items()\n if re.compile(self.column_name_pattern.strip()).match(v.name) is not None\n ]\n relevant_tests = [\n t\n for t in self.tests\n if hasattr(t, \"test_metadata\") is True\n and hasattr(t, \"attached_node\") is True\n and t.test_metadata.name == self.test_name\n and t.attached_node == self.catalog_node.unique_id\n ]\n non_complying_columns = [\n c\n for c in columns_to_check\n if f\"{self.catalog_node.unique_id}.{c}\"\n not in [f\"{t.attached_node}.{t.column_name}\" for t in relevant_tests]\n ]\n\n assert not non_complying_columns, f\"`{self.catalog_node.unique_id.split('.')[-1]}` has columns that should have a `{self.test_name}` test: {non_complying_columns}\"\n
"},{"location":"checks/manifest/check_exposures/","title":"Manifest Checks: Exposures","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_exposures/#manifest.check_exposures.CheckExposureOnNonPublicModels","title":"CheckExposureOnNonPublicModels
","text":"Exposures should be based on public models only.
Receives at execution time:
Name Type Description exposure
DbtBouncerExposureBase
The DbtBouncerExposureBase object to check.
models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Exposure paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Only exposure paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_exposure_based_on_non_public_models\n
Source code in src/dbt_bouncer/checks/manifest/check_exposures.py
class CheckExposureOnNonPublicModels(BaseCheck):\n \"\"\"Exposures should be based on public models only.\n\n Receives:\n exposure (DbtBouncerExposureBase): The DbtBouncerExposureBase object to check.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Exposure paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Only exposure paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_exposure_based_on_non_public_models\n ```\n\n \"\"\"\n\n exposure: \"DbtBouncerExposureBase\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_exposure_based_on_non_public_models\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n non_public_upstream_dependencies = []\n for model in self.exposure.depends_on.nodes:\n if (\n model.split(\".\")[0] == \"model\"\n and model.split(\".\")[1] == self.exposure.package_name\n ):\n model = next(m for m in self.models if m.unique_id == model)\n if model.access.value != \"public\":\n non_public_upstream_dependencies.append(model.name)\n\n assert not non_public_upstream_dependencies, f\"`{self.exposure.name}` is based on a model(s) that is not public: {non_public_upstream_dependencies}.\"\n
"},{"location":"checks/manifest/check_exposures/#manifest.check_exposures.CheckExposureOnView","title":"CheckExposureOnView
","text":"Exposures should not be based on views.
Parameters:
Name Type Description Default materializations_to_include
Optional[List[str]]
List of materializations to include in the check.
required Receives at execution time:
Name Type Description exposure
DbtBouncerExposureBase
The DbtBouncerExposureBase object to check.
models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Exposure paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Only exposure paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_exposure_based_on_view\n
manifest_checks:\n - name: check_exposure_based_on_view\n materializations_to_include:\n - ephemeral\n - my_custom_materialization\n - view\n
Source code in src/dbt_bouncer/checks/manifest/check_exposures.py
class CheckExposureOnView(BaseCheck):\n \"\"\"Exposures should not be based on views.\n\n Parameters:\n materializations_to_include (Optional[List[str]]): List of materializations to include in the check.\n\n Receives:\n exposure (DbtBouncerExposureBase): The DbtBouncerExposureBase object to check.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Exposure paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Only exposure paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_exposure_based_on_view\n ```\n ```yaml\n manifest_checks:\n - name: check_exposure_based_on_view\n materializations_to_include:\n - ephemeral\n - my_custom_materialization\n - view\n ```\n\n \"\"\"\n\n exposure: \"DbtBouncerExposureBase\" = Field(default=None)\n materializations_to_include: List[str] = Field(\n default=[\"ephemeral\", \"view\"],\n )\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_exposure_based_on_view\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n non_table_upstream_dependencies = []\n for model in self.exposure.depends_on.nodes:\n if (\n model.split(\".\")[0] == \"model\"\n and model.split(\".\")[1] == self.exposure.package_name\n ):\n model = next(m for m in self.models if m.unique_id == model)\n if model.config.materialized in self.materializations_to_include:\n non_table_upstream_dependencies.append(model.name)\n\n assert not non_table_upstream_dependencies, f\"`{self.exposure.name}` is based on a model that is not a table: {non_table_upstream_dependencies}.\"\n
"},{"location":"checks/manifest/check_lineage/","title":"Manifest Checks: Lineage","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_lineage/#manifest.check_lineage.CheckLineagePermittedUpstreamModels","title":"CheckLineagePermittedUpstreamModels
","text":"Upstream models must have a path that matches the provided upstream_path_pattern
.
Parameters:
Name Type Description Default upstream_path_pattern
str
Regexp pattern to match the upstream model(s) path.
required Receives at execution time:
Name Type Description manifest_obj
DbtBouncerManifest
The manifest object.
model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_lineage_permitted_upstream_models\n include: ^models/staging\n upstream_path_pattern: $^\n - name: check_lineage_permitted_upstream_models\n include: ^models/intermediate\n upstream_path_pattern: ^models/staging|^models/intermediate\n - name: check_lineage_permitted_upstream_models\n include: ^models/marts\n upstream_path_pattern: ^models/staging|^models/intermediate\n
Source code in src/dbt_bouncer/checks/manifest/check_lineage.py
class CheckLineagePermittedUpstreamModels(BaseCheck):\n \"\"\"Upstream models must have a path that matches the provided `upstream_path_pattern`.\n\n Parameters:\n upstream_path_pattern (str): Regexp pattern to match the upstream model(s) path.\n\n Receives:\n manifest_obj (DbtBouncerManifest): The manifest object.\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_lineage_permitted_upstream_models\n include: ^models/staging\n upstream_path_pattern: $^\n - name: check_lineage_permitted_upstream_models\n include: ^models/intermediate\n upstream_path_pattern: ^models/staging|^models/intermediate\n - name: check_lineage_permitted_upstream_models\n include: ^models/marts\n upstream_path_pattern: ^models/staging|^models/intermediate\n ```\n\n \"\"\"\n\n manifest_obj: \"DbtBouncerManifest\" = Field(default=None)\n model: \"DbtBouncerModelBase\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_lineage_permitted_upstream_models\"]\n upstream_path_pattern: str\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n upstream_models = [\n x\n for x in self.model.depends_on.nodes\n if x.split(\".\")[0] == \"model\"\n and x.split(\".\")[1] == self.manifest_obj.manifest.metadata.project_name\n ]\n not_permitted_upstream_models = [\n upstream_model\n for upstream_model in upstream_models\n if re.compile(self.upstream_path_pattern.strip()).match(\n clean_path_str(\n next(\n m for m in self.models if m.unique_id == upstream_model\n ).original_file_path\n ),\n )\n is None\n ]\n assert not not_permitted_upstream_models, f\"`{self.model.name}` references upstream models that are not permitted: {[m.split('.')[-1] for m in not_permitted_upstream_models]}.\"\n
"},{"location":"checks/manifest/check_lineage/#manifest.check_lineage.CheckLineageSeedCannotBeUsed","title":"CheckLineageSeedCannotBeUsed
","text":"Seed cannot be referenced in models with a path that matches the specified include
config.
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_lineage_seed_cannot_be_used\n include: ^models/intermediate|^models/marts\n
Source code in src/dbt_bouncer/checks/manifest/check_lineage.py
class CheckLineageSeedCannotBeUsed(BaseCheck):\n \"\"\"Seed cannot be referenced in models with a path that matches the specified `include` config.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_lineage_seed_cannot_be_used\n include: ^models/intermediate|^models/marts\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_lineage_seed_cannot_be_used\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert not [\n x for x in self.model.depends_on.nodes if x.split(\".\")[0] == \"seed\"\n ], f\"`{self.model.name}` references a seed even though this is not permitted.\"\n
"},{"location":"checks/manifest/check_lineage/#manifest.check_lineage.CheckLineageSourceCannotBeUsed","title":"CheckLineageSourceCannotBeUsed
","text":"Sources cannot be referenced in models with a path that matches the specified include
config.
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_lineage_source_cannot_be_used\n include: ^models/intermediate|^models/marts\n
Source code in src/dbt_bouncer/checks/manifest/check_lineage.py
class CheckLineageSourceCannotBeUsed(BaseCheck):\n \"\"\"Sources cannot be referenced in models with a path that matches the specified `include` config.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_lineage_source_cannot_be_used\n include: ^models/intermediate|^models/marts\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_lineage_source_cannot_be_used\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert not [\n x for x in self.model.depends_on.nodes if x.split(\".\")[0] == \"source\"\n ], f\"`{self.model.name}` references a source even though this is not permitted.\"\n
"},{"location":"checks/manifest/check_macros/","title":"Manifest Checks: Macros","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_macros/#manifest.check_macros.CheckMacroArgumentsDescriptionPopulated","title":"CheckMacroArgumentsDescriptionPopulated
","text":"Macro arguments must have a populated description.
Receives at execution time:
Name Type Description macro
Macros
The Macros object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_macro_arguments_description_populated\n
# Only \"common\" macros need to have their arguments populated\nmanifest_checks:\n - name: check_macro_arguments_description_populated\n include: ^macros/common\n
Source code in src/dbt_bouncer/checks/manifest/check_macros.py
class CheckMacroArgumentsDescriptionPopulated(BaseCheck):\n \"\"\"Macro arguments must have a populated description.\n\n Receives:\n macro (Macros): The Macros object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_macro_arguments_description_populated\n ```\n ```yaml\n # Only \"common\" macros need to have their arguments populated\n manifest_checks:\n - name: check_macro_arguments_description_populated\n include: ^macros/common\n ```\n\n \"\"\"\n\n macro: \"Macros\" = Field(default=None)\n name: Literal[\"check_macro_arguments_description_populated\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n environment = jinja2.Environment(autoescape=True, extensions=[TagExtension])\n ast = environment.parse(self.macro.macro_sql)\n\n if hasattr(ast.body[0], \"args\"):\n # Assume macro is a \"true\" macro\n macro_arguments = [a.name for a in ast.body[0].args]\n else:\n if \"materialization\" in [\n x.value.value\n for x in ast.body[0].nodes[0].kwargs # type: ignore[attr-defined]\n if isinstance(x.value, jinja2.nodes.Const)\n ]:\n # Materializations don't have arguments\n macro_arguments = []\n else:\n # Macro is a test\n test_macro = next(\n x\n for x in ast.body\n if not isinstance(x.nodes[0], jinja2.nodes.Call) # type: ignore[attr-defined]\n )\n macro_arguments = [\n x.name\n for x in test_macro.nodes # type: ignore[attr-defined]\n if isinstance(x, jinja2.nodes.Name)\n ]\n\n # macro_arguments: List of args parsed from macro SQL\n # macro.arguments: List of args manually added to the properties file\n\n non_complying_args = []\n for arg in macro_arguments:\n macro_doc_raw = [x for x in self.macro.arguments if x.name == arg]\n if macro_doc_raw == [] or (\n arg not in [x.name for x in self.macro.arguments]\n or len(macro_doc_raw[0].description.strip()) <= 4\n ):\n non_complying_args.append(arg)\n\n assert (\n non_complying_args == []\n ), f\"Macro `{self.macro.name}` does not have a populated description for the following argument(s): {non_complying_args}.\"\n
"},{"location":"checks/manifest/check_macros/#manifest.check_macros.CheckMacroCodeDoesNotContainRegexpPattern","title":"CheckMacroCodeDoesNotContainRegexpPattern
","text":"The raw code for a macro must not match the specified regexp pattern.
Parameters:
Name Type Description Default regexp_pattern
str
The regexp pattern that should not be matched by the macro code.
required Receives at execution time:
Name Type Description macro
Macros
The Macros object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n # Prefer `coalesce` over `ifnull`: https://docs.sqlfluff.com/en/stable/rules.html#sqlfluff.rules.sphinx.Rule_CV02\n - name: check_macro_code_does_not_contain_regexp_pattern\n regexp_pattern: .*[i][f][n][u][l][l].*\n
Source code in src/dbt_bouncer/checks/manifest/check_macros.py
class CheckMacroCodeDoesNotContainRegexpPattern(BaseCheck):\n \"\"\"The raw code for a macro must not match the specified regexp pattern.\n\n Parameters:\n regexp_pattern (str): The regexp pattern that should not be matched by the macro code.\n\n Receives:\n macro (Macros): The Macros object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n # Prefer `coalesce` over `ifnull`: https://docs.sqlfluff.com/en/stable/rules.html#sqlfluff.rules.sphinx.Rule_CV02\n - name: check_macro_code_does_not_contain_regexp_pattern\n regexp_pattern: .*[i][f][n][u][l][l].*\n ```\n\n \"\"\"\n\n macro: \"Macros\" = Field(default=None)\n name: Literal[\"check_macro_code_does_not_contain_regexp_pattern\"]\n regexp_pattern: str\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n re.compile(self.regexp_pattern.strip(), flags=re.DOTALL).match(\n self.macro.macro_sql\n )\n is None\n ), f\"Macro `{self.macro.name}` contains a banned string: `{self.regexp_pattern.strip()}`.\"\n
"},{"location":"checks/manifest/check_macros/#manifest.check_macros.CheckMacroDescriptionPopulated","title":"CheckMacroDescriptionPopulated
","text":"Macros must have a populated description.
Receives at execution time:
Name Type Description macro
Macros
The Macros object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_macro_description_populated\n
# Only \"common\" macros need to have a populated description\nmanifest_checks:\n - name: check_macro_description_populated\n include: ^macros/common\n
Source code in src/dbt_bouncer/checks/manifest/check_macros.py
class CheckMacroDescriptionPopulated(BaseCheck):\n \"\"\"Macros must have a populated description.\n\n Receives:\n macro (Macros): The Macros object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_macro_description_populated\n ```\n ```yaml\n # Only \"common\" macros need to have a populated description\n manifest_checks:\n - name: check_macro_description_populated\n include: ^macros/common\n ```\n\n \"\"\"\n\n macro: \"Macros\" = Field(default=None)\n name: Literal[\"check_macro_description_populated\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n len(self.macro.description.strip()) > 4\n ), f\"Macro `{self.macro.name}` does not have a populated description.\"\n
"},{"location":"checks/manifest/check_macros/#manifest.check_macros.CheckMacroMaxNumberOfLines","title":"CheckMacroMaxNumberOfLines
","text":"Macros may not have more than the specified number of lines.
Parameters:
Name Type Description Default max_number_of_lines
int
The maximum number of permitted lines.
required Receives at execution time:
Name Type Description macro
Macros
The Macros object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_macro_max_number_of_lines\n
manifest_checks:\n - name: check_macro_max_number_of_lines\n max_number_of_lines: 100\n
Source code in src/dbt_bouncer/checks/manifest/check_macros.py
class CheckMacroMaxNumberOfLines(BaseCheck):\n \"\"\"Macros may not have more than the specified number of lines.\n\n Parameters:\n max_number_of_lines (int): The maximum number of permitted lines.\n\n Receives:\n macro (Macros): The Macros object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_macro_max_number_of_lines\n ```\n ```yaml\n manifest_checks:\n - name: check_macro_max_number_of_lines\n max_number_of_lines: 100\n ```\n\n \"\"\"\n\n macro: \"Macros\" = Field(default=None)\n name: Literal[\"check_macro_max_number_of_lines\"]\n max_number_of_lines: int = Field(default=50)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n actual_number_of_lines = self.macro.macro_sql.count(\"\\n\") + 1\n\n assert (\n actual_number_of_lines <= self.max_number_of_lines\n ), f\"Macro `{self.macro.name}` has {actual_number_of_lines} lines, this is more than the maximum permitted number of lines ({self.max_number_of_lines}).\"\n
"},{"location":"checks/manifest/check_macros/#manifest.check_macros.CheckMacroNameMatchesFileName","title":"CheckMacroNameMatchesFileName
","text":"Macros names must be the same as the file they are contained in.
Generic tests are also macros, however to document these tests the \"name\" value must be preceded with \"test_\".
Receives at execution time:
Name Type Description macro
Macros
The Macros object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_macro_name_matches_file_name\n
Source code in src/dbt_bouncer/checks/manifest/check_macros.py
class CheckMacroNameMatchesFileName(BaseCheck):\n \"\"\"Macros names must be the same as the file they are contained in.\n\n Generic tests are also macros, however to document these tests the \"name\" value must be preceded with \"test_\".\n\n Receives:\n macro (Macros): The Macros object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_macro_name_matches_file_name\n ```\n\n \"\"\"\n\n macro: \"Macros\" = Field(default=None)\n name: Literal[\"check_macro_name_matches_file_name\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if self.macro.name.startswith(\"test_\"):\n assert (\n self.macro.name[5:]\n == clean_path_str(self.macro.original_file_path)\n .split(\"/\")[-1]\n .split(\".\")[0]\n ), f\"Macro `{self.macro.unique_id}` is not in a file named `{self.macro.name[5:]}.sql`.\"\n else:\n assert (\n self.macro.name\n == clean_path_str(self.macro.original_file_path)\n .split(\"/\")[-1]\n .split(\".\")[0]\n ), f\"Macro `{self.macro.name}` is not in a file of the same name.\"\n
"},{"location":"checks/manifest/check_macros/#manifest.check_macros.CheckMacroPropertyFileLocation","title":"CheckMacroPropertyFileLocation
","text":"Macro properties files must follow the guidance provided by dbt here.
Receives at execution time:
Name Type Description macro
Macros
The Macros object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_macro_property_file_location\n
Source code in src/dbt_bouncer/checks/manifest/check_macros.py
class CheckMacroPropertyFileLocation(BaseCheck):\n \"\"\"Macro properties files must follow the guidance provided by dbt [here](https://docs.getdbt.com/best-practices/how-we-structure/5-the-rest-of-the-project#how-we-use-the-other-folders).\n\n Receives:\n macro (Macros): The Macros object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_macro_property_file_location\n ```\n\n \"\"\"\n\n macro: \"Macros\" = Field(default=None)\n name: Literal[\"check_macro_property_file_location\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n expected_substr = \"_\".join(\n clean_path_str(self.macro.original_file_path)[6:].split(\"/\")[:-1]\n )\n\n assert (\n clean_path_str(self.macro.patch_path) is not None\n ), f\"Macro `{self.macro.name}` is not defined in a `.yml` properties file.\"\n properties_yml_name = clean_path_str(self.macro.patch_path).split(\"/\")[-1]\n\n if clean_path_str(self.macro.original_file_path).startswith(\n \"tests/\",\n ): # Do not check generic tests (which are also macros)\n pass\n elif expected_substr == \"\": # i.e. macro in ./macros\n assert (\n properties_yml_name == \"_macros.yml\"\n ), f\"The properties file for `{self.macro.name}` (`{properties_yml_name}`) should be `_macros.yml`.\"\n else:\n assert properties_yml_name.startswith(\n \"_\",\n ), f\"The properties file for `{self.macro.name}` (`{properties_yml_name}`) does not start with an underscore.\"\n assert (\n expected_substr in properties_yml_name\n ), f\"The properties file for `{self.macro.name}` (`{properties_yml_name}`) does not contain the expected substring (`{expected_substr}`).\"\n assert properties_yml_name.endswith(\n \"__macros.yml\",\n ), f\"The properties file for `{self.macro.name.name}` (`{properties_yml_name}`) does not end with `__macros.yml`.\"\n
"},{"location":"checks/manifest/check_metadata/","title":"Manifest Checks: Metadata","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_metadata/#manifest.check_metadata.CheckProjectName","title":"CheckProjectName
","text":"Enforce that the name of the dbt project matches a supplied regex. Generally used to enforce that project names conform to something like company_<DOMAIN>
.
Parameters:
Name Type Description Default project_name_pattern
str
Regex pattern to match the project name.
required Receives at execution time:
Name Type Description manifest_obj
DbtBouncerManifest
The manifest object.
Other Parameters (passed via config file):
Name Type Description severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_project_name\n project_name_pattern: ^awesome_company_\n
Source code in src/dbt_bouncer/checks/manifest/check_metadata.py
class CheckProjectName(BaseModel):\n \"\"\"Enforce that the name of the dbt project matches a supplied regex. Generally used to enforce that project names conform to something like `company_<DOMAIN>`.\n\n Parameters:\n project_name_pattern (str): Regex pattern to match the project name.\n\n Receives:\n manifest_obj (DbtBouncerManifest): The manifest object.\n\n Other Parameters:\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_project_name\n project_name_pattern: ^awesome_company_\n ```\n\n \"\"\"\n\n model_config = ConfigDict(extra=\"forbid\")\n\n index: Optional[int] = Field(\n default=None,\n description=\"Index to uniquely identify the check, calculated at runtime.\",\n )\n manifest_obj: \"DbtBouncerManifest\" = Field(default=None)\n name: Literal[\"check_project_name\"]\n project_name_pattern: str\n severity: Optional[Literal[\"error\", \"warn\"]] = Field(\n default=\"error\",\n description=\"Severity of the check, one of 'error' or 'warn'.\",\n )\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n re.compile(self.project_name_pattern.strip()).match(\n self.manifest_obj.manifest.metadata.project_name,\n )\n is not None\n ), f\"Project name (`{self.manifest_obj.manifest.metadata.project_name}`) does not conform to the supplied regex `({self.project_name_pattern.strip()})`.\"\n
"},{"location":"checks/manifest/check_models/","title":"Manifest Checks: Models","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelAccess","title":"CheckModelAccess
","text":"Models must have the specified access attribute. Requires dbt 1.7+.
Parameters:
Name Type Description Default access
Literal['private', 'protected', 'public']
The access level to check for.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n # Align with dbt best practices that marts should be `public`, everything else should be `protected`\n - name: check_model_access\n access: protected\n include: ^models/intermediate\n - name: check_model_access\n access: public\n include: ^models/marts\n - name: check_model_access\n access: protected\n include: ^models/staging\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelAccess(BaseCheck):\n \"\"\"Models must have the specified access attribute. Requires dbt 1.7+.\n\n Parameters:\n access (Literal[\"private\", \"protected\", \"public\"]): The access level to check for.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n # Align with dbt best practices that marts should be `public`, everything else should be `protected`\n - name: check_model_access\n access: protected\n include: ^models/intermediate\n - name: check_model_access\n access: public\n include: ^models/marts\n - name: check_model_access\n access: protected\n include: ^models/staging\n ```\n\n \"\"\"\n\n access: Literal[\"private\", \"protected\", \"public\"]\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_access\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n self.model.access.value == self.access\n ), f\"`{self.model.name}` has `{self.model.access.value}` access, it should have access `{self.access}`.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelContractsEnforcedForPublicModel","title":"CheckModelContractsEnforcedForPublicModel
","text":"Public models must have contracts enforced.
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_contract_enforced_for_public_model\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelContractsEnforcedForPublicModel(BaseCheck):\n \"\"\"Public models must have contracts enforced.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_contract_enforced_for_public_model\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_contract_enforced_for_public_model\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if self.model.access.value == \"public\":\n assert (\n self.model.contract.enforced is True\n ), f\"`{self.model.name}` is a public model but does not have contracts enforced.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelDescriptionPopulated","title":"CheckModelDescriptionPopulated
","text":"Models must have a populated description.
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_description_populated\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelDescriptionPopulated(BaseCheck):\n \"\"\"Models must have a populated description.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_description_populated\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_description_populated\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n len(self.model.description.strip()) > 4\n ), f\"`{self.model.name}` does not have a populated description.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelsDocumentationCoverage","title":"CheckModelsDocumentationCoverage
","text":"Set the minimum percentage of models that have a populated description.
Parameters:
Name Type Description Default min_model_documentation_coverage_pct
float
The minimum percentage of models that must have a populated description.
required Receives at execution time:
Name Type Description models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_description_populated\n min_model_documentation_coverage_pct: 90\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelsDocumentationCoverage(BaseModel):\n \"\"\"Set the minimum percentage of models that have a populated description.\n\n Parameters:\n min_model_documentation_coverage_pct (float): The minimum percentage of models that must have a populated description.\n\n Receives:\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_description_populated\n min_model_documentation_coverage_pct: 90\n ```\n\n \"\"\"\n\n model_config = ConfigDict(extra=\"forbid\")\n\n index: Optional[int] = Field(\n default=None,\n description=\"Index to uniquely identify the check, calculated at runtime.\",\n )\n min_model_documentation_coverage_pct: int = Field(\n default=100,\n ge=0,\n le=100,\n )\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_model_documentation_coverage\"]\n severity: Optional[Literal[\"error\", \"warn\"]] = Field(\n default=\"error\",\n description=\"Severity of the check, one of 'error' or 'warn'.\",\n )\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_models = len(self.models)\n models_with_description = []\n for model in self.models:\n if len(model.description.strip()) > 4:\n models_with_description.append(model.unique_id)\n\n num_models_with_descriptions = len(models_with_description)\n model_description_coverage_pct = (\n num_models_with_descriptions / num_models\n ) * 100\n\n assert (\n model_description_coverage_pct >= self.min_model_documentation_coverage_pct\n ), f\"Only {model_description_coverage_pct}% of models have a populated description, this is less than the permitted minimum of {self.min_model_documentation_coverage_pct}%.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelDocumentedInSameDirectory","title":"CheckModelDocumentedInSameDirectory
","text":"Models must be documented in the same directory where they are defined (i.e. .yml
and .sql
files are in the same directory).
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_documented_in_same_directory\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelDocumentedInSameDirectory(BaseCheck):\n \"\"\"Models must be documented in the same directory where they are defined (i.e. `.yml` and `.sql` files are in the same directory).\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_documented_in_same_directory\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_documented_in_same_directory\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n model_sql_dir = clean_path_str(self.model.original_file_path).split(\"/\")[:-1]\n assert ( # noqa: PT018\n hasattr(self.model, \"patch_path\")\n and clean_path_str(self.model.patch_path) is not None\n ), f\"`{self.model.name}` is not documented.\"\n\n model_doc_dir = clean_path_str(\n self.model.patch_path[\n clean_path_str(self.model.patch_path).find(\"models\") :\n ]\n ).split(\"/\")[:-1]\n\n assert (\n model_doc_dir == model_sql_dir\n ), f\"`{self.model.name}` is documented in a different directory to the `.sql` file: `{'/'.join(model_doc_dir)}` vs `{'/'.join(model_sql_dir)}`.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelCodeDoesNotContainRegexpPattern","title":"CheckModelCodeDoesNotContainRegexpPattern
","text":"The raw code for a model must not match the specified regexp pattern.
Parameters:
Name Type Description Default regexp_pattern
str
The regexp pattern that should not be matched by the model code.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n # Prefer `coalesce` over `ifnull`: https://docs.sqlfluff.com/en/stable/rules.html#sqlfluff.rules.sphinx.Rule_CV02\n - name: check_model_code_does_not_contain_regexp_pattern\n regexp_pattern: .*[i][f][n][u][l][l].*\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelCodeDoesNotContainRegexpPattern(BaseCheck):\n \"\"\"The raw code for a model must not match the specified regexp pattern.\n\n Parameters:\n regexp_pattern (str): The regexp pattern that should not be matched by the model code.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n # Prefer `coalesce` over `ifnull`: https://docs.sqlfluff.com/en/stable/rules.html#sqlfluff.rules.sphinx.Rule_CV02\n - name: check_model_code_does_not_contain_regexp_pattern\n regexp_pattern: .*[i][f][n][u][l][l].*\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_code_does_not_contain_regexp_pattern\"]\n regexp_pattern: str\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n re.compile(self.regexp_pattern.strip(), flags=re.DOTALL).match(\n self.model.raw_code\n )\n is None\n ), f\"`{self.model.name}` contains a banned string: `{self.regexp_pattern.strip()}`.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelDependsOnMultipleSources","title":"CheckModelDependsOnMultipleSources
","text":"Models cannot reference more than one source.
Parameters:
Name Type Description Default model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_depends_on_multiple_sources\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelDependsOnMultipleSources(BaseCheck):\n \"\"\"Models cannot reference more than one source.\n\n Parameters:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_depends_on_multiple_sources\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_depends_on_multiple_sources\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_reffed_sources = sum(\n x.split(\".\")[0] == \"source\" for x in self.model.depends_on.nodes\n )\n assert (\n num_reffed_sources <= 1\n ), f\"`{self.model.name}` references more than one source.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelDirectories","title":"CheckModelDirectories
","text":"Only specified sub-directories are permitted.
Parameters:
Name Type Description Default include
str
Regex pattern to the directory to check.
required permitted_sub_directories
List[str]
List of permitted sub-directories.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Example(s):
manifest_checks:\n- name: check_model_directories\n include: models\n permitted_sub_directories:\n - intermediate\n - marts\n - staging\n
# Restrict sub-directories within `./models/staging`\n- name: check_model_directories\n include: ^models/staging\n permitted_sub_directories:\n - crm\n - payments\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelDirectories(BaseCheck):\n \"\"\"Only specified sub-directories are permitted.\n\n Parameters:\n include (str): Regex pattern to the directory to check.\n permitted_sub_directories (List[str]): List of permitted sub-directories.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_directories\n include: models\n permitted_sub_directories:\n - intermediate\n - marts\n - staging\n ```\n ```yaml\n # Restrict sub-directories within `./models/staging`\n - name: check_model_directories\n include: ^models/staging\n permitted_sub_directories:\n - crm\n - payments\n ```\n\n \"\"\"\n\n include: str\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_directories\"]\n permitted_sub_directories: List[str]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n matched_path = re.compile(self.include.strip()).match(\n clean_path_str(self.model.original_file_path)\n )\n path_after_match = clean_path_str(self.model.original_file_path)[\n matched_path.end() + 1 :\n ]\n\n assert (\n path_after_match.split(\"/\")[0] in self.permitted_sub_directories\n ), f\"`{self.model.name}` is located in `{self.model.original_file_path.split('/')[1]}`, this is not a valid sub-directory.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelHasContractsEnforced","title":"CheckModelHasContractsEnforced
","text":"Model must have contracts enforced.
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_has_contracts_enforced\n include: ^models/marts\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelHasContractsEnforced(BaseCheck):\n \"\"\"Model must have contracts enforced.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_has_contracts_enforced\n include: ^models/marts\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_has_contracts_enforced\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n self.model.contract.enforced is True\n ), f\"`{self.model.name}` does not have contracts enforced.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelHasMetaKeys","title":"CheckModelHasMetaKeys
","text":"The meta
config for models must have the specified keys.
Parameters:
Name Type Description Default keys
NestedDict
A list (that may contain sub-lists) of required keys.
required model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_has_meta_keys\n keys:\n - maturity\n - owner\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelHasMetaKeys(BaseCheck):\n \"\"\"The `meta` config for models must have the specified keys.\n\n Parameters:\n keys (NestedDict): A list (that may contain sub-lists) of required keys.\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_has_meta_keys\n keys:\n - maturity\n - owner\n ```\n\n \"\"\"\n\n keys: NestedDict\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_has_meta_keys\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n missing_keys = find_missing_meta_keys(\n meta_config=self.model.meta,\n required_keys=self.keys.model_dump(),\n )\n assert (\n missing_keys == []\n ), f\"`{self.model.name}` is missing the following keys from the `meta` config: {[x.replace('>>', '') for x in missing_keys]}\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelHasNoUpstreamDependencies","title":"CheckModelHasNoUpstreamDependencies
","text":"Identify if models have no upstream dependencies as this likely indicates hard-coded tables references.
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_has_no_upstream_dependencies\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelHasNoUpstreamDependencies(BaseCheck):\n \"\"\"Identify if models have no upstream dependencies as this likely indicates hard-coded tables references.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_has_no_upstream_dependencies\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_has_no_upstream_dependencies\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n len(self.model.depends_on.nodes) > 0\n ), f\"`{self.model.name}` has no upstream dependencies, this likely indicates hard-coded tables references.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelHasTags","title":"CheckModelHasTags
","text":"Models must have the specified tags.
Parameters:
Name Type Description Default model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
required tags
List[str]
List of tags to check for.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_has_tags\n tags:\n - tag_1\n - tag_2\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelHasTags(BaseCheck):\n \"\"\"Models must have the specified tags.\n\n Parameters:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n tags (List[str]): List of tags to check for.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_has_tags\n tags:\n - tag_1\n - tag_2\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_has_tags\"]\n tags: List[str]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n missing_tags = [tag for tag in self.tags if tag not in self.model.tags]\n assert (\n not missing_tags\n ), f\"`{self.model.name}` is missing required tags: {missing_tags}.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelHasUniqueTest","title":"CheckModelHasUniqueTest
","text":"Models must have a test for uniqueness of a column.
Parameters:
Name Type Description Default accepted_uniqueness_tests
Optional[List[str]]
List of tests that are accepted as uniqueness tests.
required model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
required tests
List[DbtBouncerTestBase]
List of DbtBouncerTestBase objects parsed from manifest.json
.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_has_unique_test\n include: ^models/marts\n
manifest_checks:\n# Example of allowing a custom uniqueness test\n - name: check_model_has_unique_test\n accepted_uniqueness_tests:\n - expect_compound_columns_to_be_unique\n - my_custom_uniqueness_test\n - unique\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelHasUniqueTest(BaseCheck):\n \"\"\"Models must have a test for uniqueness of a column.\n\n Parameters:\n accepted_uniqueness_tests (Optional[List[str]]): List of tests that are accepted as uniqueness tests.\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n tests (List[DbtBouncerTestBase]): List of DbtBouncerTestBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_has_unique_test\n include: ^models/marts\n ```\n ```yaml\n manifest_checks:\n # Example of allowing a custom uniqueness test\n - name: check_model_has_unique_test\n accepted_uniqueness_tests:\n - expect_compound_columns_to_be_unique\n - my_custom_uniqueness_test\n - unique\n ```\n\n \"\"\"\n\n accepted_uniqueness_tests: Optional[List[str]] = Field(\n default=[\n \"expect_compound_columns_to_be_unique\",\n \"dbt_utils.unique_combination_of_columns\",\n \"unique\",\n ],\n )\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_has_unique_test\"]\n tests: List[\"DbtBouncerTestBase\"] = Field(default=[])\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_unique_tests = sum(\n test.attached_node == self.model.unique_id\n and test.test_metadata.name in self.accepted_uniqueness_tests # type: ignore[operator]\n for test in self.tests\n if hasattr(test, \"test_metadata\")\n )\n assert (\n num_unique_tests >= 1\n ), f\"`{self.model.name}` does not have a test for uniqueness of a column.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelHasUnitTests","title":"CheckModelHasUnitTests
","text":"Models must have more than the specified number of unit tests.
Parameters:
Name Type Description Default min_number_of_unit_tests
Optional[int]
The minimum number of unit tests that a model must have.
required Receives at execution time:
Name Type Description manifest_obj
DbtBouncerManifest
The DbtBouncerManifest object parsed from manifest.json
.
model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
unit_tests
List[UnitTests]
List of UnitTests objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Warning
This check is only supported for dbt 1.8.0 and above.
Example(s):
manifest_checks:\n - name: check_model_has_unit_tests\n include: ^models/marts\n
manifest_checks:\n - name: check_model_has_unit_tests\n min_number_of_unit_tests: 2\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelHasUnitTests(BaseCheck):\n \"\"\"Models must have more than the specified number of unit tests.\n\n Parameters:\n min_number_of_unit_tests (Optional[int]): The minimum number of unit tests that a model must have.\n\n Receives:\n manifest_obj (DbtBouncerManifest): The DbtBouncerManifest object parsed from `manifest.json`.\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n unit_tests (List[UnitTests]): List of UnitTests objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n !!! warning\n\n This check is only supported for dbt 1.8.0 and above.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_has_unit_tests\n include: ^models/marts\n ```\n ```yaml\n manifest_checks:\n - name: check_model_has_unit_tests\n min_number_of_unit_tests: 2\n ```\n\n \"\"\"\n\n manifest_obj: \"DbtBouncerManifest\" = Field(default=None)\n min_number_of_unit_tests: int = Field(default=1)\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_has_unit_tests\"]\n unit_tests: List[\"UnitTests\"] = Field(default=[])\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if (\n semver.Version.parse(self.manifest_obj.manifest.metadata.dbt_version)\n >= \"1.8.0\"\n ):\n num_unit_tests = len(\n [\n t.unique_id\n for t in self.unit_tests\n if t.depends_on.nodes[0] == self.model.unique_id\n ],\n )\n assert (\n num_unit_tests >= self.min_number_of_unit_tests\n ), f\"`{self.model.name}` has {num_unit_tests} unit tests, this is less than the minimum of {self.min_number_of_unit_tests}.\"\n else:\n logging.warning(\n \"The `check_model_has_unit_tests` check is only supported for dbt 1.8.0 and above.\",\n )\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelMaxChainedViews","title":"CheckModelMaxChainedViews
","text":"Models cannot have more than the specified number of upstream dependents that are not tables.
Parameters:
Name Type Description Default materializations_to_include
Optional[List[str]]
List of materializations to include in the check.
required max_chained_views
Optional[int]
The maximum number of upstream dependents that are not tables.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_max_chained_views\n
manifest_checks:\n - name: check_model_max_chained_views\n materializations_to_include:\n - ephemeral\n - my_custom_materialization\n - view\n max_chained_views: 5\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelMaxChainedViews(BaseCheck):\n \"\"\"Models cannot have more than the specified number of upstream dependents that are not tables.\n\n Parameters:\n materializations_to_include (Optional[List[str]]): List of materializations to include in the check.\n max_chained_views (Optional[int]): The maximum number of upstream dependents that are not tables.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_max_chained_views\n ```\n ```yaml\n manifest_checks:\n - name: check_model_max_chained_views\n materializations_to_include:\n - ephemeral\n - my_custom_materialization\n - view\n max_chained_views: 5\n ```\n\n \"\"\"\n\n manifest_obj: \"DbtBouncerManifest\" = Field(default=None)\n materializations_to_include: List[str] = Field(\n default=[\"ephemeral\", \"view\"],\n )\n max_chained_views: int = Field(\n default=3,\n )\n model: \"DbtBouncerModelBase\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_model_max_chained_views\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n\n def return_upstream_view_models(\n materializations,\n max_chained_views,\n models,\n model_unique_ids_to_check,\n package_name,\n depth=0,\n ):\n \"\"\"Recursive function to return model unique_id's of upstream models that are views. Depth of recursion can be specified. If no models meet the criteria then an empty list is returned.\n\n Returns\n -\n List[str]: List of model unique_id's of upstream models that are views.\n\n \"\"\"\n if depth == max_chained_views or model_unique_ids_to_check == []:\n return model_unique_ids_to_check\n\n relevant_upstream_models = []\n for model in model_unique_ids_to_check:\n upstream_nodes = list(\n next(m2 for m2 in models if m2.unique_id == model).depends_on.nodes,\n )\n if upstream_nodes != []:\n upstream_models = [\n m\n for m in upstream_nodes\n if m.split(\".\")[0] == \"model\"\n and m.split(\".\")[1] == package_name\n ]\n for i in upstream_models:\n if (\n next(\n m for m in models if m.unique_id == i\n ).config.materialized\n in materializations\n ):\n relevant_upstream_models.append(i)\n\n depth += 1\n return return_upstream_view_models(\n materializations=materializations,\n max_chained_views=max_chained_views,\n models=models,\n model_unique_ids_to_check=relevant_upstream_models,\n package_name=package_name,\n depth=depth,\n )\n\n assert (\n len(\n return_upstream_view_models(\n materializations=self.materializations_to_include,\n max_chained_views=self.max_chained_views,\n models=self.models,\n model_unique_ids_to_check=[self.model.unique_id],\n package_name=self.manifest_obj.manifest.metadata.project_name,\n ),\n )\n == 0\n ), f\"`{self.model.name}` has more than {self.max_chained_views} upstream dependents that are not tables.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelMaxFanout","title":"CheckModelMaxFanout
","text":"Models cannot have more than the specified number of downstream models.
Parameters:
Name Type Description Default max_downstream_models
Optional[int]
The maximum number of permitted downstream models.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_max_fanout\n max_downstream_models: 2\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelMaxFanout(BaseCheck):\n \"\"\"Models cannot have more than the specified number of downstream models.\n\n Parameters:\n max_downstream_models (Optional[int]): The maximum number of permitted downstream models.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_max_fanout\n max_downstream_models: 2\n ```\n\n \"\"\"\n\n max_downstream_models: int = Field(default=3)\n model: \"DbtBouncerModelBase\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_model_max_fanout\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_downstream_models = sum(\n self.model.unique_id in m.depends_on.nodes for m in self.models\n )\n\n assert (\n num_downstream_models <= self.max_downstream_models\n ), f\"`{self.model.name}` has {num_downstream_models} downstream models, which is more than the permitted maximum of {self.max_downstream_models}.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelMaxNumberOfLines","title":"CheckModelMaxNumberOfLines
","text":"Models may not have more than the specified number of lines.
Parameters:
Name Type Description Default max_number_of_lines
int
The maximum number of permitted lines.
required model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_max_number_of_lines\n
manifest_checks:\n - name: check_model_max_number_of_lines\n max_number_of_lines: 150\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelMaxNumberOfLines(BaseCheck):\n \"\"\"Models may not have more than the specified number of lines.\n\n Parameters:\n max_number_of_lines (int): The maximum number of permitted lines.\n\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_max_number_of_lines\n ```\n ```yaml\n manifest_checks:\n - name: check_model_max_number_of_lines\n max_number_of_lines: 150\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_max_number_of_lines\"]\n max_number_of_lines: int = Field(default=100)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n actual_number_of_lines = self.model.raw_code.count(\"\\n\") + 1\n\n assert (\n actual_number_of_lines <= self.max_number_of_lines\n ), f\"`{self.model.name}` has {actual_number_of_lines} lines, this is more than the maximum permitted number of lines ({self.max_number_of_lines}).\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelMaxUpstreamDependencies","title":"CheckModelMaxUpstreamDependencies
","text":"Limit the number of upstream dependencies a model has.
Parameters:
Name Type Description Default max_upstream_macros
Optional[int]
The maximum number of permitted upstream macros.
required max_upstream_models
Optional[int]
The maximum number of permitted upstream models.
required max_upstream_sources
Optional[int]
The maximum number of permitted upstream sources.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_max_upstream_dependencies\n max_upstream_models: 3\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelMaxUpstreamDependencies(BaseCheck):\n \"\"\"Limit the number of upstream dependencies a model has.\n\n Parameters:\n max_upstream_macros (Optional[int]): The maximum number of permitted upstream macros.\n max_upstream_models (Optional[int]): The maximum number of permitted upstream models.\n max_upstream_sources (Optional[int]): The maximum number of permitted upstream sources.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_max_upstream_dependencies\n max_upstream_models: 3\n ```\n\n \"\"\"\n\n max_upstream_macros: int = Field(\n default=5,\n )\n max_upstream_models: int = Field(\n default=5,\n )\n max_upstream_sources: int = Field(\n default=1,\n )\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_max_upstream_dependencies\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_upstream_macros = len(list(self.model.depends_on.macros))\n num_upstream_models = len(\n [m for m in self.model.depends_on.nodes if m.split(\".\")[0] == \"model\"],\n )\n num_upstream_sources = len(\n [m for m in self.model.depends_on.nodes if m.split(\".\")[0] == \"source\"],\n )\n\n assert (\n num_upstream_macros <= self.max_upstream_macros\n ), f\"`{self.model.name}` has {num_upstream_macros} upstream macros, which is more than the permitted maximum of {self.max_upstream_macros}.\"\n assert (\n num_upstream_models <= self.max_upstream_models\n ), f\"`{self.model.name}` has {num_upstream_models} upstream models, which is more than the permitted maximum of {self.max_upstream_models}.\"\n assert (\n num_upstream_sources <= self.max_upstream_sources\n ), f\"`{self.model.name}` has {num_upstream_sources} upstream sources, which is more than the permitted maximum of {self.max_upstream_sources}.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelNames","title":"CheckModelNames
","text":"Models must have a name that matches the supplied regex.
Parameters:
Name Type Description Default model_name_pattern
str
Regexp the model name must match.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_names\n include: ^models/intermediate\n model_name_pattern: ^int_\n - name: check_model_names\n include: ^models/staging\n model_name_pattern: ^stg_\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelNames(BaseCheck):\n \"\"\"Models must have a name that matches the supplied regex.\n\n Parameters:\n model_name_pattern (str): Regexp the model name must match.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_names\n include: ^models/intermediate\n model_name_pattern: ^int_\n - name: check_model_names\n include: ^models/staging\n model_name_pattern: ^stg_\n ```\n\n \"\"\"\n\n model_config = ConfigDict(extra=\"forbid\", protected_namespaces=())\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_names\"]\n model_name_pattern: str\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n re.compile(self.model_name_pattern.strip()).match(self.model.name)\n is not None\n ), f\"`{self.model.name}` does not match the supplied regex `{self.model_name_pattern.strip()})`.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelPropertyFileLocation","title":"CheckModelPropertyFileLocation
","text":"Model properties files must follow the guidance provided by dbt here.
Parameters:
Name Type Description Default model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_property_file_location\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelPropertyFileLocation(BaseCheck):\n \"\"\"Model properties files must follow the guidance provided by dbt [here](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview).\n\n Parameters:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_property_file_location\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_property_file_location\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert ( # noqa: PT018\n hasattr(self.model, \"patch_path\")\n and clean_path_str(self.model.patch_path) is not None\n ), f\"`{self.model.name}` is not documented.\"\n\n expected_substr = (\n \"_\".join(clean_path_str(self.model.original_file_path).split(\"/\")[1:-1])\n .replace(\"staging\", \"stg\")\n .replace(\"intermediate\", \"int\")\n .replace(\"marts\", \"\")\n )\n properties_yml_name = clean_path_str(self.model.patch_path).split(\"/\")[-1]\n\n assert properties_yml_name.startswith(\n \"_\",\n ), f\"The properties file for `{self.model.name}` (`{properties_yml_name}`) does not start with an underscore.\"\n assert (\n expected_substr in properties_yml_name\n ), f\"The properties file for `{self.model.name}` (`{properties_yml_name}`) does not contain the expected substring (`{expected_substr}`).\"\n assert properties_yml_name.endswith(\n \"__models.yml\",\n ), f\"The properties file for `{self.model.name}` (`{properties_yml_name}`) does not end with `__models.yml`.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelsTestCoverage","title":"CheckModelsTestCoverage
","text":"Set the minimum percentage of models that have at least one test.
Parameters:
Name Type Description Default min_model_test_coverage_pct
float
The minimum percentage of models that must have at least one test.
required models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
required tests
List[DbtBouncerTestBase]
List of DbtBouncerTestBase objects parsed from manifest.json
.
required Other Parameters (passed via config file):
Name Type Description severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_test_coverage\n min_model_test_coverage_pct: 90\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelsTestCoverage(BaseModel):\n \"\"\"Set the minimum percentage of models that have at least one test.\n\n Parameters:\n min_model_test_coverage_pct (float): The minimum percentage of models that must have at least one test.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n tests (List[DbtBouncerTestBase]): List of DbtBouncerTestBase objects parsed from `manifest.json`.\n\n Other Parameters:\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_test_coverage\n min_model_test_coverage_pct: 90\n ```\n\n \"\"\"\n\n model_config = ConfigDict(extra=\"forbid\")\n\n index: Optional[int] = Field(\n default=None,\n description=\"Index to uniquely identify the check, calculated at runtime.\",\n )\n name: Literal[\"check_model_test_coverage\"]\n min_model_test_coverage_pct: float = Field(\n default=100,\n ge=0,\n le=100,\n )\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n severity: Optional[Literal[\"error\", \"warn\"]] = Field(\n default=\"error\",\n description=\"Severity of the check, one of 'error' or 'warn'.\",\n )\n tests: List[\"DbtBouncerTestBase\"] = Field(default=[])\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_models = len(self.models)\n models_with_tests = []\n for model in self.models:\n for test in self.tests:\n if model.unique_id in test.depends_on.nodes:\n models_with_tests.append(model.unique_id)\n num_models_with_tests = len(set(models_with_tests))\n model_test_coverage_pct = (num_models_with_tests / num_models) * 100\n\n assert (\n model_test_coverage_pct >= self.min_model_test_coverage_pct\n ), f\"Only {model_test_coverage_pct}% of models have at least one test, this is less than the permitted minimum of {self.min_model_test_coverage_pct}%.\"\n
"},{"location":"checks/manifest/check_semantic_models/","title":"Manifest Checks: Semantic Models","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_semantic_models/#manifest.check_semantic_models.CheckSemanticModelOnNonPublicModels","title":"CheckSemanticModelOnNonPublicModels
","text":"Semantic models should be based on public models only.
Receives at execution time:
Name Type Description models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
semantic_model
DbtBouncerSemanticModelBase
The DbtBouncerSemanticModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the semantic model path (i.e the .yml file where the semantic model is configured). Semantic model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the semantic model path (i.e the .yml file where the semantic model is configured). Only semantic model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_semantic_model_based_on_non_public_models\n
Source code in src/dbt_bouncer/checks/manifest/check_semantic_models.py
class CheckSemanticModelOnNonPublicModels(BaseCheck):\n \"\"\"Semantic models should be based on public models only.\n\n Receives:\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n semantic_model (DbtBouncerSemanticModelBase): The DbtBouncerSemanticModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the semantic model path (i.e the .yml file where the semantic model is configured). Semantic model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the semantic model path (i.e the .yml file where the semantic model is configured). Only semantic model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_semantic_model_based_on_non_public_models\n ```\n\n \"\"\"\n\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_semantic_model_based_on_non_public_models\"]\n semantic_model: \"DbtBouncerSemanticModelBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n non_public_upstream_dependencies = []\n for model in self.semantic_model.depends_on.nodes:\n if (\n model.split(\".\")[0] == \"model\"\n and model.split(\".\")[1] == self.semantic_model.package_name\n ):\n model = next(m for m in self.models if m.unique_id == model)\n if model.access.value != \"public\":\n non_public_upstream_dependencies.append(model.name)\n\n assert not non_public_upstream_dependencies, f\"Semantic model `{self.semantic_model.name}` is based on a model(s) that is not public: {non_public_upstream_dependencies}.\"\n
"},{"location":"checks/manifest/check_sources/","title":"Manifest Checks: Sources","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceDescriptionPopulated","title":"CheckSourceDescriptionPopulated
","text":"Sources must have a populated description.
Receives at execution time:
Name Type Description source
DbtBouncerSourceBase
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_description_populated\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceDescriptionPopulated(BaseCheck):\n \"\"\"Sources must have a populated description.\n\n Receives:\n source (DbtBouncerSourceBase): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_description_populated\n ```\n\n \"\"\"\n\n name: Literal[\"check_source_description_populated\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n len(self.source.description.strip()) > 4\n ), f\"`{self.source.source_name}.{self.source.name}` does not have a populated description.\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceFreshnessPopulated","title":"CheckSourceFreshnessPopulated
","text":"Sources must have a populated freshness.
Receives at execution time:
Name Type Description source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_freshness_populated\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceFreshnessPopulated(BaseCheck):\n \"\"\"Sources must have a populated freshness.\n\n Receives:\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_freshness_populated\n ```\n\n \"\"\"\n\n name: Literal[\"check_source_freshness_populated\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n error_msg = f\"`{self.source.source_name}.{self.source.name}` does not have a populated freshness.\"\n assert self.source.freshness is not None, error_msg\n assert (\n self.source.freshness.error_after.count is not None\n and self.source.freshness.error_after.period is not None\n ) or (\n self.source.freshness.warn_after.count is not None\n and self.source.freshness.warn_after.period is not None\n ), error_msg\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceHasMetaKeys","title":"CheckSourceHasMetaKeys
","text":"The meta
config for sources must have the specified keys.
Parameters:
Name Type Description Default keys
NestedDict
A list (that may contain sub-lists) of required keys.
required Receives at execution time:
Name Type Description source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_has_meta_keys\n keys:\n - contact:\n - email\n - slack\n - owner\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceHasMetaKeys(BaseCheck):\n \"\"\"The `meta` config for sources must have the specified keys.\n\n Parameters:\n keys (NestedDict): A list (that may contain sub-lists) of required keys.\n\n Receives:\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_has_meta_keys\n keys:\n - contact:\n - email\n - slack\n - owner\n ```\n\n \"\"\"\n\n keys: \"NestedDict\"\n name: Literal[\"check_source_has_meta_keys\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n missing_keys = find_missing_meta_keys(\n meta_config=self.source.meta,\n required_keys=self.keys.model_dump(),\n )\n\n assert (\n missing_keys == []\n ), f\"`{self.source.source_name}.{self.source.name}` is missing the following keys from the `meta` config: {[x.replace('>>', '') for x in missing_keys]}\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceHasTags","title":"CheckSourceHasTags
","text":"Sources must have the specified tags.
Parameters:
Name Type Description Default source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
required tags
List[str]
List of tags to check for.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_has_tags\n tags:\n - tag_1\n - tag_2\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceHasTags(BaseCheck):\n \"\"\"Sources must have the specified tags.\n\n Parameters:\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n tags (List[str]): List of tags to check for.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_has_tags\n tags:\n - tag_1\n - tag_2\n ```\n\n \"\"\"\n\n name: Literal[\"check_source_has_tags\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n tags: List[str]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n missing_tags = [tag for tag in self.tags if tag not in self.source.tags]\n assert not missing_tags, f\"`{self.source.source_name}.{self.source.name}` is missing required tags: {missing_tags}.\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceLoaderPopulated","title":"CheckSourceLoaderPopulated
","text":"Sources must have a populated loader.
Parameters:
Name Type Description Default source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_loader_populated\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceLoaderPopulated(BaseCheck):\n \"\"\"Sources must have a populated loader.\n\n Parameters:\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_loader_populated\n ```\n\n \"\"\"\n\n name: Literal[\"check_source_loader_populated\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n self.source.loader != \"\"\n ), f\"`{self.source.source_name}.{self.source.name}` does not have a populated loader.\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceNames","title":"CheckSourceNames
","text":"Sources must have a name that matches the supplied regex.
Parameters:
Name Type Description Default source_name_pattern
str
Regexp the source name must match.
required Receives at execution time:
Name Type Description source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_names\n source_name_pattern: >\n ^[a-z0-9_]*$\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceNames(BaseCheck):\n \"\"\"Sources must have a name that matches the supplied regex.\n\n Parameters:\n source_name_pattern (str): Regexp the source name must match.\n\n Receives:\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_names\n source_name_pattern: >\n ^[a-z0-9_]*$\n ```\n\n \"\"\"\n\n name: Literal[\"check_source_names\"]\n source_name_pattern: str\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n re.compile(self.source_name_pattern.strip()).match(self.source.name)\n is not None\n ), f\"`{self.source.source_name}.{self.source.name}` does not match the supplied regex `({self.source_name_pattern.strip()})`.\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceNotOrphaned","title":"CheckSourceNotOrphaned
","text":"Sources must be referenced in at least one model.
Receives at execution time:
Name Type Description models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_not_orphaned\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceNotOrphaned(BaseCheck):\n \"\"\"Sources must be referenced in at least one model.\n\n Receives:\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_not_orphaned\n ```\n\n \"\"\"\n\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_source_not_orphaned\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_refs = sum(\n self.source.unique_id in model.depends_on.nodes for model in self.models\n )\n assert (\n num_refs >= 1\n ), f\"Source `{self.source.source_name}.{self.source.name}` is orphaned, i.e. not referenced by any model.\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourcePropertyFileLocation","title":"CheckSourcePropertyFileLocation
","text":"Source properties files must follow the guidance provided by dbt here.
Receives at execution time:
Name Type Description source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_property_file_location\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourcePropertyFileLocation(BaseCheck):\n \"\"\"Source properties files must follow the guidance provided by dbt [here](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview).\n\n Receives:\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_property_file_location\n ```\n\n \"\"\"\n\n name: Literal[\"check_source_property_file_location\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n path_cleaned = clean_path_str(self.source.original_file_path).replace(\n \"models/staging\", \"\"\n )\n expected_substring = \"_\".join(path_cleaned.split(\"/\")[:-1])\n\n assert path_cleaned.split(\n \"/\",\n )[\n -1\n ].startswith(\n \"_\",\n ), f\"The properties file for `{self.source.source_name}.{self.source.name}` (`{path_cleaned}`) does not start with an underscore.\"\n assert (\n expected_substring in path_cleaned\n ), f\"The properties file for `{self.source.source_name}.{self.source.name}` (`{path_cleaned}`) does not contain the expected substring (`{expected_substring}`).\"\n assert path_cleaned.split(\n \"/\",\n )[\n -1\n ].endswith(\n \"__sources.yml\",\n ), f\"The properties file for `{self.source.source_name}.{self.source.name}` (`{path_cleaned}`) does not end with `__sources.yml`.\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceUsedByModelsInSameDirectory","title":"CheckSourceUsedByModelsInSameDirectory
","text":"Sources can only be referenced by models that are located in the same directory where the source is defined.
Parameters:
Name Type Description Default models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
required source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_used_by_models_in_same_directory\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceUsedByModelsInSameDirectory(BaseCheck):\n \"\"\"Sources can only be referenced by models that are located in the same directory where the source is defined.\n\n Parameters:\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_used_by_models_in_same_directory\n ```\n\n \"\"\"\n\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_source_used_by_models_in_same_directory\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n reffed_models_not_in_same_dir = []\n for model in self.models:\n if (\n self.source.unique_id in model.depends_on.nodes\n and model.original_file_path.split(\"/\")[:-1]\n != self.source.original_file_path.split(\"/\")[:-1]\n ):\n reffed_models_not_in_same_dir.append(model.unique_id.split(\".\")[0])\n\n assert (\n len(reffed_models_not_in_same_dir) == 0\n ), f\"Source `{self.source.source_name}.{self.source.name}` is referenced by models defined in a different directory: {reffed_models_not_in_same_dir}\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceUsedByOnlyOneModel","title":"CheckSourceUsedByOnlyOneModel
","text":"Each source can be referenced by a maximum of one model.
Receives at execution time:
Name Type Description models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_used_by_only_one_model\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceUsedByOnlyOneModel(BaseCheck):\n \"\"\"Each source can be referenced by a maximum of one model.\n\n Receives:\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_used_by_only_one_model\n ```\n\n \"\"\"\n\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_source_used_by_only_one_model\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_refs = sum(\n self.source.unique_id in model.depends_on.nodes for model in self.models\n )\n assert (\n num_refs <= 1\n ), f\"Source `{self.source.source_name}.{self.source.name}` is referenced by more than one model.\"\n
"},{"location":"checks/manifest/check_unit_tests/","title":"Manifest Checks: Unit Tests","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_unit_tests/#manifest.check_unit_tests.CheckUnitTestExpectFormats","title":"CheckUnitTestExpectFormats
","text":"Unit tests can only use the specified formats.
Warning
This check is only supported for dbt 1.8.0 and above.
Parameters:
Name Type Description Default permitted_formats
Optional[List[Literal['csv', 'dict', 'sql']]]
A list of formats that are allowed to be used for expect
input in a unit test.
required Receives at execution time:
Name Type Description manifest_obj
DbtBouncerManifest
The DbtBouncerManifest object parsed from manifest.json
.
unit_test
UnitTests
The UnitTests object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Unit test paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Only unit test paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_unit_test_expect_format\n permitted_formats:\n - csv\n
Source code in src/dbt_bouncer/checks/manifest/check_unit_tests.py
class CheckUnitTestExpectFormats(BaseCheck):\n \"\"\"Unit tests can only use the specified formats.\n\n !!! warning\n\n This check is only supported for dbt 1.8.0 and above.\n\n Parameters:\n permitted_formats (Optional[List[Literal[\"csv\", \"dict\", \"sql\"]]]): A list of formats that are allowed to be used for `expect` input in a unit test.\n\n Receives:\n manifest_obj (DbtBouncerManifest): The DbtBouncerManifest object parsed from `manifest.json`.\n unit_test (UnitTests): The UnitTests object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Unit test paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Only unit test paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_unit_test_expect_format\n permitted_formats:\n - csv\n ```\n\n \"\"\"\n\n manifest_obj: \"DbtBouncerManifest\" = Field(default=None)\n name: Literal[\"check_unit_test_expect_format\"]\n permitted_formats: List[Literal[\"csv\", \"dict\", \"sql\"]] = Field(\n default=[\"csv\", \"dict\", \"sql\"],\n )\n unit_test: \"UnitTests\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if (\n semver.Version.parse(self.manifest_obj.manifest.metadata.dbt_version)\n >= \"1.8.0\"\n ):\n assert (\n self.unit_test.expect.format.value in self.permitted_formats\n ), f\"Unit test `{self.unit_test.name}` has an `expect` format that is not permitted. Permitted formats are: {self.permitted_formats}.\"\n else:\n logging.warning(\n \"The `check_unit_test_expect_format` check is only supported for dbt 1.8.0 and above.\",\n )\n
"},{"location":"checks/manifest/check_unit_tests/#manifest.check_unit_tests.CheckUnitTestGivenFormats","title":"CheckUnitTestGivenFormats
","text":"Unit tests can only use the specified formats.
Warning
This check is only supported for dbt 1.8.0 and above.
Parameters:
Name Type Description Default permitted_formats
Optional[List[Literal['csv', 'dict', 'sql']]]
A list of formats that are allowed to be used for expect
input in a unit test.
required Receives at execution time:
Name Type Description manifest_obj
DbtBouncerManifest
The DbtBouncerManifest object parsed from manifest.json
.
unit_test
UnitTests
The UnitTests object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Unit test paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Only unit test paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_unit_test_given_formats\n permitted_formats:\n - csv\n
Source code in src/dbt_bouncer/checks/manifest/check_unit_tests.py
class CheckUnitTestGivenFormats(BaseCheck):\n \"\"\"Unit tests can only use the specified formats.\n\n !!! warning\n\n This check is only supported for dbt 1.8.0 and above.\n\n Parameters:\n permitted_formats (Optional[List[Literal[\"csv\", \"dict\", \"sql\"]]]): A list of formats that are allowed to be used for `expect` input in a unit test.\n\n Receives:\n manifest_obj (DbtBouncerManifest): The DbtBouncerManifest object parsed from `manifest.json`.\n unit_test (UnitTests): The UnitTests object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Unit test paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Only unit test paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_unit_test_given_formats\n permitted_formats:\n - csv\n ```\n\n \"\"\"\n\n manifest_obj: \"DbtBouncerManifest\" = Field(default=None)\n name: Literal[\"check_unit_test_given_formats\"]\n permitted_formats: List[Literal[\"csv\", \"dict\", \"sql\"]] = Field(\n default=[\"csv\", \"dict\", \"sql\"],\n )\n unit_test: \"UnitTests\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if (\n semver.Version.parse(self.manifest_obj.manifest.metadata.dbt_version)\n >= \"1.8.0\"\n ):\n given_formats = [i.format.value for i in self.unit_test.given]\n assert all(\n e in self.permitted_formats for e in given_formats\n ), f\"Unit test `{self.unit_test.name}` has given formats which are not permitted. Permitted formats are: {self.permitted_formats}.\"\n else:\n logging.warning(\n \"The `check_unit_test_given_formats` check is only supported for dbt 1.8.0 and above.\",\n )\n
"},{"location":"checks/run_results/check_run_results/","title":"Run Results Checks","text":"Note
The below checks require both manifest.json
and run_results.json
to be present.
"},{"location":"checks/run_results/check_run_results/#run_results.check_run_results.CheckRunResultsMaxGigabytesBilled","title":"CheckRunResultsMaxGigabytesBilled
","text":"Each result can have a maximum number of gigabytes billed.
Note
Note that this check only works for the dbt-bigquery
adapter.
Parameters:
Name Type Description Default max_gigabytes_billed
float
The maximum number of gigabytes billed.
required run_result
DbtBouncerRunResult
The DbtBouncerRunResult object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the resource path. Resource paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the resource path. Only resource paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
run_results_checks:\n - name: check_run_results_max_gigabytes_billed\n max_gigabytes_billed: 100\n
Source code in src/dbt_bouncer/checks/run_results/check_run_results.py
class CheckRunResultsMaxGigabytesBilled(BaseCheck):\n \"\"\"Each result can have a maximum number of gigabytes billed.\n\n !!! note\n\n Note that this check only works for the `dbt-bigquery` adapter.\n\n Parameters:\n max_gigabytes_billed (float): The maximum number of gigabytes billed.\n run_result (DbtBouncerRunResult): The DbtBouncerRunResult object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the resource path. Resource paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the resource path. Only resource paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Raises: # noqa:DOC502\n KeyError: If the `dbt-bigquery` adapter is not used.\n\n Example(s):\n ```yaml\n run_results_checks:\n - name: check_run_results_max_gigabytes_billed\n max_gigabytes_billed: 100\n ```\n\n \"\"\"\n\n max_gigabytes_billed: float\n name: Literal[\"check_run_results_max_gigabytes_billed\"]\n run_result: Optional[\"DbtBouncerRunResultBase\"] = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n try:\n gigabytes_billed = self.run_result.adapter_response[\"bytes_billed\"] / (\n 1000**3\n )\n except KeyError as e:\n raise RuntimeError( # noqa: DOC501\n \"`bytes_billed` not found in adapter response. Are you using the `dbt-bigquery` adapter?\",\n ) from e\n\n assert (\n gigabytes_billed < self.max_gigabytes_billed\n ), f\"`{self.run_result.unique_id.split('.')[-2]}` results in ({gigabytes_billed} billed bytes, this is greater than permitted ({self.max_gigabytes_billed}).\"\n
"},{"location":"checks/run_results/check_run_results/#run_results.check_run_results.CheckRunResultsMaxExecutionTime","title":"CheckRunResultsMaxExecutionTime
","text":"Each result can take a maximum duration (seconds).
Parameters:
Name Type Description Default max_execution_time_seconds
float
The maximum execution time (seconds) allowed for a node.
required Receives at execution time:
Name Type Description run_result
DbtBouncerRunResult
The DbtBouncerRunResult object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the resource path. Resource paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the resource path. Only resource paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
run_results_checks:\n - name: check_run_results_max_execution_time\n max_execution_time_seconds: 60\n
run_results_checks:\n - name: check_run_results_max_execution_time\n include: ^models/staging # Not a good idea, here for demonstration purposes only\n max_execution_time_seconds: 10\n
Source code in src/dbt_bouncer/checks/run_results/check_run_results.py
class CheckRunResultsMaxExecutionTime(BaseCheck):\n \"\"\"Each result can take a maximum duration (seconds).\n\n Parameters:\n max_execution_time_seconds (float): The maximum execution time (seconds) allowed for a node.\n\n Receives:\n run_result (DbtBouncerRunResult): The DbtBouncerRunResult object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the resource path. Resource paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the resource path. Only resource paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n run_results_checks:\n - name: check_run_results_max_execution_time\n max_execution_time_seconds: 60\n ```\n ```yaml\n run_results_checks:\n - name: check_run_results_max_execution_time\n include: ^models/staging # Not a good idea, here for demonstration purposes only\n max_execution_time_seconds: 10\n ```\n\n \"\"\"\n\n max_execution_time_seconds: float\n name: Literal[\"check_run_results_max_execution_time\"]\n run_result: Optional[\"DbtBouncerRunResultBase\"] = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n self.run_result.execution_time <= self.max_execution_time_seconds\n ), f\"`{self.run_result.unique_id.split('.')[-1]}` has an execution time ({self.run_result.execution_time} greater than permitted ({self.max_execution_time_seconds}s).\"\n
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":""},{"location":"#welcome-to-dbt-bouncer","title":"Welcome to dbt-bouncer","text":"dbt-bouncer
is an open-source tool that allows you to configure and enforce conventions for your dbt project. The conventions are run against dbt's artifact files (think ./target/manifest.json
) resulting in speedy tests. Conventions can be specified in a .yml
file, allowing maximum customisation to the conventions you wish to follow (or create \ud83d\ude00).
Check out our Getting Started
guide.
"},{"location":"#terminology","title":"Terminology","text":" - Check: A check is a rule run against a dbt artifact.
- Config file: A
.yml
file that specifies which checks to run along with any parameters. - dbt artifacts directory: The directory that contains the dbt artifacts (
manifest.json
, etc.), generally this is ./target
.
"},{"location":"#aims","title":"Aims","text":"dbt-bouncer
aims to:
- Provide a 100% configurable way to enforce conventions in a dbt project.
- Be as fast as possible, running checks against dbt artifacts.
- Be as easy as possible to use, with a simple config file written in
YML
. - Be as flexible as possible, allowing checks to be written in python.
- Provide immediate feedback when run as part of a CI pipeline.
"},{"location":"#about","title":"About","text":"dbt-bouncer
is free software, released under the MIT license. It originated at Xebia Data in Amsterdam, Netherlands. Source code is available on GitHub.
All contributions, in the form of bug reports, pull requests, feedback or discussion are welcome. See the contributing guide for more information.
"},{"location":"CONTRIBUTING/","title":"Contributing to dbt-bouncer
","text":"dbt-bouncer
is open source software. Whether you are a seasoned open source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project.
"},{"location":"CONTRIBUTING/#about-this-document","title":"About this document","text":"There are many ways to contribute to the ongoing development of dbt-bouncer
, such as by participating in discussions and issues.
The rest of this document serves as a more granular guide for contributing code changes to dbt-bouncer
(this repository). It is not intended as a guide for using dbt-bouncer
, and some pieces assume a level of familiarity with Python development (virtualenvs, Poetry
, etc). Specific code snippets in this guide assume you are using macOS or Linux and are comfortable with the command line.
If you get stuck, we're happy to help! Just open an issue or draft PR and we'll do our best to help out.
"},{"location":"CONTRIBUTING/#note","title":"Note","text":" - Branches: All pull requests from community contributors should target the
main
branch (default).
"},{"location":"CONTRIBUTING/#getting-the-code","title":"Getting the code","text":""},{"location":"CONTRIBUTING/#installing-git","title":"Installing git","text":"You will need git
in order to download and modify the dbt-bouncer
source code. On macOS, the best way to download git is to just install Xcode.
"},{"location":"CONTRIBUTING/#contributors","title":"Contributors","text":"You can contribute to dbt-bouncer
by forking the dbt-bouncer
repository. For a detailed overview on forking, check out the GitHub docs on forking. In short, you will need to:
- Fork the
dbt-bouncer
repository. - Clone your fork locally.
- Check out a new branch for your proposed changes.
- Push changes to your fork.
- Open a pull request against
godatadriven/dbt-bouncer
from your forked repository.
"},{"location":"CONTRIBUTING/#setting-up-an-environment","title":"Setting up an environment","text":"There are some tools that will be helpful to you in developing locally. While this is the list relevant for dbt-bouncer
development, many of these tools are used commonly across open-source python projects.
"},{"location":"CONTRIBUTING/#tools","title":"Tools","text":"These are the tools used in dbt-bouncer
development and testing:
click
to create our CLI interface. - GitHub Actions for automating tests and checks, once a PR is pushed to the
dbt-bouncer
repository. make
to run multiple setup or test steps in combination. mypy
for static type checking. Poetry
to manage our python virtual environment. pre-commit
to easily run those checks. Pydantic
to validate our configuration file. pytest
to define, discover, and run tests. Ruff
to lint and format python code.
A deep understanding of these tools in not required to effectively contribute to dbt-bouncer
, but we recommend checking out the attached documentation if you're interested in learning more about each one.
"},{"location":"CONTRIBUTING/#virtual-environments","title":"Virtual environments","text":"We strongly recommend using virtual environments when developing code in dbt-bouncer
. We recommend creating this virtualenv in the root of the dbt-bouncer
repository. To create a new virtualenv, run:
poetry shell\n
This will create a new Python virtual environment.
"},{"location":"CONTRIBUTING/#setting-environment-variables","title":"Setting environment variables","text":"Set required environment variables by copying .env.example
to .env
and updating the values.
"},{"location":"CONTRIBUTING/#running-dbt-bouncer-in-development","title":"Running dbt-bouncer
in development","text":""},{"location":"CONTRIBUTING/#installation","title":"Installation","text":"First make sure that you set up your virtualenv
as described in Setting up an environment. Next, install dbt-bouncer
, its dependencies and pre-commit
:
poetry install\npoetry run pre-commit install\n
When installed in this way, any changes you make to your local copy of the source code will be reflected immediately in your next dbt-bouncer
run.
"},{"location":"CONTRIBUTING/#running-dbt-bouncer","title":"Running dbt-bouncer
","text":"With your virtualenv activated, the dbt-bouncer
script should point back to the source code you've cloned on your machine. You can verify this by running which dbt-bouncer
. This command should show you a path to an executable in your virtualenv. You can run dbt-bouncer
using the provided example configuration file via:
poetry run dbt-bouncer --config-file dbt-bouncer-example.yml\n
"},{"location":"CONTRIBUTING/#testing","title":"Testing","text":"Once you're able to manually test that your code change is working as expected, it's important to run existing automated tests, as well as adding some new ones. These tests will ensure that: - Your code changes do not unexpectedly break other established functionality - Your code changes can handle all known edge cases - The functionality you're adding will keep working in the future
"},{"location":"CONTRIBUTING/#note_1","title":"Note","text":" - Generating dbt artifacts: If you change the configuration of the dbt project located in
dbt_project
then you will need to re-generate the dbt artifacts used in testing. To do so, run:
make build-artifacts\n
"},{"location":"CONTRIBUTING/#test-commands","title":"Test commands","text":"There are a few methods for running tests locally.
"},{"location":"CONTRIBUTING/#makefile","title":"makefile
","text":"There are multiple targets in the makefile
to run common test suites, most notably:
# Runs unit tests\nmake test-unit\n\n# Runs integration tests\nmake test-integration\n\n# Runs all tests\nmake test\n
"},{"location":"CONTRIBUTING/#pre-commit","title":"pre-commit
","text":"pre-commit
takes care of running all code-checks for formatting and linting. Run poetry run pre-commit install
to install pre-commit
in your local environment. Once this is done you can use the git pre-commit hooks to ensure proper formatting and linting.
"},{"location":"CONTRIBUTING/#pytest","title":"pytest
","text":"Finally, you can also run a specific test or group of tests using pytest
directly. With a virtualenv active and dev dependencies installed you can do things like:
# run all unit tests in a file\npoetry run pytest ./tests/unit/checks/catalog/test_columns.py\n\n# run a specific unit test\npoetry run pytest ./tests/unit/checks/catalog/test_columns.py::test_check_columns_are_documented_in_public_models\n
See pytest usage docs for an overview of useful command-line options.
"},{"location":"CONTRIBUTING/#assorted-development-tips","title":"Assorted development tips","text":" - Append
# type: ignore
to the end of a line if you need to disable mypy
on that line, preferably with the specific rule to ignore such as # type: ignore[union-attr]
.
"},{"location":"CONTRIBUTING/#adding-a-new-check","title":"Adding a new check","text":"To add a new check follow the below steps:
- In
./src/dbt_bouncer/checks
choose the appropriate directory for your check. For example, if your check only requires the manifest.json
then use the manifest
directory, if your check requires the catalog.json
then use the catalog
directory. - Within the chosen directory assess if a suitable file already exists. For example, if your check applies to a model then
manifest/check_models.py
is a suitable location. -
Within the chosen file, add a Pydantic model, this object must meet the following criteria:
- Start with \"Check\".
- Inherit from
dbt_bouncer.check_base.BaseCheck
. - Have a
name
attribute that is a string whose value is the snake case equivalent of the class name. - A
default
value provided for optional input arguments and arguments that are received at execution time. - Have a doc string that includes a description of the check, a list of possible input parameters and at least one example.
- A clear message in the event of a failure.
-
After the check is added, add the check to dbt-bouncer-example.yml
and run dbt-bouncer --config-file dbt-bouncer-example.yml
to ensure the check succeeds.
- (Optional) If the dbt project located in
./dbt_project
needs to be updated then do so and also run make build-artifacts
to generate the new test artifacts. - Add at least one happy path and one unhappy path test to
./tests
. The appropriate test file will be the one matching the directory of the check. For example, if the check is in ./src/dbt_bouncer/checks/catalog/check_columns.py
then the test file will be ./tests/unit/checks/catalog/test_columns.py
. - Run
make test
to ensure the tests pass. - Open a PR \ud83c\udf89!
"},{"location":"CONTRIBUTING/#submitting-a-pull-request","title":"Submitting a Pull Request","text":"Code can be merged into the current development branch main
by opening a pull request. If the proposal looks like it's on the right track, then a dbt-bouncer
maintainer will review the PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code. Once merged, your contribution will be available for the next release of dbt-bouncer
.
Automated tests run via GitHub Actions. If you're a first-time contributor, all tests will require a maintainer to approve.
Once all tests are passing and your PR has been approved, a dbt-bouncer
maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada:
"},{"location":"cli/","title":"CLI","text":"This page provides documentation for the dbt-bouncer
CLI.
"},{"location":"cli/#dbt-bouncer","title":"dbt-bouncer","text":"Entrypoint for dbt-bouncer.
Raises: RuntimeError: If output file has an invalid extension.
Usage:
dbt-bouncer [OPTIONS]\n
Options:
--config-file PUREPATH Location of the YML config file.\n --output-file PATH Location of the json file where check metadata will\n be saved.\n -v, --verbosity Verbosity.\n --version Show the version and exit.\n --help Show this message and exit.\n
"},{"location":"cli/#exit-codes","title":"Exit codes","text":"dbt-bouncer
returns the following exit codes:
-
0
: All checks have succeeded.
-
1
:
- At least one check has failed. Check the logs for more information.
- A fatal error occurred during check setup or check execution. Check the logs for more information.
"},{"location":"config_file/","title":"Config file","text":"dbt-bouncer
requires a config file which determines what checks are run. The following options are available, in order of priority:
- A file passed via the
--config-file
CLI flag. - A file named
dbt-bouncer.yml
in the current working directory. - A
[tool.dbt-bouncer]
section in pyproject.toml
.
Here is an example config file in yaml
:
# [Optional] Directory where the dbt artifacts exists, generally the `target` directory inside a dbt project. Defaults to `./target`.\ndbt_artifacts_dir: target\n\nmanifest_checks:\n - name: check_macro_name_matches_file_name\n - name: check_model_names\n include: ^models/staging\n model_name_pattern: ^stg_\n
And the same config in toml
:
[tool.dbt-bouncer]\n# [Optional] Directory where the dbt artifacts exists, generally the `target` directory inside a dbt project. Defaults to `./target`.\ndbt_artifacts_dir = \"target\"\n\n[[tool.dbt-bouncer.manifest_checks]]\nname = \"check_macro_name_matches_file_name\"\n\n[[tool.dbt-bouncer.manifest_checks]]\nname = \"check_model_names\"\ninclude = \"^models/staging\"\nmodel_name_pattern = \"^stg_\"\n
For more example config files, see here.
"},{"location":"config_file/#common-arguments","title":"Common arguments","text":""},{"location":"config_file/#exclude-and-include","title":"Exclude and Include","text":"Most (but not all) checks accept the following optional arguments:
exclude
: Regexp to match which original file paths to exclude. include
: Regexp to match which original file paths to include.
Example per resource type:
Exposures
: The original file path to the properties file where the source is defined, e.g. ^models/marts/finance
will match exposures defined in ./models/marts/finance/_exposures.yml
. Macros
: The original file path to the macro file, e.g. ^macros/system
will match files like ./macros/system/generate_schema_name.sql
. Models
: The original file path to the model file, e.g. ^marts
will match files like ./models/marts/customers.sql
. Run results
: The original file path to the file associated with the resource, e.g. ^seeds/finance
will match seeds in ./seeds/finance
, ^models/staging
will match models and tests in ./models/staging
. Semantic models
: The original file path to the properties file where the semantic model is defined, e.g. ^models/marts/finance
will match semantic models defined in ./models/marts/finance/_finance__semantic_models.yml
. Sources
: The original file path to the properties file where the source is defined, e.g. ^models/staging/crm
will match sources defined in ./models/staging/crm/_crm__sources.yml
. Unit tests
: The original file path to the properties file where the unit test is defined, e.g. ^models/staging/crm
will match unit tests defined in ^staging/crm/_stg_crm__unit_tests.yml
.
To determine if a check accepts these arguments view the Checks page.
Note
exclude
and include
can be specified at both the check level and the global level. Should both levels be specified, then the check level is applied. All the below examples result in the check_model_names
check being run on all models in ./models/staging
:
# Specify `include` at the check level only\nmanifest_checks:\n - name: check_model_names\n include: ^models/staging\n model_name_pattern: ^stg_\n
# Specify `include` at the check and global levels\ninclude: ^models/marts\nmanifest_checks:\n - name: check_model_names\n include: ^models/staging\n model_name_pattern: ^stg_\n
# Specify `include` at the global level only\ninclude: ^models/staging\nmanifest_checks:\n - name: check_model_names\n model_name_pattern: ^stg_\n
Note
When compiled on Windows machines, keys such as original_file_path
, patch_path
and path
take the form:
models\\\\staging\\\\crm\\\\model_1.sql\n
When compiled on Linux and Mac machines, these same keys take the form:
models/staging/crm/model_1.sql\n
dbt-bouncer
converts all of these paths to the Linux/Mac form, hence when you are supplying values to exclude
and include
you should use the Linux/Mac form.
"},{"location":"config_file/#severity","title":"Severity","text":"All checks accept a severity
argument, valid values are:
error
: If the check fails then dbt-bouncer
will return a non-zero exit code. warn
: If the check fails then dbt-bouncer
will return a non-zero exit code.
severity
can also be specified globally, this is useful when applying dbt-bouncer
to a pre-existing dbt project. It allows you to run dbt-bouncer
, identify the checks that fail and address the failures in your own time without receiving non-zero exit codes:
# Specify `severity` at the global levels: all checks will have a `warn` severity, avoiding non-zero exit codes.\nseverity: warn\n\nmanifest_checks:\n - name: check_exposure_based_on_view\n ...\n
Note
severity
can be specified at both the check level and the global level. Should both levels be specified, then the global level is applied.
# No `severity` specified: check will have an `error` severity.\nmanifest_checks:\n - name: check_exposure_based_on_view\n
# Specify `severity` at the check level only: check will have a `warn` severity.\nmanifest_checks:\n - name: check_exposure_based_on_view\n severity: warn\n
# Specify `severity` at the check and global levels: check will have a `warn` severity.\nseverity: warn\nmanifest_checks:\n - name: check_exposure_based_on_view\n severity: error\n
# Specify `severity` at the global level only: check will have a `warn` severity.\nseverity: warn\nmanifest_checks:\n - name: check_exposure_based_on_view\n
"},{"location":"faq/","title":"Frequently Asked Questions","text":""},{"location":"faq/#can-other-tools-perform-the-same-checks-as-dbt-bouncer","title":"Can other tools perform the same checks as dbt-bouncer
?","text":"There are several other tools that perform similar tasks as dbt-bouncer
.
- dbt-checkpoint: A collection of
pre-commit
hooks for dbt projects. Tests are written in python. Configuration is performed via .pre-commit-config.yaml
. Provided the dbt artifacts have already been generated, dbt-checkpoint
does not need access to the underlying database. The hooks execute when a new commit is made, as such dbt-checkpoint
is designed to be run only as part of pre-commit
. - dbt-project-evaluator: This is a dbt package from dbt Labs. Tests are written in
.sql
files using a combination of Jinja and SQL. Configuration is performed via dbt_project.yml
and seed files (i.e. csv files). Requires a connection to underlying database. Designed to be run both in a CI pipeline and also during active development. - dbt-score: This is a python package installable via
pip
. A collection of tests that apply only to dbt models. Tests can be executed from the command line. Tests are written in python. Configuration is performed via a pyproject.toml
file. Provided the dbt artifacts have already been generated, dbt-score
does not need access to the underlying database. Designed to be run during development.
While the above tools inhabit the same space as dbt-bouncer
they do not provide what we consider to be the optimum experience that dbt-bouncer
provides:
- Designed to run both locally and in a CI pipeline.
- Configurable via a file format,
YML
, that dbt developers are already familiar with. - Does not require database access.
- Can run tests against any of dbt's artifacts.
- Allows tests to be written in python.
As such we consider dbt-bouncer
to be the best tool to enforce conventions in a dbt project.
Tip
dbt-bouncer
can perform all the tests currently included in dbt-checkpoint
, dbt-project-evaluator
and dbt-score
. If you see an existing test that is not possible with dbt-bouncer
, open an issue and we'll add it!
"},{"location":"faq/#how-to-set-up-dbt-bouncer-in-a-monorepo","title":"How to set up dbt-bouncer
in a monorepo?","text":"A monorepo may consist of one directory with a dbt project and other directories with unrelated code. It may be desired for dbt-bouncer
to be configured from the root directory. Sample directory tree:
.\n\u251c\u2500\u2500 dbt-bouncer.yml\n\u251c\u2500\u2500 README.md\n\u251c\u2500\u2500 dbt-project\n\u2502 \u251c\u2500\u2500 models\n\u2502 \u251c\u2500\u2500 dbt_project.yml\n\u2502 \u2514\u2500\u2500 profiles.yml\n\u2514\u2500\u2500 package-a\n \u251c\u2500\u2500 src\n \u251c\u2500\u2500 tests\n \u2514\u2500\u2500 package.json\n
To ease configuration you can use exclude
or include
at the global level (see Config File for more details). For the above example dbt-bouncer.yml
could be configured as:
dbt_artifacts_dir: dbt-project/target\ninclude: ^dbt-project\n\nmanifest_checks:\n - name: check_exposure_based_on_non_public_models\n
dbt-bouncer
can now be run from the root directory.
"},{"location":"getting_started/","title":"Getting Started","text":""},{"location":"getting_started/#how-to-run-dbt-bouncer","title":"How to run dbt-bouncer
","text":" -
Generate dbt artifacts by running a dbt command:
dbt parse
to generate a manifest.json
artifact. dbt docs generate
to generate a catalog.json
artifact (necessary if you are using catalog checks). dbt run
(or any other command that implies it e.g. dbt build
) to generate a run_results.json
artifact (necessary if you are using run results checks).
-
Create a dbt-bouncer.yml
config file, details here.
-
Run dbt-bouncer
to validate that your conventions are being maintained.
"},{"location":"getting_started/#installing-with-python","title":"Installing with Python","text":"Install from pypi.org:
pip install dbt-bouncer # or via any other package manager\n
Run:
dbt-bouncer --config-file <PATH_TO_CONFIG_FILE>\n
Running dbt-bouncer (X.X.X)...\nLoaded config from dbt-bouncer-example.yml...\nValidating conf...\n
dbt-bouncer
also supports a verbose mode, run:
dbt-bouncer --config-file <PATH_TO_CONFIG_FILE> -v\n
Running dbt-bouncer (X.X.X)...\nconfig_file=PosixPath('dbt-bouncer-example.yml')\nconfig_file_source='COMMANDLINE'\nConfig file passed via command line: dbt-bouncer-example.yml\nLoading config from /home/pslattery/repos/dbt-bouncer/dbt-bouncer-example.yml...\nLoading config from dbt-bouncer-example.yml...\nLoaded config from dbt-bouncer-example.yml...\nconf={'dbt_artifacts_dir': 'dbt_project/target', 'catalog_checks': [{'name': 'check_column_name_complies_to_column_type', 'column_name_pattern': '^is_.*', 'exclude': '^staging', 'types': ['BOOLEAN']}]}\nValidating conf...\n
"},{"location":"getting_started/#running-as-an-executable-using-uv","title":"Running as an executable using uv","text":"Run dbt-bouncer
as a standalone Python executable using uv
:
uvx dbt-bouncer --config-file <PATH_TO_CONFIG_FILE>\n
"},{"location":"getting_started/#github-actions","title":"GitHub Actions","text":"Run dbt-bouncer
as part of your CI pipeline:
name: CI pipeline\n\non:\n pull_request:\n branches:\n - main\n\njobs:\n run-dbt-bouncer:\n permissions:\n pull-requests: write # Required to write a comment on the PR\n runs-on: ubuntu-latest\n steps:\n - name: Checkout\n uses: actions/checkout@v4\n\n - name: Generate or fetch dbt artifacts\n run: ...\n\n - uses: godatadriven/dbt-bouncer@vX.X\n with:\n config-file: ./<PATH_TO_CONFIG_FILE>\n output-file: results.json # optional, default does not save a results file\n send-pr-comment: true # optional, defaults to true\n verbose: false # optional, defaults to false\n
We recommend pinning both a major and minor version number.
"},{"location":"getting_started/#docker","title":"Docker","text":"Run dbt-bouncer
via Docker:
docker run --rm \\\n --volume \"$PWD\":/app \\\n ghcr.io/godatadriven/dbt-bouncer:vX.X.X \\\n --config-file /app/<PATH_TO_CONFIG_FILE>\n
"},{"location":"getting_started/#pex","title":"Pex","text":"You can also run the .pex
(Python EXecutable) artifact directly once you have a python executable (3.8 -> 3.12) installed:
wget https://github.com/godatadriven/dbt-bouncer/releases/download/vX.X.X/dbt-bouncer.pex -O dbt-bouncer.pex\n\npython dbt-bouncer.pex --config-file $PWD/<PATH_TO_CONFIG_FILE>\n
"},{"location":"getting_started/#how-to-contribute-a-check-to-dbt-bouncer","title":"How to contribute a check to dbt-bouncer
","text":"See Adding a new check.
"},{"location":"getting_started/#how-to-add-a-custom-check-to-dbt-bouncer","title":"How to add a custom check to dbt-bouncer
","text":"In addition to the checks built into dbt-bouncer
, the ability to add custom checks is supported. This allows users to write checks that are specific to the conventions of their projects. To add a custom check:
- Create an empty directory and add a
custom_checks_dir
key to your config file. The value of this key should be the path to the directory you just created, relative to where the config file is located. - In this directory create an empty
__init__.py
file. - In this directory create a subdirectory named
catalog
, manifest
or run_results
depending on the type of artifact you want to check. -
In this subdirectory create a python file that defines a check. The check must meet the following criteria:
- Start with \"Check\".
- Inherit from dbt_bouncer.check_base.BaseCheck.
- Have a name attribute that is a string whose value is the snake case equivalent of the class name.
- A default value provided for optional input arguments and arguments that are received at execution time.
- Have a doc string that includes a description of the check, a list of possible input parameters and at least one example.
- A clear message in the event of a failure.
-
In your config file, add the name of the check and any desired arguments.
- Run
dbt-bouncer
, your custom check will be executed.
An example:
-
Directory tree:
.\n\u251c\u2500\u2500 dbt-bouncer.yml\n\u251c\u2500\u2500 dbt_project.yml\n\u251c\u2500\u2500 my_custom_checks\n| \u251c\u2500\u2500 __init__.py\n| \u2514\u2500\u2500 manifest\n| \u2514\u2500\u2500 check_custom_to_me.py\n\u2514\u2500\u2500 target\n \u2514\u2500\u2500 manifest.json\n
-
Contents of check_custom_to_me.py
:
from typing import TYPE_CHECKING, Literal\n\nfrom pydantic import Field\n\nfrom dbt_bouncer.check_base import BaseCheck\n\nif TYPE_CHECKING:\n import warnings\n\n with warnings.catch_warnings():\n warnings.filterwarnings(\"ignore\", category=UserWarning)\n from dbt_bouncer.parsers import DbtBouncerModelBase\n\n\nclass CheckModelDepcrecationDate(BaseCheck):\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_deprecation_date\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n\n assert self.model.deprecation_date is not None, f\"`{self.model.name}` requires a `deprecation_date` to be set.\"\n
-
Contents of dbt-bouncer.yml
:
custom_checks_dir: my_custom_checks\n\nmanifest_checks:\n - name: check_model_deprecation_date\n include: ^models/staging/legacy_erp\n
"},{"location":"checks/","title":"Checks","text":"dbt-bouncer
runs checks against artifacts from dbt. These checks fall into three categories:
- Catalog checks:
- Catalog Sources
- Columns
- Manifest checks:
- Exposures
- Lineage
- Macros
- Metadata
- Models
- Semantic Models
- Sources
- Unit Tests
- Run Results checks:
- Run Results
"},{"location":"checks/catalog/check_catalog_sources/","title":"Catalog Checks: Catalog Sources","text":"Note
The below checks require both catalog.json
and manifest.json
to be present.
"},{"location":"checks/catalog/check_catalog_sources/#catalog.check_catalog_sources.CheckSourceColumnsAreAllDocumented","title":"CheckSourceColumnsAreAllDocumented
","text":"All columns in a source should be included in the source's properties file, i.e. .yml
file.
Receives at execution time:
Name Type Description catalog_source
CatalogTable
The CatalogTable object to check.
sources
List[DbtBouncerSourceBase]
List of DbtBouncerSourceBase objects parsed from catalog.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
catalog_checks:\n - name: check_source_columns_are_all_documented\n
Source code in src/dbt_bouncer/checks/catalog/check_catalog_sources.py
class CheckSourceColumnsAreAllDocumented(BaseCheck):\n \"\"\"All columns in a source should be included in the source's properties file, i.e. `.yml` file.\n\n Receives:\n catalog_source (CatalogTable): The CatalogTable object to check.\n sources (List[DbtBouncerSourceBase]): List of DbtBouncerSourceBase objects parsed from `catalog.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n catalog_checks:\n - name: check_source_columns_are_all_documented\n ```\n\n \"\"\"\n\n catalog_source: \"CatalogTable\" = Field(default=None)\n name: Literal[\"check_source_columns_are_all_documented\"]\n sources: List[\"DbtBouncerSourceBase\"] = Field(default=[])\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n source = next(\n s for s in self.sources if s.unique_id == self.catalog_source.unique_id\n )\n undocumented_columns = [\n v.name\n for _, v in self.catalog_source.columns.items()\n if v.name not in source.columns\n ]\n assert not undocumented_columns, f\"`{self.catalog_source.unique_id}` has columns that are not included in the sources properties file: {undocumented_columns}\"\n
"},{"location":"checks/catalog/check_columns/","title":"Catalog Checks: Columns","text":"Note
The below checks require both catalog.json
and manifest.json
to be present.
"},{"location":"checks/catalog/check_columns/#catalog.check_columns.CheckColumnDescriptionPopulated","title":"CheckColumnDescriptionPopulated
","text":"Columns must have a populated description.
Receives at execution time:
Name Type Description catalog_node
CatalogTable
The CatalogTable object to check.
models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_column_description_populated\n include: ^models/marts\n
Source code in src/dbt_bouncer/checks/catalog/check_columns.py
class CheckColumnDescriptionPopulated(BaseCheck):\n \"\"\"Columns must have a populated description.\n\n Receives:\n catalog_node (CatalogTable): The CatalogTable object to check.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_column_description_populated\n include: ^models/marts\n ```\n\n \"\"\"\n\n catalog_node: \"CatalogTable\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_column_description_populated\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if self.catalog_node.unique_id.split(\".\")[0] == \"model\":\n model = next(\n m for m in self.models if m.unique_id == self.catalog_node.unique_id\n )\n non_complying_columns = []\n for _, v in self.catalog_node.columns.items():\n if (\n model.columns.get(v.name) is None\n or len(model.columns[v.name].description.strip()) <= 4\n ):\n non_complying_columns.append(v.name)\n\n assert not non_complying_columns, f\"`{self.catalog_node.unique_id.split('.')[-1]}` has columns that do not have a populated description: {non_complying_columns}\"\n
"},{"location":"checks/catalog/check_columns/#catalog.check_columns.CheckColumnNameCompliesToColumnType","title":"CheckColumnNameCompliesToColumnType
","text":"Columns with specified data types must comply to the specified regexp naming pattern.
Parameters:
Name Type Description Default column_name_pattern
str
Regex pattern to match the model name.
required types
List[str]
List of data types to check.
required Receives at execution time:
Name Type Description catalog_node
CatalogTable
The CatalogTable object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
catalog_checks:\n # DATE columns must end with \"_date\"\n - name: check_column_name_complies_to_column_type\n column_name_pattern: .*_date$\n types:\n - DATE\n
catalog_checks:\n # BOOLEAN columns must start with \"is_\"\n - name: check_column_name_complies_to_column_type\n column_name_pattern: ^is_.*\n types:\n - BOOLEAN\n
catalog_checks:\n # Columns of all types must consist of lowercase letters and underscores. Note that the specified types depend on the underlying database.\n - name: check_column_name_complies_to_column_type\n column_name_pattern: ^[a-z_]*$\n types:\n - BIGINT\n - BOOLEAN\n - DATE\n - DOUBLE\n - INTEGER\n - VARCHAR\n
Source code in src/dbt_bouncer/checks/catalog/check_columns.py
class CheckColumnNameCompliesToColumnType(BaseCheck):\n \"\"\"Columns with specified data types must comply to the specified regexp naming pattern.\n\n Parameters:\n column_name_pattern (str): Regex pattern to match the model name.\n types (List[str]): List of data types to check.\n\n Receives:\n catalog_node (CatalogTable): The CatalogTable object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n catalog_checks:\n # DATE columns must end with \"_date\"\n - name: check_column_name_complies_to_column_type\n column_name_pattern: .*_date$\n types:\n - DATE\n ```\n ```yaml\n catalog_checks:\n # BOOLEAN columns must start with \"is_\"\n - name: check_column_name_complies_to_column_type\n column_name_pattern: ^is_.*\n types:\n - BOOLEAN\n ```\n ```yaml\n catalog_checks:\n # Columns of all types must consist of lowercase letters and underscores. Note that the specified types depend on the underlying database.\n - name: check_column_name_complies_to_column_type\n column_name_pattern: ^[a-z_]*$\n types:\n - BIGINT\n - BOOLEAN\n - DATE\n - DOUBLE\n - INTEGER\n - VARCHAR\n ```\n\n \"\"\"\n\n catalog_node: \"CatalogTable\" = Field(default=None)\n column_name_pattern: str\n name: Literal[\"check_column_name_complies_to_column_type\"]\n types: List[str]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n non_complying_columns = [\n v.name\n for _, v in self.catalog_node.columns.items()\n if v.type in self.types\n and re.compile(self.column_name_pattern.strip()).match(v.name) is None\n ]\n\n assert not non_complying_columns, f\"`{self.catalog_node.unique_id.split('.')[-1]}` has columns that don't comply with the specified regexp pattern (`{self.column_name_pattern}`): {non_complying_columns}\"\n
"},{"location":"checks/catalog/check_columns/#catalog.check_columns.CheckColumnsAreAllDocumented","title":"CheckColumnsAreAllDocumented
","text":"All columns in a model should be included in the model's properties file, i.e. .yml
file.
Receives at execution time:
Name Type Description catalog_node
CatalogTable
The CatalogTable object to check.
models
List[DbtBouncerModel]
List of DbtBouncerModel objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
catalog_checks:\n - name: check_columns_are_all_documented\n
Source code in src/dbt_bouncer/checks/catalog/check_columns.py
class CheckColumnsAreAllDocumented(BaseCheck):\n \"\"\"All columns in a model should be included in the model's properties file, i.e. `.yml` file.\n\n Receives:\n catalog_node (CatalogTable): The CatalogTable object to check.\n models (List[DbtBouncerModel]): List of DbtBouncerModel objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n catalog_checks:\n - name: check_columns_are_all_documented\n ```\n\n \"\"\"\n\n catalog_node: \"CatalogTable\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_columns_are_all_documented\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if self.catalog_node.unique_id.split(\".\")[0] == \"model\":\n model = next(\n m for m in self.models if m.unique_id == self.catalog_node.unique_id\n )\n undocumented_columns = [\n v.name\n for _, v in self.catalog_node.columns.items()\n if v.name not in model.columns\n ]\n assert not undocumented_columns, f\"`{self.catalog_node.unique_id.split('.')[-1]}` has columns that are not included in the models properties file: {undocumented_columns}\"\n
"},{"location":"checks/catalog/check_columns/#catalog.check_columns.CheckColumnsAreDocumentedInPublicModels","title":"CheckColumnsAreDocumentedInPublicModels
","text":"Columns should have a populated description in public models.
Receives at execution time:
Name Type Description catalog_node
CatalogTable
The CatalogTable object to check.
models
List[DbtBouncerModel]
List of DbtBouncerModel objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
catalog_checks:\n - name: check_columns_are_documented_in_public_models\n
Source code in src/dbt_bouncer/checks/catalog/check_columns.py
class CheckColumnsAreDocumentedInPublicModels(BaseCheck):\n \"\"\"Columns should have a populated description in public models.\n\n Receives:\n catalog_node (CatalogTable): The CatalogTable object to check.\n models (List[DbtBouncerModel]): List of DbtBouncerModel objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n catalog_checks:\n - name: check_columns_are_documented_in_public_models\n ```\n\n \"\"\"\n\n catalog_node: \"CatalogTable\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_columns_are_documented_in_public_models\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if self.catalog_node.unique_id.split(\".\")[0] == \"model\":\n model = next(\n m for m in self.models if m.unique_id == self.catalog_node.unique_id\n )\n non_complying_columns = []\n for _, v in self.catalog_node.columns.items():\n if model.access.value == \"public\":\n column_config = model.columns.get(v.name)\n if (\n column_config is None\n or len(column_config.description.strip()) < 4\n ):\n non_complying_columns.append(v.name)\n\n assert not non_complying_columns, f\"`{self.catalog_node.unique_id.split('.')[-1]}` is a public model but has columns that don't have a populated description: {non_complying_columns}\"\n
"},{"location":"checks/catalog/check_columns/#catalog.check_columns.CheckColumnHasSpecifiedTest","title":"CheckColumnHasSpecifiedTest
","text":"Columns that match the specified regexp pattern must have a specified test.
Parameters:
Name Type Description Default column_name_pattern
str
Regex pattern to match the column name.
required test_name
str
Name of the test to check for.
required Receives at execution time:
Name Type Description catalog_node
CatalogTable
The CatalogTable object to check.
tests
List[DbtBouncerTestBase]
List of DbtBouncerTestBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
catalog_checks:\n - name: check_column_has_specified_test\n column_name_pattern: ^is_.*\n test_name: not_null\n
Source code in src/dbt_bouncer/checks/catalog/check_columns.py
class CheckColumnHasSpecifiedTest(BaseCheck):\n \"\"\"Columns that match the specified regexp pattern must have a specified test.\n\n Parameters:\n column_name_pattern (str): Regex pattern to match the column name.\n test_name (str): Name of the test to check for.\n\n Receives:\n catalog_node (CatalogTable): The CatalogTable object to check.\n tests (List[DbtBouncerTestBase]): List of DbtBouncerTestBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n catalog_checks:\n - name: check_column_has_specified_test\n column_name_pattern: ^is_.*\n test_name: not_null\n ```\n\n \"\"\"\n\n catalog_node: \"CatalogTable\" = Field(default=None)\n column_name_pattern: str\n name: Literal[\"check_column_has_specified_test\"]\n test_name: str\n tests: List[\"DbtBouncerTestBase\"] = Field(default=[])\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n columns_to_check = [\n v.name\n for _, v in self.catalog_node.columns.items()\n if re.compile(self.column_name_pattern.strip()).match(v.name) is not None\n ]\n relevant_tests = [\n t\n for t in self.tests\n if hasattr(t, \"test_metadata\") is True\n and hasattr(t, \"attached_node\") is True\n and t.test_metadata.name == self.test_name\n and t.attached_node == self.catalog_node.unique_id\n ]\n non_complying_columns = [\n c\n for c in columns_to_check\n if f\"{self.catalog_node.unique_id}.{c}\"\n not in [f\"{t.attached_node}.{t.column_name}\" for t in relevant_tests]\n ]\n\n assert not non_complying_columns, f\"`{self.catalog_node.unique_id.split('.')[-1]}` has columns that should have a `{self.test_name}` test: {non_complying_columns}\"\n
"},{"location":"checks/manifest/check_exposures/","title":"Manifest Checks: Exposures","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_exposures/#manifest.check_exposures.CheckExposureOnNonPublicModels","title":"CheckExposureOnNonPublicModels
","text":"Exposures should be based on public models only.
Receives at execution time:
Name Type Description exposure
DbtBouncerExposureBase
The DbtBouncerExposureBase object to check.
models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Exposure paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Only exposure paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_exposure_based_on_non_public_models\n
Source code in src/dbt_bouncer/checks/manifest/check_exposures.py
class CheckExposureOnNonPublicModels(BaseCheck):\n \"\"\"Exposures should be based on public models only.\n\n Receives:\n exposure (DbtBouncerExposureBase): The DbtBouncerExposureBase object to check.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Exposure paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Only exposure paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_exposure_based_on_non_public_models\n ```\n\n \"\"\"\n\n exposure: \"DbtBouncerExposureBase\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_exposure_based_on_non_public_models\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n non_public_upstream_dependencies = []\n for model in self.exposure.depends_on.nodes:\n if (\n model.split(\".\")[0] == \"model\"\n and model.split(\".\")[1] == self.exposure.package_name\n ):\n model = next(m for m in self.models if m.unique_id == model)\n if model.access.value != \"public\":\n non_public_upstream_dependencies.append(model.name)\n\n assert not non_public_upstream_dependencies, f\"`{self.exposure.name}` is based on a model(s) that is not public: {non_public_upstream_dependencies}.\"\n
"},{"location":"checks/manifest/check_exposures/#manifest.check_exposures.CheckExposureOnView","title":"CheckExposureOnView
","text":"Exposures should not be based on views.
Parameters:
Name Type Description Default materializations_to_include
Optional[List[str]]
List of materializations to include in the check.
required Receives at execution time:
Name Type Description exposure
DbtBouncerExposureBase
The DbtBouncerExposureBase object to check.
models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Exposure paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Only exposure paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_exposure_based_on_view\n
manifest_checks:\n - name: check_exposure_based_on_view\n materializations_to_include:\n - ephemeral\n - my_custom_materialization\n - view\n
Source code in src/dbt_bouncer/checks/manifest/check_exposures.py
class CheckExposureOnView(BaseCheck):\n \"\"\"Exposures should not be based on views.\n\n Parameters:\n materializations_to_include (Optional[List[str]]): List of materializations to include in the check.\n\n Receives:\n exposure (DbtBouncerExposureBase): The DbtBouncerExposureBase object to check.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Exposure paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the exposure path (i.e the .yml file where the exposure is configured). Only exposure paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_exposure_based_on_view\n ```\n ```yaml\n manifest_checks:\n - name: check_exposure_based_on_view\n materializations_to_include:\n - ephemeral\n - my_custom_materialization\n - view\n ```\n\n \"\"\"\n\n exposure: \"DbtBouncerExposureBase\" = Field(default=None)\n materializations_to_include: List[str] = Field(\n default=[\"ephemeral\", \"view\"],\n )\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_exposure_based_on_view\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n non_table_upstream_dependencies = []\n for model in self.exposure.depends_on.nodes:\n if (\n model.split(\".\")[0] == \"model\"\n and model.split(\".\")[1] == self.exposure.package_name\n ):\n model = next(m for m in self.models if m.unique_id == model)\n if model.config.materialized in self.materializations_to_include:\n non_table_upstream_dependencies.append(model.name)\n\n assert not non_table_upstream_dependencies, f\"`{self.exposure.name}` is based on a model that is not a table: {non_table_upstream_dependencies}.\"\n
"},{"location":"checks/manifest/check_lineage/","title":"Manifest Checks: Lineage","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_lineage/#manifest.check_lineage.CheckLineagePermittedUpstreamModels","title":"CheckLineagePermittedUpstreamModels
","text":"Upstream models must have a path that matches the provided upstream_path_pattern
.
Parameters:
Name Type Description Default upstream_path_pattern
str
Regexp pattern to match the upstream model(s) path.
required Receives at execution time:
Name Type Description manifest_obj
DbtBouncerManifest
The manifest object.
model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_lineage_permitted_upstream_models\n include: ^models/staging\n upstream_path_pattern: $^\n - name: check_lineage_permitted_upstream_models\n include: ^models/intermediate\n upstream_path_pattern: ^models/staging|^models/intermediate\n - name: check_lineage_permitted_upstream_models\n include: ^models/marts\n upstream_path_pattern: ^models/staging|^models/intermediate\n
Source code in src/dbt_bouncer/checks/manifest/check_lineage.py
class CheckLineagePermittedUpstreamModels(BaseCheck):\n \"\"\"Upstream models must have a path that matches the provided `upstream_path_pattern`.\n\n Parameters:\n upstream_path_pattern (str): Regexp pattern to match the upstream model(s) path.\n\n Receives:\n manifest_obj (DbtBouncerManifest): The manifest object.\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_lineage_permitted_upstream_models\n include: ^models/staging\n upstream_path_pattern: $^\n - name: check_lineage_permitted_upstream_models\n include: ^models/intermediate\n upstream_path_pattern: ^models/staging|^models/intermediate\n - name: check_lineage_permitted_upstream_models\n include: ^models/marts\n upstream_path_pattern: ^models/staging|^models/intermediate\n ```\n\n \"\"\"\n\n manifest_obj: \"DbtBouncerManifest\" = Field(default=None)\n model: \"DbtBouncerModelBase\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_lineage_permitted_upstream_models\"]\n upstream_path_pattern: str\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n upstream_models = [\n x\n for x in self.model.depends_on.nodes\n if x.split(\".\")[0] == \"model\"\n and x.split(\".\")[1] == self.manifest_obj.manifest.metadata.project_name\n ]\n not_permitted_upstream_models = [\n upstream_model\n for upstream_model in upstream_models\n if re.compile(self.upstream_path_pattern.strip()).match(\n clean_path_str(\n next(\n m for m in self.models if m.unique_id == upstream_model\n ).original_file_path\n ),\n )\n is None\n ]\n assert not not_permitted_upstream_models, f\"`{self.model.name}` references upstream models that are not permitted: {[m.split('.')[-1] for m in not_permitted_upstream_models]}.\"\n
"},{"location":"checks/manifest/check_lineage/#manifest.check_lineage.CheckLineageSeedCannotBeUsed","title":"CheckLineageSeedCannotBeUsed
","text":"Seed cannot be referenced in models with a path that matches the specified include
config.
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_lineage_seed_cannot_be_used\n include: ^models/intermediate|^models/marts\n
Source code in src/dbt_bouncer/checks/manifest/check_lineage.py
class CheckLineageSeedCannotBeUsed(BaseCheck):\n \"\"\"Seed cannot be referenced in models with a path that matches the specified `include` config.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_lineage_seed_cannot_be_used\n include: ^models/intermediate|^models/marts\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_lineage_seed_cannot_be_used\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert not [\n x for x in self.model.depends_on.nodes if x.split(\".\")[0] == \"seed\"\n ], f\"`{self.model.name}` references a seed even though this is not permitted.\"\n
"},{"location":"checks/manifest/check_lineage/#manifest.check_lineage.CheckLineageSourceCannotBeUsed","title":"CheckLineageSourceCannotBeUsed
","text":"Sources cannot be referenced in models with a path that matches the specified include
config.
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_lineage_source_cannot_be_used\n include: ^models/intermediate|^models/marts\n
Source code in src/dbt_bouncer/checks/manifest/check_lineage.py
class CheckLineageSourceCannotBeUsed(BaseCheck):\n \"\"\"Sources cannot be referenced in models with a path that matches the specified `include` config.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_lineage_source_cannot_be_used\n include: ^models/intermediate|^models/marts\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_lineage_source_cannot_be_used\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert not [\n x for x in self.model.depends_on.nodes if x.split(\".\")[0] == \"source\"\n ], f\"`{self.model.name}` references a source even though this is not permitted.\"\n
"},{"location":"checks/manifest/check_macros/","title":"Manifest Checks: Macros","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_macros/#manifest.check_macros.CheckMacroArgumentsDescriptionPopulated","title":"CheckMacroArgumentsDescriptionPopulated
","text":"Macro arguments must have a populated description.
Receives at execution time:
Name Type Description macro
Macros
The Macros object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_macro_arguments_description_populated\n
# Only \"common\" macros need to have their arguments populated\nmanifest_checks:\n - name: check_macro_arguments_description_populated\n include: ^macros/common\n
Source code in src/dbt_bouncer/checks/manifest/check_macros.py
class CheckMacroArgumentsDescriptionPopulated(BaseCheck):\n \"\"\"Macro arguments must have a populated description.\n\n Receives:\n macro (Macros): The Macros object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_macro_arguments_description_populated\n ```\n ```yaml\n # Only \"common\" macros need to have their arguments populated\n manifest_checks:\n - name: check_macro_arguments_description_populated\n include: ^macros/common\n ```\n\n \"\"\"\n\n macro: \"Macros\" = Field(default=None)\n name: Literal[\"check_macro_arguments_description_populated\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n environment = jinja2.Environment(autoescape=True, extensions=[TagExtension])\n ast = environment.parse(self.macro.macro_sql)\n\n if hasattr(ast.body[0], \"args\"):\n # Assume macro is a \"true\" macro\n macro_arguments = [a.name for a in ast.body[0].args]\n else:\n if \"materialization\" in [\n x.value.value\n for x in ast.body[0].nodes[0].kwargs # type: ignore[attr-defined]\n if isinstance(x.value, jinja2.nodes.Const)\n ]:\n # Materializations don't have arguments\n macro_arguments = []\n else:\n # Macro is a test\n test_macro = next(\n x\n for x in ast.body\n if not isinstance(x.nodes[0], jinja2.nodes.Call) # type: ignore[attr-defined]\n )\n macro_arguments = [\n x.name\n for x in test_macro.nodes # type: ignore[attr-defined]\n if isinstance(x, jinja2.nodes.Name)\n ]\n\n # macro_arguments: List of args parsed from macro SQL\n # macro.arguments: List of args manually added to the properties file\n\n non_complying_args = []\n for arg in macro_arguments:\n macro_doc_raw = [x for x in self.macro.arguments if x.name == arg]\n if macro_doc_raw == [] or (\n arg not in [x.name for x in self.macro.arguments]\n or len(macro_doc_raw[0].description.strip()) <= 4\n ):\n non_complying_args.append(arg)\n\n assert (\n non_complying_args == []\n ), f\"Macro `{self.macro.name}` does not have a populated description for the following argument(s): {non_complying_args}.\"\n
"},{"location":"checks/manifest/check_macros/#manifest.check_macros.CheckMacroCodeDoesNotContainRegexpPattern","title":"CheckMacroCodeDoesNotContainRegexpPattern
","text":"The raw code for a macro must not match the specified regexp pattern.
Parameters:
Name Type Description Default regexp_pattern
str
The regexp pattern that should not be matched by the macro code.
required Receives at execution time:
Name Type Description macro
Macros
The Macros object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n # Prefer `coalesce` over `ifnull`: https://docs.sqlfluff.com/en/stable/rules.html#sqlfluff.rules.sphinx.Rule_CV02\n - name: check_macro_code_does_not_contain_regexp_pattern\n regexp_pattern: .*[i][f][n][u][l][l].*\n
Source code in src/dbt_bouncer/checks/manifest/check_macros.py
class CheckMacroCodeDoesNotContainRegexpPattern(BaseCheck):\n \"\"\"The raw code for a macro must not match the specified regexp pattern.\n\n Parameters:\n regexp_pattern (str): The regexp pattern that should not be matched by the macro code.\n\n Receives:\n macro (Macros): The Macros object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n # Prefer `coalesce` over `ifnull`: https://docs.sqlfluff.com/en/stable/rules.html#sqlfluff.rules.sphinx.Rule_CV02\n - name: check_macro_code_does_not_contain_regexp_pattern\n regexp_pattern: .*[i][f][n][u][l][l].*\n ```\n\n \"\"\"\n\n macro: \"Macros\" = Field(default=None)\n name: Literal[\"check_macro_code_does_not_contain_regexp_pattern\"]\n regexp_pattern: str\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n re.compile(self.regexp_pattern.strip(), flags=re.DOTALL).match(\n self.macro.macro_sql\n )\n is None\n ), f\"Macro `{self.macro.name}` contains a banned string: `{self.regexp_pattern.strip()}`.\"\n
"},{"location":"checks/manifest/check_macros/#manifest.check_macros.CheckMacroDescriptionPopulated","title":"CheckMacroDescriptionPopulated
","text":"Macros must have a populated description.
Receives at execution time:
Name Type Description macro
Macros
The Macros object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_macro_description_populated\n
# Only \"common\" macros need to have a populated description\nmanifest_checks:\n - name: check_macro_description_populated\n include: ^macros/common\n
Source code in src/dbt_bouncer/checks/manifest/check_macros.py
class CheckMacroDescriptionPopulated(BaseCheck):\n \"\"\"Macros must have a populated description.\n\n Receives:\n macro (Macros): The Macros object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_macro_description_populated\n ```\n ```yaml\n # Only \"common\" macros need to have a populated description\n manifest_checks:\n - name: check_macro_description_populated\n include: ^macros/common\n ```\n\n \"\"\"\n\n macro: \"Macros\" = Field(default=None)\n name: Literal[\"check_macro_description_populated\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n len(self.macro.description.strip()) > 4\n ), f\"Macro `{self.macro.name}` does not have a populated description.\"\n
"},{"location":"checks/manifest/check_macros/#manifest.check_macros.CheckMacroMaxNumberOfLines","title":"CheckMacroMaxNumberOfLines
","text":"Macros may not have more than the specified number of lines.
Parameters:
Name Type Description Default max_number_of_lines
int
The maximum number of permitted lines.
required Receives at execution time:
Name Type Description macro
Macros
The Macros object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_macro_max_number_of_lines\n
manifest_checks:\n - name: check_macro_max_number_of_lines\n max_number_of_lines: 100\n
Source code in src/dbt_bouncer/checks/manifest/check_macros.py
class CheckMacroMaxNumberOfLines(BaseCheck):\n \"\"\"Macros may not have more than the specified number of lines.\n\n Parameters:\n max_number_of_lines (int): The maximum number of permitted lines.\n\n Receives:\n macro (Macros): The Macros object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_macro_max_number_of_lines\n ```\n ```yaml\n manifest_checks:\n - name: check_macro_max_number_of_lines\n max_number_of_lines: 100\n ```\n\n \"\"\"\n\n macro: \"Macros\" = Field(default=None)\n name: Literal[\"check_macro_max_number_of_lines\"]\n max_number_of_lines: int = Field(default=50)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n actual_number_of_lines = self.macro.macro_sql.count(\"\\n\") + 1\n\n assert (\n actual_number_of_lines <= self.max_number_of_lines\n ), f\"Macro `{self.macro.name}` has {actual_number_of_lines} lines, this is more than the maximum permitted number of lines ({self.max_number_of_lines}).\"\n
"},{"location":"checks/manifest/check_macros/#manifest.check_macros.CheckMacroNameMatchesFileName","title":"CheckMacroNameMatchesFileName
","text":"Macros names must be the same as the file they are contained in.
Generic tests are also macros, however to document these tests the \"name\" value must be preceded with \"test_\".
Receives at execution time:
Name Type Description macro
Macros
The Macros object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_macro_name_matches_file_name\n
Source code in src/dbt_bouncer/checks/manifest/check_macros.py
class CheckMacroNameMatchesFileName(BaseCheck):\n \"\"\"Macros names must be the same as the file they are contained in.\n\n Generic tests are also macros, however to document these tests the \"name\" value must be preceded with \"test_\".\n\n Receives:\n macro (Macros): The Macros object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_macro_name_matches_file_name\n ```\n\n \"\"\"\n\n macro: \"Macros\" = Field(default=None)\n name: Literal[\"check_macro_name_matches_file_name\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if self.macro.name.startswith(\"test_\"):\n assert (\n self.macro.name[5:]\n == clean_path_str(self.macro.original_file_path)\n .split(\"/\")[-1]\n .split(\".\")[0]\n ), f\"Macro `{self.macro.unique_id}` is not in a file named `{self.macro.name[5:]}.sql`.\"\n else:\n assert (\n self.macro.name\n == clean_path_str(self.macro.original_file_path)\n .split(\"/\")[-1]\n .split(\".\")[0]\n ), f\"Macro `{self.macro.name}` is not in a file of the same name.\"\n
"},{"location":"checks/manifest/check_macros/#manifest.check_macros.CheckMacroPropertyFileLocation","title":"CheckMacroPropertyFileLocation
","text":"Macro properties files must follow the guidance provided by dbt here.
Receives at execution time:
Name Type Description macro
Macros
The Macros object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_macro_property_file_location\n
Source code in src/dbt_bouncer/checks/manifest/check_macros.py
class CheckMacroPropertyFileLocation(BaseCheck):\n \"\"\"Macro properties files must follow the guidance provided by dbt [here](https://docs.getdbt.com/best-practices/how-we-structure/5-the-rest-of-the-project#how-we-use-the-other-folders).\n\n Receives:\n macro (Macros): The Macros object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the macro path. Macro paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_macro_property_file_location\n ```\n\n \"\"\"\n\n macro: \"Macros\" = Field(default=None)\n name: Literal[\"check_macro_property_file_location\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n expected_substr = \"_\".join(\n clean_path_str(self.macro.original_file_path)[6:].split(\"/\")[:-1]\n )\n\n assert (\n clean_path_str(self.macro.patch_path) is not None\n ), f\"Macro `{self.macro.name}` is not defined in a `.yml` properties file.\"\n properties_yml_name = clean_path_str(self.macro.patch_path).split(\"/\")[-1]\n\n if clean_path_str(self.macro.original_file_path).startswith(\n \"tests/\",\n ): # Do not check generic tests (which are also macros)\n pass\n elif expected_substr == \"\": # i.e. macro in ./macros\n assert (\n properties_yml_name == \"_macros.yml\"\n ), f\"The properties file for `{self.macro.name}` (`{properties_yml_name}`) should be `_macros.yml`.\"\n else:\n assert properties_yml_name.startswith(\n \"_\",\n ), f\"The properties file for `{self.macro.name}` (`{properties_yml_name}`) does not start with an underscore.\"\n assert (\n expected_substr in properties_yml_name\n ), f\"The properties file for `{self.macro.name}` (`{properties_yml_name}`) does not contain the expected substring (`{expected_substr}`).\"\n assert properties_yml_name.endswith(\n \"__macros.yml\",\n ), f\"The properties file for `{self.macro.name.name}` (`{properties_yml_name}`) does not end with `__macros.yml`.\"\n
"},{"location":"checks/manifest/check_metadata/","title":"Manifest Checks: Metadata","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_metadata/#manifest.check_metadata.CheckProjectName","title":"CheckProjectName
","text":"Enforce that the name of the dbt project matches a supplied regex. Generally used to enforce that project names conform to something like company_<DOMAIN>
.
Parameters:
Name Type Description Default project_name_pattern
str
Regex pattern to match the project name.
required Receives at execution time:
Name Type Description manifest_obj
DbtBouncerManifest
The manifest object.
Other Parameters (passed via config file):
Name Type Description severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_project_name\n project_name_pattern: ^awesome_company_\n
Source code in src/dbt_bouncer/checks/manifest/check_metadata.py
class CheckProjectName(BaseModel):\n \"\"\"Enforce that the name of the dbt project matches a supplied regex. Generally used to enforce that project names conform to something like `company_<DOMAIN>`.\n\n Parameters:\n project_name_pattern (str): Regex pattern to match the project name.\n\n Receives:\n manifest_obj (DbtBouncerManifest): The manifest object.\n\n Other Parameters:\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_project_name\n project_name_pattern: ^awesome_company_\n ```\n\n \"\"\"\n\n model_config = ConfigDict(extra=\"forbid\")\n\n index: Optional[int] = Field(\n default=None,\n description=\"Index to uniquely identify the check, calculated at runtime.\",\n )\n manifest_obj: \"DbtBouncerManifest\" = Field(default=None)\n name: Literal[\"check_project_name\"]\n project_name_pattern: str\n severity: Optional[Literal[\"error\", \"warn\"]] = Field(\n default=\"error\",\n description=\"Severity of the check, one of 'error' or 'warn'.\",\n )\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n re.compile(self.project_name_pattern.strip()).match(\n self.manifest_obj.manifest.metadata.project_name,\n )\n is not None\n ), f\"Project name (`{self.manifest_obj.manifest.metadata.project_name}`) does not conform to the supplied regex `({self.project_name_pattern.strip()})`.\"\n
"},{"location":"checks/manifest/check_models/","title":"Manifest Checks: Models","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelAccess","title":"CheckModelAccess
","text":"Models must have the specified access attribute. Requires dbt 1.7+.
Parameters:
Name Type Description Default access
Literal['private', 'protected', 'public']
The access level to check for.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n # Align with dbt best practices that marts should be `public`, everything else should be `protected`\n - name: check_model_access\n access: protected\n include: ^models/intermediate\n - name: check_model_access\n access: public\n include: ^models/marts\n - name: check_model_access\n access: protected\n include: ^models/staging\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelAccess(BaseCheck):\n \"\"\"Models must have the specified access attribute. Requires dbt 1.7+.\n\n Parameters:\n access (Literal[\"private\", \"protected\", \"public\"]): The access level to check for.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n # Align with dbt best practices that marts should be `public`, everything else should be `protected`\n - name: check_model_access\n access: protected\n include: ^models/intermediate\n - name: check_model_access\n access: public\n include: ^models/marts\n - name: check_model_access\n access: protected\n include: ^models/staging\n ```\n\n \"\"\"\n\n access: Literal[\"private\", \"protected\", \"public\"]\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_access\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n self.model.access.value == self.access\n ), f\"`{self.model.name}` has `{self.model.access.value}` access, it should have access `{self.access}`.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelContractsEnforcedForPublicModel","title":"CheckModelContractsEnforcedForPublicModel
","text":"Public models must have contracts enforced.
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_contract_enforced_for_public_model\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelContractsEnforcedForPublicModel(BaseCheck):\n \"\"\"Public models must have contracts enforced.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_contract_enforced_for_public_model\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_contract_enforced_for_public_model\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if self.model.access.value == \"public\":\n assert (\n self.model.contract.enforced is True\n ), f\"`{self.model.name}` is a public model but does not have contracts enforced.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelDescriptionPopulated","title":"CheckModelDescriptionPopulated
","text":"Models must have a populated description.
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_description_populated\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelDescriptionPopulated(BaseCheck):\n \"\"\"Models must have a populated description.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_description_populated\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_description_populated\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n len(self.model.description.strip()) > 4\n ), f\"`{self.model.name}` does not have a populated description.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelsDocumentationCoverage","title":"CheckModelsDocumentationCoverage
","text":"Set the minimum percentage of models that have a populated description.
Parameters:
Name Type Description Default min_model_documentation_coverage_pct
float
The minimum percentage of models that must have a populated description.
required Receives at execution time:
Name Type Description models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_description_populated\n min_model_documentation_coverage_pct: 90\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelsDocumentationCoverage(BaseModel):\n \"\"\"Set the minimum percentage of models that have a populated description.\n\n Parameters:\n min_model_documentation_coverage_pct (float): The minimum percentage of models that must have a populated description.\n\n Receives:\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_description_populated\n min_model_documentation_coverage_pct: 90\n ```\n\n \"\"\"\n\n model_config = ConfigDict(extra=\"forbid\")\n\n index: Optional[int] = Field(\n default=None,\n description=\"Index to uniquely identify the check, calculated at runtime.\",\n )\n min_model_documentation_coverage_pct: int = Field(\n default=100,\n ge=0,\n le=100,\n )\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_model_documentation_coverage\"]\n severity: Optional[Literal[\"error\", \"warn\"]] = Field(\n default=\"error\",\n description=\"Severity of the check, one of 'error' or 'warn'.\",\n )\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_models = len(self.models)\n models_with_description = []\n for model in self.models:\n if len(model.description.strip()) > 4:\n models_with_description.append(model.unique_id)\n\n num_models_with_descriptions = len(models_with_description)\n model_description_coverage_pct = (\n num_models_with_descriptions / num_models\n ) * 100\n\n assert (\n model_description_coverage_pct >= self.min_model_documentation_coverage_pct\n ), f\"Only {model_description_coverage_pct}% of models have a populated description, this is less than the permitted minimum of {self.min_model_documentation_coverage_pct}%.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelDocumentedInSameDirectory","title":"CheckModelDocumentedInSameDirectory
","text":"Models must be documented in the same directory where they are defined (i.e. .yml
and .sql
files are in the same directory).
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_documented_in_same_directory\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelDocumentedInSameDirectory(BaseCheck):\n \"\"\"Models must be documented in the same directory where they are defined (i.e. `.yml` and `.sql` files are in the same directory).\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_documented_in_same_directory\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_documented_in_same_directory\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n model_sql_dir = clean_path_str(self.model.original_file_path).split(\"/\")[:-1]\n assert ( # noqa: PT018\n hasattr(self.model, \"patch_path\")\n and clean_path_str(self.model.patch_path) is not None\n ), f\"`{self.model.name}` is not documented.\"\n\n model_doc_dir = clean_path_str(\n self.model.patch_path[\n clean_path_str(self.model.patch_path).find(\"models\") :\n ]\n ).split(\"/\")[:-1]\n\n assert (\n model_doc_dir == model_sql_dir\n ), f\"`{self.model.name}` is documented in a different directory to the `.sql` file: `{'/'.join(model_doc_dir)}` vs `{'/'.join(model_sql_dir)}`.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelCodeDoesNotContainRegexpPattern","title":"CheckModelCodeDoesNotContainRegexpPattern
","text":"The raw code for a model must not match the specified regexp pattern.
Parameters:
Name Type Description Default regexp_pattern
str
The regexp pattern that should not be matched by the model code.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n # Prefer `coalesce` over `ifnull`: https://docs.sqlfluff.com/en/stable/rules.html#sqlfluff.rules.sphinx.Rule_CV02\n - name: check_model_code_does_not_contain_regexp_pattern\n regexp_pattern: .*[i][f][n][u][l][l].*\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelCodeDoesNotContainRegexpPattern(BaseCheck):\n \"\"\"The raw code for a model must not match the specified regexp pattern.\n\n Parameters:\n regexp_pattern (str): The regexp pattern that should not be matched by the model code.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n # Prefer `coalesce` over `ifnull`: https://docs.sqlfluff.com/en/stable/rules.html#sqlfluff.rules.sphinx.Rule_CV02\n - name: check_model_code_does_not_contain_regexp_pattern\n regexp_pattern: .*[i][f][n][u][l][l].*\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_code_does_not_contain_regexp_pattern\"]\n regexp_pattern: str\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n re.compile(self.regexp_pattern.strip(), flags=re.DOTALL).match(\n self.model.raw_code\n )\n is None\n ), f\"`{self.model.name}` contains a banned string: `{self.regexp_pattern.strip()}`.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelDependsOnMultipleSources","title":"CheckModelDependsOnMultipleSources
","text":"Models cannot reference more than one source.
Parameters:
Name Type Description Default model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_depends_on_multiple_sources\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelDependsOnMultipleSources(BaseCheck):\n \"\"\"Models cannot reference more than one source.\n\n Parameters:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_depends_on_multiple_sources\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_depends_on_multiple_sources\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_reffed_sources = sum(\n x.split(\".\")[0] == \"source\" for x in self.model.depends_on.nodes\n )\n assert (\n num_reffed_sources <= 1\n ), f\"`{self.model.name}` references more than one source.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelDirectories","title":"CheckModelDirectories
","text":"Only specified sub-directories are permitted.
Parameters:
Name Type Description Default include
str
Regex pattern to the directory to check.
required permitted_sub_directories
List[str]
List of permitted sub-directories.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Example(s):
manifest_checks:\n- name: check_model_directories\n include: models\n permitted_sub_directories:\n - intermediate\n - marts\n - staging\n
# Restrict sub-directories within `./models/staging`\n- name: check_model_directories\n include: ^models/staging\n permitted_sub_directories:\n - crm\n - payments\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelDirectories(BaseCheck):\n \"\"\"Only specified sub-directories are permitted.\n\n Parameters:\n include (str): Regex pattern to the directory to check.\n permitted_sub_directories (List[str]): List of permitted sub-directories.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_directories\n include: models\n permitted_sub_directories:\n - intermediate\n - marts\n - staging\n ```\n ```yaml\n # Restrict sub-directories within `./models/staging`\n - name: check_model_directories\n include: ^models/staging\n permitted_sub_directories:\n - crm\n - payments\n ```\n\n \"\"\"\n\n include: str\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_directories\"]\n permitted_sub_directories: List[str]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n matched_path = re.compile(self.include.strip()).match(\n clean_path_str(self.model.original_file_path)\n )\n path_after_match = clean_path_str(self.model.original_file_path)[\n matched_path.end() + 1 :\n ]\n\n assert (\n path_after_match.split(\"/\")[0] in self.permitted_sub_directories\n ), f\"`{self.model.name}` is located in `{self.model.original_file_path.split('/')[1]}`, this is not a valid sub-directory.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelHasContractsEnforced","title":"CheckModelHasContractsEnforced
","text":"Model must have contracts enforced.
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_has_contracts_enforced\n include: ^models/marts\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelHasContractsEnforced(BaseCheck):\n \"\"\"Model must have contracts enforced.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_has_contracts_enforced\n include: ^models/marts\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_has_contracts_enforced\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n self.model.contract.enforced is True\n ), f\"`{self.model.name}` does not have contracts enforced.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelHasMetaKeys","title":"CheckModelHasMetaKeys
","text":"The meta
config for models must have the specified keys.
Parameters:
Name Type Description Default keys
NestedDict
A list (that may contain sub-lists) of required keys.
required model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_has_meta_keys\n keys:\n - maturity\n - owner\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelHasMetaKeys(BaseCheck):\n \"\"\"The `meta` config for models must have the specified keys.\n\n Parameters:\n keys (NestedDict): A list (that may contain sub-lists) of required keys.\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_has_meta_keys\n keys:\n - maturity\n - owner\n ```\n\n \"\"\"\n\n keys: NestedDict\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_has_meta_keys\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n missing_keys = find_missing_meta_keys(\n meta_config=self.model.meta,\n required_keys=self.keys.model_dump(),\n )\n assert (\n missing_keys == []\n ), f\"`{self.model.name}` is missing the following keys from the `meta` config: {[x.replace('>>', '') for x in missing_keys]}\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelHasNoUpstreamDependencies","title":"CheckModelHasNoUpstreamDependencies
","text":"Identify if models have no upstream dependencies as this likely indicates hard-coded tables references.
Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_has_no_upstream_dependencies\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelHasNoUpstreamDependencies(BaseCheck):\n \"\"\"Identify if models have no upstream dependencies as this likely indicates hard-coded tables references.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_has_no_upstream_dependencies\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_has_no_upstream_dependencies\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n len(self.model.depends_on.nodes) > 0\n ), f\"`{self.model.name}` has no upstream dependencies, this likely indicates hard-coded tables references.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelHasTags","title":"CheckModelHasTags
","text":"Models must have the specified tags.
Parameters:
Name Type Description Default model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
required tags
List[str]
List of tags to check for.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_has_tags\n tags:\n - tag_1\n - tag_2\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelHasTags(BaseCheck):\n \"\"\"Models must have the specified tags.\n\n Parameters:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n tags (List[str]): List of tags to check for.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_has_tags\n tags:\n - tag_1\n - tag_2\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_has_tags\"]\n tags: List[str]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n missing_tags = [tag for tag in self.tags if tag not in self.model.tags]\n assert (\n not missing_tags\n ), f\"`{self.model.name}` is missing required tags: {missing_tags}.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelHasUniqueTest","title":"CheckModelHasUniqueTest
","text":"Models must have a test for uniqueness of a column.
Parameters:
Name Type Description Default accepted_uniqueness_tests
Optional[List[str]]
List of tests that are accepted as uniqueness tests.
required model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
required tests
List[DbtBouncerTestBase]
List of DbtBouncerTestBase objects parsed from manifest.json
.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_has_unique_test\n include: ^models/marts\n
manifest_checks:\n# Example of allowing a custom uniqueness test\n - name: check_model_has_unique_test\n accepted_uniqueness_tests:\n - expect_compound_columns_to_be_unique\n - my_custom_uniqueness_test\n - unique\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelHasUniqueTest(BaseCheck):\n \"\"\"Models must have a test for uniqueness of a column.\n\n Parameters:\n accepted_uniqueness_tests (Optional[List[str]]): List of tests that are accepted as uniqueness tests.\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n tests (List[DbtBouncerTestBase]): List of DbtBouncerTestBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_has_unique_test\n include: ^models/marts\n ```\n ```yaml\n manifest_checks:\n # Example of allowing a custom uniqueness test\n - name: check_model_has_unique_test\n accepted_uniqueness_tests:\n - expect_compound_columns_to_be_unique\n - my_custom_uniqueness_test\n - unique\n ```\n\n \"\"\"\n\n accepted_uniqueness_tests: Optional[List[str]] = Field(\n default=[\n \"expect_compound_columns_to_be_unique\",\n \"dbt_utils.unique_combination_of_columns\",\n \"unique\",\n ],\n )\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_has_unique_test\"]\n tests: List[\"DbtBouncerTestBase\"] = Field(default=[])\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_unique_tests = sum(\n test.attached_node == self.model.unique_id\n and test.test_metadata.name in self.accepted_uniqueness_tests # type: ignore[operator]\n for test in self.tests\n if hasattr(test, \"test_metadata\")\n )\n assert (\n num_unique_tests >= 1\n ), f\"`{self.model.name}` does not have a test for uniqueness of a column.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelHasUnitTests","title":"CheckModelHasUnitTests
","text":"Models must have more than the specified number of unit tests.
Parameters:
Name Type Description Default min_number_of_unit_tests
Optional[int]
The minimum number of unit tests that a model must have.
required Receives at execution time:
Name Type Description manifest_obj
DbtBouncerManifest
The DbtBouncerManifest object parsed from manifest.json
.
model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
unit_tests
List[UnitTests]
List of UnitTests objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Warning
This check is only supported for dbt 1.8.0 and above.
Example(s):
manifest_checks:\n - name: check_model_has_unit_tests\n include: ^models/marts\n
manifest_checks:\n - name: check_model_has_unit_tests\n min_number_of_unit_tests: 2\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelHasUnitTests(BaseCheck):\n \"\"\"Models must have more than the specified number of unit tests.\n\n Parameters:\n min_number_of_unit_tests (Optional[int]): The minimum number of unit tests that a model must have.\n\n Receives:\n manifest_obj (DbtBouncerManifest): The DbtBouncerManifest object parsed from `manifest.json`.\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n unit_tests (List[UnitTests]): List of UnitTests objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n !!! warning\n\n This check is only supported for dbt 1.8.0 and above.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_has_unit_tests\n include: ^models/marts\n ```\n ```yaml\n manifest_checks:\n - name: check_model_has_unit_tests\n min_number_of_unit_tests: 2\n ```\n\n \"\"\"\n\n manifest_obj: \"DbtBouncerManifest\" = Field(default=None)\n min_number_of_unit_tests: int = Field(default=1)\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_has_unit_tests\"]\n unit_tests: List[\"UnitTests\"] = Field(default=[])\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if (\n semver.Version.parse(self.manifest_obj.manifest.metadata.dbt_version)\n >= \"1.8.0\"\n ):\n num_unit_tests = len(\n [\n t.unique_id\n for t in self.unit_tests\n if t.depends_on.nodes[0] == self.model.unique_id\n ],\n )\n assert (\n num_unit_tests >= self.min_number_of_unit_tests\n ), f\"`{self.model.name}` has {num_unit_tests} unit tests, this is less than the minimum of {self.min_number_of_unit_tests}.\"\n else:\n logging.warning(\n \"The `check_model_has_unit_tests` check is only supported for dbt 1.8.0 and above.\",\n )\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelMaxChainedViews","title":"CheckModelMaxChainedViews
","text":"Models cannot have more than the specified number of upstream dependents that are not tables.
Parameters:
Name Type Description Default materializations_to_include
Optional[List[str]]
List of materializations to include in the check.
required max_chained_views
Optional[int]
The maximum number of upstream dependents that are not tables.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_max_chained_views\n
manifest_checks:\n - name: check_model_max_chained_views\n materializations_to_include:\n - ephemeral\n - my_custom_materialization\n - view\n max_chained_views: 5\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelMaxChainedViews(BaseCheck):\n \"\"\"Models cannot have more than the specified number of upstream dependents that are not tables.\n\n Parameters:\n materializations_to_include (Optional[List[str]]): List of materializations to include in the check.\n max_chained_views (Optional[int]): The maximum number of upstream dependents that are not tables.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_max_chained_views\n ```\n ```yaml\n manifest_checks:\n - name: check_model_max_chained_views\n materializations_to_include:\n - ephemeral\n - my_custom_materialization\n - view\n max_chained_views: 5\n ```\n\n \"\"\"\n\n manifest_obj: \"DbtBouncerManifest\" = Field(default=None)\n materializations_to_include: List[str] = Field(\n default=[\"ephemeral\", \"view\"],\n )\n max_chained_views: int = Field(\n default=3,\n )\n model: \"DbtBouncerModelBase\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_model_max_chained_views\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n\n def return_upstream_view_models(\n materializations,\n max_chained_views,\n models,\n model_unique_ids_to_check,\n package_name,\n depth=0,\n ):\n \"\"\"Recursive function to return model unique_id's of upstream models that are views. Depth of recursion can be specified. If no models meet the criteria then an empty list is returned.\n\n Returns\n -\n List[str]: List of model unique_id's of upstream models that are views.\n\n \"\"\"\n if depth == max_chained_views or model_unique_ids_to_check == []:\n return model_unique_ids_to_check\n\n relevant_upstream_models = []\n for model in model_unique_ids_to_check:\n upstream_nodes = list(\n next(m2 for m2 in models if m2.unique_id == model).depends_on.nodes,\n )\n if upstream_nodes != []:\n upstream_models = [\n m\n for m in upstream_nodes\n if m.split(\".\")[0] == \"model\"\n and m.split(\".\")[1] == package_name\n ]\n for i in upstream_models:\n if (\n next(\n m for m in models if m.unique_id == i\n ).config.materialized\n in materializations\n ):\n relevant_upstream_models.append(i)\n\n depth += 1\n return return_upstream_view_models(\n materializations=materializations,\n max_chained_views=max_chained_views,\n models=models,\n model_unique_ids_to_check=relevant_upstream_models,\n package_name=package_name,\n depth=depth,\n )\n\n assert (\n len(\n return_upstream_view_models(\n materializations=self.materializations_to_include,\n max_chained_views=self.max_chained_views,\n models=self.models,\n model_unique_ids_to_check=[self.model.unique_id],\n package_name=self.manifest_obj.manifest.metadata.project_name,\n ),\n )\n == 0\n ), f\"`{self.model.name}` has more than {self.max_chained_views} upstream dependents that are not tables.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelMaxFanout","title":"CheckModelMaxFanout
","text":"Models cannot have more than the specified number of downstream models.
Parameters:
Name Type Description Default max_downstream_models
Optional[int]
The maximum number of permitted downstream models.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_max_fanout\n max_downstream_models: 2\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelMaxFanout(BaseCheck):\n \"\"\"Models cannot have more than the specified number of downstream models.\n\n Parameters:\n max_downstream_models (Optional[int]): The maximum number of permitted downstream models.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_max_fanout\n max_downstream_models: 2\n ```\n\n \"\"\"\n\n max_downstream_models: int = Field(default=3)\n model: \"DbtBouncerModelBase\" = Field(default=None)\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_model_max_fanout\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_downstream_models = sum(\n self.model.unique_id in m.depends_on.nodes for m in self.models\n )\n\n assert (\n num_downstream_models <= self.max_downstream_models\n ), f\"`{self.model.name}` has {num_downstream_models} downstream models, which is more than the permitted maximum of {self.max_downstream_models}.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelMaxNumberOfLines","title":"CheckModelMaxNumberOfLines
","text":"Models may not have more than the specified number of lines.
Parameters:
Name Type Description Default max_number_of_lines
int
The maximum number of permitted lines.
required model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_max_number_of_lines\n
manifest_checks:\n - name: check_model_max_number_of_lines\n max_number_of_lines: 150\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelMaxNumberOfLines(BaseCheck):\n \"\"\"Models may not have more than the specified number of lines.\n\n Parameters:\n max_number_of_lines (int): The maximum number of permitted lines.\n\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_max_number_of_lines\n ```\n ```yaml\n manifest_checks:\n - name: check_model_max_number_of_lines\n max_number_of_lines: 150\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_max_number_of_lines\"]\n max_number_of_lines: int = Field(default=100)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n actual_number_of_lines = self.model.raw_code.count(\"\\n\") + 1\n\n assert (\n actual_number_of_lines <= self.max_number_of_lines\n ), f\"`{self.model.name}` has {actual_number_of_lines} lines, this is more than the maximum permitted number of lines ({self.max_number_of_lines}).\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelMaxUpstreamDependencies","title":"CheckModelMaxUpstreamDependencies
","text":"Limit the number of upstream dependencies a model has.
Parameters:
Name Type Description Default max_upstream_macros
Optional[int]
The maximum number of permitted upstream macros.
required max_upstream_models
Optional[int]
The maximum number of permitted upstream models.
required max_upstream_sources
Optional[int]
The maximum number of permitted upstream sources.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_max_upstream_dependencies\n max_upstream_models: 3\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelMaxUpstreamDependencies(BaseCheck):\n \"\"\"Limit the number of upstream dependencies a model has.\n\n Parameters:\n max_upstream_macros (Optional[int]): The maximum number of permitted upstream macros.\n max_upstream_models (Optional[int]): The maximum number of permitted upstream models.\n max_upstream_sources (Optional[int]): The maximum number of permitted upstream sources.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_max_upstream_dependencies\n max_upstream_models: 3\n ```\n\n \"\"\"\n\n max_upstream_macros: int = Field(\n default=5,\n )\n max_upstream_models: int = Field(\n default=5,\n )\n max_upstream_sources: int = Field(\n default=1,\n )\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_max_upstream_dependencies\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_upstream_macros = len(list(self.model.depends_on.macros))\n num_upstream_models = len(\n [m for m in self.model.depends_on.nodes if m.split(\".\")[0] == \"model\"],\n )\n num_upstream_sources = len(\n [m for m in self.model.depends_on.nodes if m.split(\".\")[0] == \"source\"],\n )\n\n assert (\n num_upstream_macros <= self.max_upstream_macros\n ), f\"`{self.model.name}` has {num_upstream_macros} upstream macros, which is more than the permitted maximum of {self.max_upstream_macros}.\"\n assert (\n num_upstream_models <= self.max_upstream_models\n ), f\"`{self.model.name}` has {num_upstream_models} upstream models, which is more than the permitted maximum of {self.max_upstream_models}.\"\n assert (\n num_upstream_sources <= self.max_upstream_sources\n ), f\"`{self.model.name}` has {num_upstream_sources} upstream sources, which is more than the permitted maximum of {self.max_upstream_sources}.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelNames","title":"CheckModelNames
","text":"Models must have a name that matches the supplied regex.
Parameters:
Name Type Description Default model_name_pattern
str
Regexp the model name must match.
required Receives at execution time:
Name Type Description model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_names\n include: ^models/intermediate\n model_name_pattern: ^int_\n - name: check_model_names\n include: ^models/staging\n model_name_pattern: ^stg_\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelNames(BaseCheck):\n \"\"\"Models must have a name that matches the supplied regex.\n\n Parameters:\n model_name_pattern (str): Regexp the model name must match.\n\n Receives:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_names\n include: ^models/intermediate\n model_name_pattern: ^int_\n - name: check_model_names\n include: ^models/staging\n model_name_pattern: ^stg_\n ```\n\n \"\"\"\n\n model_config = ConfigDict(extra=\"forbid\", protected_namespaces=())\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_names\"]\n model_name_pattern: str\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n re.compile(self.model_name_pattern.strip()).match(self.model.name)\n is not None\n ), f\"`{self.model.name}` does not match the supplied regex `{self.model_name_pattern.strip()})`.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelPropertyFileLocation","title":"CheckModelPropertyFileLocation
","text":"Model properties files must follow the guidance provided by dbt here.
Parameters:
Name Type Description Default model
DbtBouncerModelBase
The DbtBouncerModelBase object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the model path. Model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the model path. Only model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_property_file_location\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelPropertyFileLocation(BaseCheck):\n \"\"\"Model properties files must follow the guidance provided by dbt [here](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview).\n\n Parameters:\n model (DbtBouncerModelBase): The DbtBouncerModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the model path. Model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the model path. Only model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_property_file_location\n ```\n\n \"\"\"\n\n model: \"DbtBouncerModelBase\" = Field(default=None)\n name: Literal[\"check_model_property_file_location\"]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert ( # noqa: PT018\n hasattr(self.model, \"patch_path\")\n and clean_path_str(self.model.patch_path) is not None\n ), f\"`{self.model.name}` is not documented.\"\n\n expected_substr = (\n \"_\".join(clean_path_str(self.model.original_file_path).split(\"/\")[1:-1])\n .replace(\"staging\", \"stg\")\n .replace(\"intermediate\", \"int\")\n .replace(\"marts\", \"\")\n )\n properties_yml_name = clean_path_str(self.model.patch_path).split(\"/\")[-1]\n\n assert properties_yml_name.startswith(\n \"_\",\n ), f\"The properties file for `{self.model.name}` (`{properties_yml_name}`) does not start with an underscore.\"\n assert (\n expected_substr in properties_yml_name\n ), f\"The properties file for `{self.model.name}` (`{properties_yml_name}`) does not contain the expected substring (`{expected_substr}`).\"\n assert properties_yml_name.endswith(\n \"__models.yml\",\n ), f\"The properties file for `{self.model.name}` (`{properties_yml_name}`) does not end with `__models.yml`.\"\n
"},{"location":"checks/manifest/check_models/#manifest.check_models.CheckModelsTestCoverage","title":"CheckModelsTestCoverage
","text":"Set the minimum percentage of models that have at least one test.
Parameters:
Name Type Description Default min_model_test_coverage_pct
float
The minimum percentage of models that must have at least one test.
required models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
required tests
List[DbtBouncerTestBase]
List of DbtBouncerTestBase objects parsed from manifest.json
.
required Other Parameters (passed via config file):
Name Type Description severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_model_test_coverage\n min_model_test_coverage_pct: 90\n
Source code in src/dbt_bouncer/checks/manifest/check_models.py
class CheckModelsTestCoverage(BaseModel):\n \"\"\"Set the minimum percentage of models that have at least one test.\n\n Parameters:\n min_model_test_coverage_pct (float): The minimum percentage of models that must have at least one test.\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n tests (List[DbtBouncerTestBase]): List of DbtBouncerTestBase objects parsed from `manifest.json`.\n\n Other Parameters:\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_model_test_coverage\n min_model_test_coverage_pct: 90\n ```\n\n \"\"\"\n\n model_config = ConfigDict(extra=\"forbid\")\n\n index: Optional[int] = Field(\n default=None,\n description=\"Index to uniquely identify the check, calculated at runtime.\",\n )\n name: Literal[\"check_model_test_coverage\"]\n min_model_test_coverage_pct: float = Field(\n default=100,\n ge=0,\n le=100,\n )\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n severity: Optional[Literal[\"error\", \"warn\"]] = Field(\n default=\"error\",\n description=\"Severity of the check, one of 'error' or 'warn'.\",\n )\n tests: List[\"DbtBouncerTestBase\"] = Field(default=[])\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_models = len(self.models)\n models_with_tests = []\n for model in self.models:\n for test in self.tests:\n if model.unique_id in test.depends_on.nodes:\n models_with_tests.append(model.unique_id)\n num_models_with_tests = len(set(models_with_tests))\n model_test_coverage_pct = (num_models_with_tests / num_models) * 100\n\n assert (\n model_test_coverage_pct >= self.min_model_test_coverage_pct\n ), f\"Only {model_test_coverage_pct}% of models have at least one test, this is less than the permitted minimum of {self.min_model_test_coverage_pct}%.\"\n
"},{"location":"checks/manifest/check_semantic_models/","title":"Manifest Checks: Semantic Models","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_semantic_models/#manifest.check_semantic_models.CheckSemanticModelOnNonPublicModels","title":"CheckSemanticModelOnNonPublicModels
","text":"Semantic models should be based on public models only.
Receives at execution time:
Name Type Description models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
semantic_model
DbtBouncerSemanticModelBase
The DbtBouncerSemanticModelBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the semantic model path (i.e the .yml file where the semantic model is configured). Semantic model paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the semantic model path (i.e the .yml file where the semantic model is configured). Only semantic model paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_semantic_model_based_on_non_public_models\n
Source code in src/dbt_bouncer/checks/manifest/check_semantic_models.py
class CheckSemanticModelOnNonPublicModels(BaseCheck):\n \"\"\"Semantic models should be based on public models only.\n\n Receives:\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n semantic_model (DbtBouncerSemanticModelBase): The DbtBouncerSemanticModelBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the semantic model path (i.e the .yml file where the semantic model is configured). Semantic model paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the semantic model path (i.e the .yml file where the semantic model is configured). Only semantic model paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_semantic_model_based_on_non_public_models\n ```\n\n \"\"\"\n\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_semantic_model_based_on_non_public_models\"]\n semantic_model: \"DbtBouncerSemanticModelBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n non_public_upstream_dependencies = []\n for model in self.semantic_model.depends_on.nodes:\n if (\n model.split(\".\")[0] == \"model\"\n and model.split(\".\")[1] == self.semantic_model.package_name\n ):\n model = next(m for m in self.models if m.unique_id == model)\n if model.access.value != \"public\":\n non_public_upstream_dependencies.append(model.name)\n\n assert not non_public_upstream_dependencies, f\"Semantic model `{self.semantic_model.name}` is based on a model(s) that is not public: {non_public_upstream_dependencies}.\"\n
"},{"location":"checks/manifest/check_sources/","title":"Manifest Checks: Sources","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceDescriptionPopulated","title":"CheckSourceDescriptionPopulated
","text":"Sources must have a populated description.
Receives at execution time:
Name Type Description source
DbtBouncerSourceBase
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_description_populated\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceDescriptionPopulated(BaseCheck):\n \"\"\"Sources must have a populated description.\n\n Receives:\n source (DbtBouncerSourceBase): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_description_populated\n ```\n\n \"\"\"\n\n name: Literal[\"check_source_description_populated\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n len(self.source.description.strip()) > 4\n ), f\"`{self.source.source_name}.{self.source.name}` does not have a populated description.\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceFreshnessPopulated","title":"CheckSourceFreshnessPopulated
","text":"Sources must have a populated freshness.
Receives at execution time:
Name Type Description source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_freshness_populated\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceFreshnessPopulated(BaseCheck):\n \"\"\"Sources must have a populated freshness.\n\n Receives:\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_freshness_populated\n ```\n\n \"\"\"\n\n name: Literal[\"check_source_freshness_populated\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n error_msg = f\"`{self.source.source_name}.{self.source.name}` does not have a populated freshness.\"\n assert self.source.freshness is not None, error_msg\n assert (\n self.source.freshness.error_after.count is not None\n and self.source.freshness.error_after.period is not None\n ) or (\n self.source.freshness.warn_after.count is not None\n and self.source.freshness.warn_after.period is not None\n ), error_msg\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceHasMetaKeys","title":"CheckSourceHasMetaKeys
","text":"The meta
config for sources must have the specified keys.
Parameters:
Name Type Description Default keys
NestedDict
A list (that may contain sub-lists) of required keys.
required Receives at execution time:
Name Type Description source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_has_meta_keys\n keys:\n - contact:\n - email\n - slack\n - owner\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceHasMetaKeys(BaseCheck):\n \"\"\"The `meta` config for sources must have the specified keys.\n\n Parameters:\n keys (NestedDict): A list (that may contain sub-lists) of required keys.\n\n Receives:\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_has_meta_keys\n keys:\n - contact:\n - email\n - slack\n - owner\n ```\n\n \"\"\"\n\n keys: \"NestedDict\"\n name: Literal[\"check_source_has_meta_keys\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n missing_keys = find_missing_meta_keys(\n meta_config=self.source.meta,\n required_keys=self.keys.model_dump(),\n )\n\n assert (\n missing_keys == []\n ), f\"`{self.source.source_name}.{self.source.name}` is missing the following keys from the `meta` config: {[x.replace('>>', '') for x in missing_keys]}\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceHasTags","title":"CheckSourceHasTags
","text":"Sources must have the specified tags.
Parameters:
Name Type Description Default source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
required tags
List[str]
List of tags to check for.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_has_tags\n tags:\n - tag_1\n - tag_2\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceHasTags(BaseCheck):\n \"\"\"Sources must have the specified tags.\n\n Parameters:\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n tags (List[str]): List of tags to check for.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_has_tags\n tags:\n - tag_1\n - tag_2\n ```\n\n \"\"\"\n\n name: Literal[\"check_source_has_tags\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n tags: List[str]\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n missing_tags = [tag for tag in self.tags if tag not in self.source.tags]\n assert not missing_tags, f\"`{self.source.source_name}.{self.source.name}` is missing required tags: {missing_tags}.\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceLoaderPopulated","title":"CheckSourceLoaderPopulated
","text":"Sources must have a populated loader.
Parameters:
Name Type Description Default source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_loader_populated\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceLoaderPopulated(BaseCheck):\n \"\"\"Sources must have a populated loader.\n\n Parameters:\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_loader_populated\n ```\n\n \"\"\"\n\n name: Literal[\"check_source_loader_populated\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n self.source.loader != \"\"\n ), f\"`{self.source.source_name}.{self.source.name}` does not have a populated loader.\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceNames","title":"CheckSourceNames
","text":"Sources must have a name that matches the supplied regex.
Parameters:
Name Type Description Default source_name_pattern
str
Regexp the source name must match.
required Receives at execution time:
Name Type Description source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_names\n source_name_pattern: >\n ^[a-z0-9_]*$\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceNames(BaseCheck):\n \"\"\"Sources must have a name that matches the supplied regex.\n\n Parameters:\n source_name_pattern (str): Regexp the source name must match.\n\n Receives:\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_names\n source_name_pattern: >\n ^[a-z0-9_]*$\n ```\n\n \"\"\"\n\n name: Literal[\"check_source_names\"]\n source_name_pattern: str\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n re.compile(self.source_name_pattern.strip()).match(self.source.name)\n is not None\n ), f\"`{self.source.source_name}.{self.source.name}` does not match the supplied regex `({self.source_name_pattern.strip()})`.\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceNotOrphaned","title":"CheckSourceNotOrphaned
","text":"Sources must be referenced in at least one model.
Receives at execution time:
Name Type Description models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_not_orphaned\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceNotOrphaned(BaseCheck):\n \"\"\"Sources must be referenced in at least one model.\n\n Receives:\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_not_orphaned\n ```\n\n \"\"\"\n\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_source_not_orphaned\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_refs = sum(\n self.source.unique_id in model.depends_on.nodes for model in self.models\n )\n assert (\n num_refs >= 1\n ), f\"Source `{self.source.source_name}.{self.source.name}` is orphaned, i.e. not referenced by any model.\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourcePropertyFileLocation","title":"CheckSourcePropertyFileLocation
","text":"Source properties files must follow the guidance provided by dbt here.
Receives at execution time:
Name Type Description source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_property_file_location\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourcePropertyFileLocation(BaseCheck):\n \"\"\"Source properties files must follow the guidance provided by dbt [here](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview).\n\n Receives:\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_property_file_location\n ```\n\n \"\"\"\n\n name: Literal[\"check_source_property_file_location\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n path_cleaned = clean_path_str(self.source.original_file_path).replace(\n \"models/staging\", \"\"\n )\n expected_substring = \"_\".join(path_cleaned.split(\"/\")[:-1])\n\n assert path_cleaned.split(\n \"/\",\n )[\n -1\n ].startswith(\n \"_\",\n ), f\"The properties file for `{self.source.source_name}.{self.source.name}` (`{path_cleaned}`) does not start with an underscore.\"\n assert (\n expected_substring in path_cleaned\n ), f\"The properties file for `{self.source.source_name}.{self.source.name}` (`{path_cleaned}`) does not contain the expected substring (`{expected_substring}`).\"\n assert path_cleaned.split(\n \"/\",\n )[\n -1\n ].endswith(\n \"__sources.yml\",\n ), f\"The properties file for `{self.source.source_name}.{self.source.name}` (`{path_cleaned}`) does not end with `__sources.yml`.\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceUsedByModelsInSameDirectory","title":"CheckSourceUsedByModelsInSameDirectory
","text":"Sources can only be referenced by models that are located in the same directory where the source is defined.
Parameters:
Name Type Description Default models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
required source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_used_by_models_in_same_directory\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceUsedByModelsInSameDirectory(BaseCheck):\n \"\"\"Sources can only be referenced by models that are located in the same directory where the source is defined.\n\n Parameters:\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_used_by_models_in_same_directory\n ```\n\n \"\"\"\n\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_source_used_by_models_in_same_directory\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n reffed_models_not_in_same_dir = []\n for model in self.models:\n if (\n self.source.unique_id in model.depends_on.nodes\n and model.original_file_path.split(\"/\")[:-1]\n != self.source.original_file_path.split(\"/\")[:-1]\n ):\n reffed_models_not_in_same_dir.append(model.unique_id.split(\".\")[0])\n\n assert (\n len(reffed_models_not_in_same_dir) == 0\n ), f\"Source `{self.source.source_name}.{self.source.name}` is referenced by models defined in a different directory: {reffed_models_not_in_same_dir}\"\n
"},{"location":"checks/manifest/check_sources/#manifest.check_sources.CheckSourceUsedByOnlyOneModel","title":"CheckSourceUsedByOnlyOneModel
","text":"Each source can be referenced by a maximum of one model.
Receives at execution time:
Name Type Description models
List[DbtBouncerModelBase]
List of DbtBouncerModelBase objects parsed from manifest.json
.
source
DbtBouncerSource
The DbtBouncerSourceBase object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_source_used_by_only_one_model\n
Source code in src/dbt_bouncer/checks/manifest/check_sources.py
class CheckSourceUsedByOnlyOneModel(BaseCheck):\n \"\"\"Each source can be referenced by a maximum of one model.\n\n Receives:\n models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.\n source (DbtBouncerSource): The DbtBouncerSourceBase object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_source_used_by_only_one_model\n ```\n\n \"\"\"\n\n models: List[\"DbtBouncerModelBase\"] = Field(default=[])\n name: Literal[\"check_source_used_by_only_one_model\"]\n source: \"DbtBouncerSourceBase\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n num_refs = sum(\n self.source.unique_id in model.depends_on.nodes for model in self.models\n )\n assert (\n num_refs <= 1\n ), f\"Source `{self.source.source_name}.{self.source.name}` is referenced by more than one model.\"\n
"},{"location":"checks/manifest/check_unit_tests/","title":"Manifest Checks: Unit Tests","text":"Note
The below checks require manifest.json
to be present.
"},{"location":"checks/manifest/check_unit_tests/#manifest.check_unit_tests.CheckUnitTestExpectFormats","title":"CheckUnitTestExpectFormats
","text":"Unit tests can only use the specified formats.
Warning
This check is only supported for dbt 1.8.0 and above.
Parameters:
Name Type Description Default permitted_formats
Optional[List[Literal['csv', 'dict', 'sql']]]
A list of formats that are allowed to be used for expect
input in a unit test.
required Receives at execution time:
Name Type Description manifest_obj
DbtBouncerManifest
The DbtBouncerManifest object parsed from manifest.json
.
unit_test
UnitTests
The UnitTests object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Unit test paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Only unit test paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_unit_test_expect_format\n permitted_formats:\n - csv\n
Source code in src/dbt_bouncer/checks/manifest/check_unit_tests.py
class CheckUnitTestExpectFormats(BaseCheck):\n \"\"\"Unit tests can only use the specified formats.\n\n !!! warning\n\n This check is only supported for dbt 1.8.0 and above.\n\n Parameters:\n permitted_formats (Optional[List[Literal[\"csv\", \"dict\", \"sql\"]]]): A list of formats that are allowed to be used for `expect` input in a unit test.\n\n Receives:\n manifest_obj (DbtBouncerManifest): The DbtBouncerManifest object parsed from `manifest.json`.\n unit_test (UnitTests): The UnitTests object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Unit test paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Only unit test paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_unit_test_expect_format\n permitted_formats:\n - csv\n ```\n\n \"\"\"\n\n manifest_obj: \"DbtBouncerManifest\" = Field(default=None)\n name: Literal[\"check_unit_test_expect_format\"]\n permitted_formats: List[Literal[\"csv\", \"dict\", \"sql\"]] = Field(\n default=[\"csv\", \"dict\", \"sql\"],\n )\n unit_test: \"UnitTests\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if (\n semver.Version.parse(self.manifest_obj.manifest.metadata.dbt_version)\n >= \"1.8.0\"\n ):\n assert (\n self.unit_test.expect.format.value in self.permitted_formats\n ), f\"Unit test `{self.unit_test.name}` has an `expect` format that is not permitted. Permitted formats are: {self.permitted_formats}.\"\n else:\n logging.warning(\n \"The `check_unit_test_expect_format` check is only supported for dbt 1.8.0 and above.\",\n )\n
"},{"location":"checks/manifest/check_unit_tests/#manifest.check_unit_tests.CheckUnitTestGivenFormats","title":"CheckUnitTestGivenFormats
","text":"Unit tests can only use the specified formats.
Warning
This check is only supported for dbt 1.8.0 and above.
Parameters:
Name Type Description Default permitted_formats
Optional[List[Literal['csv', 'dict', 'sql']]]
A list of formats that are allowed to be used for expect
input in a unit test.
required Receives at execution time:
Name Type Description manifest_obj
DbtBouncerManifest
The DbtBouncerManifest object parsed from manifest.json
.
unit_test
UnitTests
The UnitTests object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Unit test paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Only unit test paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
manifest_checks:\n - name: check_unit_test_given_formats\n permitted_formats:\n - csv\n
Source code in src/dbt_bouncer/checks/manifest/check_unit_tests.py
class CheckUnitTestGivenFormats(BaseCheck):\n \"\"\"Unit tests can only use the specified formats.\n\n !!! warning\n\n This check is only supported for dbt 1.8.0 and above.\n\n Parameters:\n permitted_formats (Optional[List[Literal[\"csv\", \"dict\", \"sql\"]]]): A list of formats that are allowed to be used for `expect` input in a unit test.\n\n Receives:\n manifest_obj (DbtBouncerManifest): The DbtBouncerManifest object parsed from `manifest.json`.\n unit_test (UnitTests): The UnitTests object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Unit test paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the unit test path (i.e the .yml file where the unit test is configured). Only unit test paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n manifest_checks:\n - name: check_unit_test_given_formats\n permitted_formats:\n - csv\n ```\n\n \"\"\"\n\n manifest_obj: \"DbtBouncerManifest\" = Field(default=None)\n name: Literal[\"check_unit_test_given_formats\"]\n permitted_formats: List[Literal[\"csv\", \"dict\", \"sql\"]] = Field(\n default=[\"csv\", \"dict\", \"sql\"],\n )\n unit_test: \"UnitTests\" = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n if (\n semver.Version.parse(self.manifest_obj.manifest.metadata.dbt_version)\n >= \"1.8.0\"\n ):\n given_formats = [i.format.value for i in self.unit_test.given]\n assert all(\n e in self.permitted_formats for e in given_formats\n ), f\"Unit test `{self.unit_test.name}` has given formats which are not permitted. Permitted formats are: {self.permitted_formats}.\"\n else:\n logging.warning(\n \"The `check_unit_test_given_formats` check is only supported for dbt 1.8.0 and above.\",\n )\n
"},{"location":"checks/run_results/check_run_results/","title":"Run Results Checks","text":"Note
The below checks require both manifest.json
and run_results.json
to be present.
"},{"location":"checks/run_results/check_run_results/#run_results.check_run_results.CheckRunResultsMaxGigabytesBilled","title":"CheckRunResultsMaxGigabytesBilled
","text":"Each result can have a maximum number of gigabytes billed.
Note
Note that this check only works for the dbt-bigquery
adapter.
Parameters:
Name Type Description Default max_gigabytes_billed
float
The maximum number of gigabytes billed.
required run_result
DbtBouncerRunResult
The DbtBouncerRunResult object to check.
required Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the resource path. Resource paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the resource path. Only resource paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
run_results_checks:\n - name: check_run_results_max_gigabytes_billed\n max_gigabytes_billed: 100\n
Source code in src/dbt_bouncer/checks/run_results/check_run_results.py
class CheckRunResultsMaxGigabytesBilled(BaseCheck):\n \"\"\"Each result can have a maximum number of gigabytes billed.\n\n !!! note\n\n Note that this check only works for the `dbt-bigquery` adapter.\n\n Parameters:\n max_gigabytes_billed (float): The maximum number of gigabytes billed.\n run_result (DbtBouncerRunResult): The DbtBouncerRunResult object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the resource path. Resource paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the resource path. Only resource paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Raises: # noqa:DOC502\n KeyError: If the `dbt-bigquery` adapter is not used.\n\n Example(s):\n ```yaml\n run_results_checks:\n - name: check_run_results_max_gigabytes_billed\n max_gigabytes_billed: 100\n ```\n\n \"\"\"\n\n max_gigabytes_billed: float\n name: Literal[\"check_run_results_max_gigabytes_billed\"]\n run_result: Optional[\"DbtBouncerRunResultBase\"] = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n try:\n gigabytes_billed = self.run_result.adapter_response[\"bytes_billed\"] / (\n 1000**3\n )\n except KeyError as e:\n raise RuntimeError( # noqa: DOC501\n \"`bytes_billed` not found in adapter response. Are you using the `dbt-bigquery` adapter?\",\n ) from e\n\n assert (\n gigabytes_billed < self.max_gigabytes_billed\n ), f\"`{self.run_result.unique_id.split('.')[-2]}` results in ({gigabytes_billed} billed bytes, this is greater than permitted ({self.max_gigabytes_billed}).\"\n
"},{"location":"checks/run_results/check_run_results/#run_results.check_run_results.CheckRunResultsMaxExecutionTime","title":"CheckRunResultsMaxExecutionTime
","text":"Each result can take a maximum duration (seconds).
Parameters:
Name Type Description Default max_execution_time_seconds
float
The maximum execution time (seconds) allowed for a node.
required Receives at execution time:
Name Type Description run_result
DbtBouncerRunResult
The DbtBouncerRunResult object to check.
Other Parameters (passed via config file):
Name Type Description exclude
Optional[str]
Regex pattern to match the resource path. Resource paths that match the pattern will not be checked.
include
Optional[str]
Regex pattern to match the resource path. Only resource paths that match the pattern will be checked.
severity
Optional[Literal['error', 'warn']]
Severity level of the check. Default: error
.
Example(s):
run_results_checks:\n - name: check_run_results_max_execution_time\n max_execution_time_seconds: 60\n
run_results_checks:\n - name: check_run_results_max_execution_time\n include: ^models/staging # Not a good idea, here for demonstration purposes only\n max_execution_time_seconds: 10\n
Source code in src/dbt_bouncer/checks/run_results/check_run_results.py
class CheckRunResultsMaxExecutionTime(BaseCheck):\n \"\"\"Each result can take a maximum duration (seconds).\n\n Parameters:\n max_execution_time_seconds (float): The maximum execution time (seconds) allowed for a node.\n\n Receives:\n run_result (DbtBouncerRunResult): The DbtBouncerRunResult object to check.\n\n Other Parameters:\n exclude (Optional[str]): Regex pattern to match the resource path. Resource paths that match the pattern will not be checked.\n include (Optional[str]): Regex pattern to match the resource path. Only resource paths that match the pattern will be checked.\n severity (Optional[Literal[\"error\", \"warn\"]]): Severity level of the check. Default: `error`.\n\n Example(s):\n ```yaml\n run_results_checks:\n - name: check_run_results_max_execution_time\n max_execution_time_seconds: 60\n ```\n ```yaml\n run_results_checks:\n - name: check_run_results_max_execution_time\n include: ^models/staging # Not a good idea, here for demonstration purposes only\n max_execution_time_seconds: 10\n ```\n\n \"\"\"\n\n max_execution_time_seconds: float\n name: Literal[\"check_run_results_max_execution_time\"]\n run_result: Optional[\"DbtBouncerRunResultBase\"] = Field(default=None)\n\n def execute(self) -> None:\n \"\"\"Execute the check.\"\"\"\n assert (\n self.run_result.execution_time <= self.max_execution_time_seconds\n ), f\"`{self.run_result.unique_id.split('.')[-1]}` has an execution time ({self.run_result.execution_time} greater than permitted ({self.max_execution_time_seconds}s).\"\n
"}]}
\ No newline at end of file