Implementation of wikis and tags feature #250
Conversation
99Lys
changed the title
99Lys
force-pushed
the
DX-98038-DBT-wiki-and-tags-extension-implementation
branch
from
67892c0 to
79bb2f9
Compare
howareyouman
changed the base branch from
DX-98038-DBT-wiki-and-tags-extension-implementation
to
main
99Lys
force-pushed
the
DX-98038-DBT-wiki-and-tags-extension-implementation
branch
2 times, most recently
from
ec03855 to
2a84e48
Compare
99Lys
force-pushed
the
DX-98038-DBT-wiki-and-tags-extension-implementation
branch
from
2a84e48 to
47f998f
Compare
There was a problem hiding this comment.
Great PR!
@howareyouman already pointed pretty much everything, I just have a few questions where I might be lacking context to understand
| def setUp(self, project): | ||
| run_dbt(["seed"]) | ||
| run_dbt() |
There was a problem hiding this comment.
Asking as from what I know dbt seed is used to load csv files from /seeds but I don't see any
There was a problem hiding this comment.
This setUp function originally exists in BasePersistDocsBase class (from where BasePersistDocs is inherited). This is a fixture set to be running automatically @pytest.fixture(scope="class", autouse=True) in autouse=True. Since I wanted to have control on when to run dbt, I just copy pasted it and changed the annotation to autouse=False. Maybe I should've added a comment about this.
I did something similar for function test_has_comments_pglike by replacing its content so this test is not being run.
I don't know what is the best practice here, please let me know if there is a best way to do this! Thanks
There was a problem hiding this comment.
Could we chain dbt commands here? seed and run in 1 command?
There was a problem hiding this comment.
@99Lys If you will take a look on the code of run_dbt() function you will take a look that it's uses command run. So to chain operations in this function you can easily run run_dbt(["seed", "run"]) and that will be 1 dbt command run
| return { | ||
| "table_model.sql": _MODELS__TABLE, | ||
| "view_model.sql": _MODELS__VIEW, | ||
| "schema.yml": _PROPERTIES__SCHEMA_YML, |
There was a problem hiding this comment.
Do we need to set the schema in here? Isn't this handled by the unique_schema fixture? I'm unsure if the models fixture should also be used to set a schema.yml
There was a problem hiding this comment.
From what I understood, unique_schema is only "building" the correct path where the models are being created and that is later used in dbt_profile_data to update the project configuration.
Initially, I was setting schema.yml in properties. But since I needed to update it later, I moved to models because properties/ doesn't seem to exist so I could only perform write_file(<updated_schema.yml>, project.project_root, "models", "schema.yml").
Also I don't think it's a problem to configure schema.yml in models because in an actual project by default there is where it is stored.
Summary
Wikis and tags feature implementation.
Description
One of dbt's core features is source documentation. Right now, this documentation does not propagate into Dremio’s wiki, leading to two separate sources of truth.
The goal is to display dbt documentation - more specifically, wikis and tags - inside of Dremio.
Test Results
All added tests are running
Changelog
relationoption frompersist_docsconfigurationdremio__persist_docscreatedpersist_docsmacroRelated Issue
#86
#196