Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make node resolution environment aware #71

Open
zendesk-sova opened this issue Jun 27, 2024 · 4 comments
Open

Make node resolution environment aware #71

zendesk-sova opened this issue Jun 27, 2024 · 4 comments
Assignees
Labels
enhancement New feature or request

Comments

@zendesk-sova
Copy link

zendesk-sova commented Jun 27, 2024

Is your feature request related to a problem? Please describe.
Hi folks,
Thanks for the great work, I wonder is it possible to make node resolution process to be environment (target) aware?

So in our use case lower environments (targets) like dev, ci are using one set of Snowflake tables for defer and we can put it in the manifest, but for higher environments (target) like prod the same model should be resolved to a different fully qualified name. If I rephrase the question can we hack generate_schema_name and generate_database_name macros to node resolution?

Describe alternatives you've considered
Not quite sure if there is a solution other than maintain N different manifests one for each environment, which doesn't look like an optimal and potentially error prone method.

Additional context
I'm ready to contribute to the solution with enough guidance provided.

@zendesk-sova zendesk-sova added enhancement New feature or request triage This issue is being investigated labels Jun 27, 2024
@nicholasyager
Copy link
Owner

Hi @zendesk-sova 👋🏻 Thanks for submitting this feature request. I think I understand the request, but I want to make sure that I have a complete mental model before I start digging in too deep. Can you please describe in a little more details precisely what the project and environment configuration looks like for your set-up? It sounds like there are two separate environments with separate snowflake objects attempting to share a manifest, and I'm not quite sure what this looks like in practice.

Thanks!

@zendesk-sova
Copy link
Author

Hi @nicholasyager ,
Yes, we have multi-project multi-environment setup with complex database resolution logic. If I understand correctly plugin just takes fully qualified name from the manifest and uses it in models, and I'm asking if it's possible to dynamically resolve what database/schema would've been if we applied custom generate macros at the moment of execution.

The other moment with fixed prebuilt manifests, how to work with multiple targets locally as $DBT_TARGET environment variable is not synchronized with --target <target_name> and such kind of operations become extremely error prone.

@sergey-vdovin
Copy link

Hello everybody!
Thanks for the contributions to [dbt-loom].

Is looks like a scenario for an alternative resolution is what [dbt-mesh] is doing, I am thinking about how that could be implemented in [dbt-loom] as well:

https://docs.getdbt.com/docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref

Safeguarding production data with staging environments
When working in a Development environment, cross-project refs normally resolve to the Production environment of the project. However, to protect production data, set up a Staging deployment environment within your projects. With a staging environment integrated into the project, any references from external projects during development workflows resolve to the Staging environment. This adds a layer of security between your Deployment and Production environments by limiting access to production data.

@nicholasyager
Copy link
Owner

Thanks for the link, @sergey-vdovin! Based on the docs provided, it sounds like dbt Mesh is tracking state for both staging and production environments. I have a hunch that their proprietary systems are simply tracking manifests for all environments. This hunch is based on the requirement that one has a completed staging environment run prior to attempting to use cross-project refs.

In our case, dbt-loom is very simplistic. It has no concept of an environment. Rather, it assumes that the manifests being piped into it are exactly what we want. For this feature to be realistically implemented, we'd also need to have a concept of environment external to the concept of project. This is doable, but require more forethought and design around how we want to configure and implement. Maybe it can be rolled in with #61?

In any case, I'm open to suggestions!

@nicholasyager nicholasyager removed the triage This issue is being investigated label Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants