Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose environment variables and dbt vars configuration via ProjectConfig #712

Closed
tatiana opened this issue Nov 23, 2023 · 3 comments · Fixed by #735
Closed

Expose environment variables and dbt vars configuration via ProjectConfig #712

tatiana opened this issue Nov 23, 2023 · 3 comments · Fixed by #735
Assignees
Labels
area:config Related to configuration, like YAML files, environment variables, or executer configuration area:docs Relating to documentation, changes, fixes, improvement dbt:run Primarily related to dbt run command or functionality parsing:dbt_ls Issues, questions, or features related to dbt_ls parsing priority:medium Medium priority issues are important issues that may have a workaround and medium impact
Milestone

Comments

@tatiana
Copy link
Collaborator

tatiana commented Nov 23, 2023

Context

As of Cosmos 1.2.4, users cannot define dbt --vars when running LoadMode.DBT_LS during project parsing / DAG rendering. They are able to define environment variables, thanks to #690 .

At the moment, users have to also define the environment variables and dbt vars for the task execution, using:
operator_args={"env": ..., "var": ...}
as described in https://astronomer.github.io/astronomer-cosmos/configuration/operator-args.html

This discussion started as part of this review:
#690 (review)

Acceptance criteria

  • Deprecate operator args env & var, raising warnings, and fully deprecate them as of Cosmos 2.x
  • Deprecate RenderConfig.env_vars, raising warnings, and fully deprecate them as of Cosmos 2.x
  • Expose both dbt_vars and env_vars within the ProjectConfig
  • Update the documentation and example DAGs to use the new ProjectConfig arguments
Copy link

dosubot bot commented Nov 23, 2023

🤖

It seems like you've got this under control, if you want help or have specific questions, let me know what I can do for you!

@dosubot dosubot bot added area:config Related to configuration, like YAML files, environment variables, or executer configuration area:docs Relating to documentation, changes, fixes, improvement dbt:run Primarily related to dbt run command or functionality parsing:dbt_ls Issues, questions, or features related to dbt_ls parsing priority:medium Medium priority issues are important issues that may have a workaround and medium impact labels Nov 23, 2023
@jbandoro
Copy link
Collaborator

Thanks @tatiana for the detailed AC, I can work on contributing this 👍

@tatiana tatiana added this to the 1.3.0 milestone Nov 30, 2023
@jbandoro
Copy link
Collaborator

jbandoro commented Nov 30, 2023

@tatiana do you by chance have a dbt example where dbt variables passed to dbt ls will change the models listed. I'd like to have tests for this when use load method of dbt ls with --vars but haven't found any examples of where variables could be used in order to change models listed, and consequently the dbt graph cosmos would render.

edit: I found a dbt issue here, where the reference uses a variable - will give this a go!

jbandoro added a commit that referenced this issue Dec 7, 2023
## Description

Currently users have to specify environment variables in both
`RenderConfig` and `operator_args` for the dbt dag so that they're used
during rendering and execution. dbt variables cannot currently be used
in rendering, only during execution in `operator_args`. This PR exposes
`env_vars` and `dbt_vars` in `ProjectConfig` and uses the dbt variables
in dbt ls load mode.

Updates in this PR:

- Deprecates `operator_args` "env" and "var", raising warnings that they
will be removed in Cosmos 2.x
- Deprecates `RenderConfig.env_vars` raising warnings that it will be
removed in Cosmos 2.x
- Adds both `dbt_vars` and `env_vars` within `ProjectConfig`
  - dbt variables are used in dbt ls load method
- Raises an exception if **both** operator_args and ProjectConfig
variables are used.
- Updates docs and example DAGs to use ProjectConfig args.

## Related Issue(s)

Closes #712 
Closes #544

## Breaking Change?

None

## Checklist

- [x] I have made corresponding changes to the documentation (if
required)
- [x] I have added tests that prove my fix is effective or that my
feature works

---------

Co-authored-by: Tatiana Al-Chueyr <[email protected]>
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this issue Jul 14, 2024
…tronomer#735)

## Description

Currently users have to specify environment variables in both
`RenderConfig` and `operator_args` for the dbt dag so that they're used
during rendering and execution. dbt variables cannot currently be used
in rendering, only during execution in `operator_args`. This PR exposes
`env_vars` and `dbt_vars` in `ProjectConfig` and uses the dbt variables
in dbt ls load mode.

Updates in this PR:

- Deprecates `operator_args` "env" and "var", raising warnings that they
will be removed in Cosmos 2.x
- Deprecates `RenderConfig.env_vars` raising warnings that it will be
removed in Cosmos 2.x
- Adds both `dbt_vars` and `env_vars` within `ProjectConfig`
  - dbt variables are used in dbt ls load method
- Raises an exception if **both** operator_args and ProjectConfig
variables are used.
- Updates docs and example DAGs to use ProjectConfig args.

## Related Issue(s)

Closes astronomer#712 
Closes astronomer#544

## Breaking Change?

None

## Checklist

- [x] I have made corresponding changes to the documentation (if
required)
- [x] I have added tests that prove my fix is effective or that my
feature works

---------

Co-authored-by: Tatiana Al-Chueyr <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:config Related to configuration, like YAML files, environment variables, or executer configuration area:docs Relating to documentation, changes, fixes, improvement dbt:run Primarily related to dbt run command or functionality parsing:dbt_ls Issues, questions, or features related to dbt_ls parsing priority:medium Medium priority issues are important issues that may have a workaround and medium impact
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants