Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding dbt_packages when dbt_deps is False #730

Merged
merged 13 commits into from
Dec 5, 2023
Merged

Adding dbt_packages when dbt_deps is False #730

merged 13 commits into from
Dec 5, 2023

Conversation

DanMawdsleyBA
Copy link
Contributor

@DanMawdsleyBA DanMawdsleyBA commented Nov 30, 2023 •
edited by jbandoro
Loading

Description

Related Issue(s)

resolves #727, resolves #715

Breaking Change?

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

@DanMawdsleyBA DanMawdsleyBA requested a review from a team as a code owner November 30, 2023 13:22
@DanMawdsleyBA DanMawdsleyBA requested a review from a team November 30, 2023 13:22
@dosubot dosubot bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Nov 30, 2023
@netlify Netlify
Copy link

netlify bot commented Nov 30, 2023 •
edited
Loading

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name Link
🔨 Latest commit 7e0a37e
🔍 Latest deploy log https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/656f2b3601954e000808774b

@dosubot dosubot bot added area:dependencies Related to dependencies, like Python packages, library versions, etc dbt:deps Primarily related to dbt deps command or functionality labels Nov 30, 2023
@codecov Codecov
Copy link

codecov bot commented Nov 30, 2023 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (6ccde5c) 93.12% compared to head (7e0a37e) 93.13%.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main     #730   +/-   ##
=======================================
  Coverage   93.12%   93.13%           
=======================================
  Files          55       55           
  Lines        2429     2431    +2     
=======================================
+ Hits         2262     2264    +2     
  Misses        167      167

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tatiana tatiana added this to the 1.3.0 milestone Nov 30, 2023
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Nov 30, 2023
@DanMawdsleyBA
Copy link
Contributor Author

@tatiana Added a test for the change hope it works (May need some help if it doesn't). If you're happy could you approve the integration tests

@DanMawdsleyBA DanMawdsleyBA changed the title Adding dbt_packages when dbt_deps is true Adding dbt_packages when dbt_deps is False Nov 30, 2023
@DanMawdsleyBA
Copy link
Contributor Author

@tatiana Would you be able to help me fix the integration test? I'm trying to add a test that manually install dbt deps beforehand and then runs cosmos with dbt_deps is False which should mean it uses the pre-installed dbt-packages

tatiana
tatiana approved these changes Dec 5, 2023
View reviewed changes
Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot, @DanMawdsleyBA , we'll be releasing this as part of an alpha 1.3 this week!

@tatiana tatiana merged commit d2248b7 into astronomer:main Dec 5, 2023
42 checks passed
@DanMawdsleyBA
Copy link
Contributor Author

That's great thanks! We have a release on Thursday would be great to have it in time for that. Thanks for your help @tatiana

This was referenced Dec 5, 2023
Closed
Merged
tatiana pushed a commit that referenced this pull request Dec 6, 2023
@jbandoro
Tests: use temp dir for `test_load_via_dbt_ls_without_dbt_deps_and_pr…
3bc60a9 
…einstalled_dbt_packages` (#748)

This is a follow up to #730 so that the fixture `tmp_dbt_project_dir` is
used for the test instead of the actual project directory since it
installs dbt packages that need to be cleaned up.
tatiana added a commit that referenced this pull request Dec 7, 2023
@tatiana
Release 1.3.0a3
ea18bb1 
Features

* Add ProfileMapping for Vertica by @perttus in #540 and #688
* Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649
* Add support to select using (some) graph operators when using LoadMode.CUSTOM and LoadMode.DBT_MANIFEST by @tatiana in #728
* Add cosmos/propagate_logs Airflow config support for disabling log pr… by @agreenburg in #648
* Add operator_args full_refresh as a templated field by @joppevos in #623
* Expose environment variables and dbt variables in ProjectConfig by @jbandoro in #735

Enhancements

* Make Pydantic an optional dependency by @pixie79 in #736
* Create a symbolic link to dbt_packages when dbt_deps is False when using LoadMode.DBT_LS by @DanMawdsleyBA in #730
* Support no profile_config for ExecutionMode.KUBERNETES and ExecutionMode.DOCKER by @MrBones757 and @tatiana in #681 and #731
* Add aws_session_token for Athena mapping by @benjamin-awd in #663

Others

* Replace flake8 for Ruff by @joppevos in #743
* Reduce code complexity to 8 by @joppevos in #738
* Update conflict matrix between Airflow and dbt versions by @tatiana in #731
* Speed up integration tests by @jbandoro in #732
@tatiana tatiana mentioned this pull request Dec 7, 2023
Merged
@tatiana
Copy link
Collaborator

tatiana commented Dec 7, 2023

@DanMawdsleyBA It's part of 1.3.0a3:
https://pypi.org/project/astronomer-cosmos/1.3.0a3/

Please let us know if you find any issues!

tatiana added a commit that referenced this pull request Dec 7, 2023
@tatiana
Release 1.3.0a3
edb2667 
Features

* Add ProfileMapping for Vertica by @perttus in #540 and #688
* Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649
* Add support to select using (some) graph operators when using LoadMode.CUSTOM and LoadMode.DBT_MANIFEST by @tatiana in #728
* Add cosmos/propagate_logs Airflow config support for disabling log pr… by @agreenburg in #648
* Add operator_args full_refresh as a templated field by @joppevos in #623
* Expose environment variables and dbt variables in ProjectConfig by @jbandoro in #735

Enhancements

* Make Pydantic an optional dependency by @pixie79 in #736
* Create a symbolic link to dbt_packages when dbt_deps is False when using LoadMode.DBT_LS by @DanMawdsleyBA in #730
* Support no profile_config for ExecutionMode.KUBERNETES and ExecutionMode.DOCKER by @MrBones757 and @tatiana in #681 and #731
* Add aws_session_token for Athena mapping by @benjamin-awd in #663

Others

* Replace flake8 for Ruff by @joppevos in #743
* Reduce code complexity to 8 by @joppevos in #738
* Update conflict matrix between Airflow and dbt versions by @tatiana in #731
* Speed up integration tests by @jbandoro in #732
jbandoro pushed a commit that referenced this pull request Dec 7, 2023
@tatiana
Release 1.3.0a3 (#753)
b1cdc3c 
**Features**

* Add `ProfileMapping` for Snowflake encrypted private key path by
@ivanstillfront in #608
* Add support for Snowflake encrypted private key environment variable
by @DanMawdsleyBA in #649
* Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in
#616
* Add support to select using (some) graph operators when using
`LoadMode.CUSTOM` and `LoadMode.DBT_MANIFEST` by @tatiana in #728
* Add cosmos/propagate_logs Airflow config support for disabling log
propagation by @agreenburg in #648
* Add operator_args ``full_refresh`` as a templated field by @joppevos
in #623
* Expose environment variables and dbt variables in ``ProjectConfig`` by
@jbandoro in #735

**Enhancements**

* Make Pydantic an optional dependency by @pixie79 in #736
* Create a symbolic link to `dbt_packages` when `dbt_deps` is False when
using `LoadMode.DBT_LS` by @DanMawdsleyBA in #730
* Support no `profile_config` for `ExecutionMode.KUBERNETES` and
`ExecutionMode.DOCKER` by @MrBones757 and @tatiana in #681 and #731
* Add `aws_session_token` for Athena mapping by @benjamin-awd in #663

**Others**

* Replace flake8 for Ruff by @joppevos in #743
* Reduce code complexity to 8 by @joppevos in #738
* Update conflict matrix between Airflow and dbt versions by @tatiana in
#731
* Speed up integration tests by @jbandoro in #732
@tatiana tatiana mentioned this pull request Jan 4, 2024
Merged
tatiana added a commit that referenced this pull request Jan 4, 2024
@tatiana
Release 1.3.0 (#781)
0de8bdd 
**Features**

* Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in #733
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)).
* Add support to select using (some) graph operators when using
``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in #728
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude))
* Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in
#755,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)).
* Add ``ProfileMapping`` for Vertica by @perttus in #540, #688 and #741,
as
([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)).
* Add ``ProfileMapping`` for Snowflake encrypted private key path by
@ivanstillfront in #608, as ([documentation](
https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)).
* Add support for Snowflake encrypted private key environment variable
by @DanMawdsleyBA in #649
* Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro
in #616,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)).
* Add cosmos/propagate_logs Airflow config support for disabling log
propagation by @agreenburg in #648,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)).
* Add operator_args ``full_refresh`` as a templated field by @joppevos
in #623
* Expose environment variables and dbt variables in ``ProjectConfig`` by
@jbandoro in #735
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)).
* Support disabling event tracking when using Cosmos profile mapping by
@jbandoro in #768,
([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)).

**Enhancements**

* Make Pydantic an optional dependency by @pixie79 in #736
* Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False
when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in #730
* Add ``aws_session_token`` for Athena mapping by @benjamin-awd in #663
* Retrieve temporary credentials from ``conn_id`` for Athena by @octiva
in #758
* Extend ``DbtDocsLocalOperator`` with static flag by @joppevos  in #759

**Bug fixes**

* Remove Pydantic upper version restriction so Cosmos can be used with
Airflow 2.8 by @jlaneve in #772

**Others**

* Replace flake8 for Ruff by @joppevos in #743
* Reduce code complexity to 8 by @joppevos in #738
* Speed up integration tests by @jbandoro in #732
* Fix README quickstart link in by @RNHTTR in #776
* Add package location to work with hatchling 1.19.0 by @jbandoro in
#761
* Fix type check error in ``DbtKubernetesBaseOperator.build_env_args``
by @jbandoro in #766
* Improve ``DBT_MANIFEST`` documentation by @dwreeves in #757
* Update conflict matrix between Airflow and dbt versions by @tatiana in
#731 and #779
* pre-commit updates in #775, #770, #762
ykuc pushed a commit to ykuc/astronomer-cosmos that referenced this pull request Jan 11, 2024
@tatiana @YuriyK90
Release 1.3.0 (astronomer#781)
3aa9331 
**Features**

* Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in astronomer#733
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)).
* Add support to select using (some) graph operators when using
``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in astronomer#728
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude))
* Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in
astronomer#755,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)).
* Add ``ProfileMapping`` for Vertica by @perttus in astronomer#540, astronomer#688 and astronomer#741,
as
([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)).
* Add ``ProfileMapping`` for Snowflake encrypted private key path by
@ivanstillfront in astronomer#608, as ([documentation](
https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)).
* Add support for Snowflake encrypted private key environment variable
by @DanMawdsleyBA in astronomer#649
* Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro
in astronomer#616,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)).
* Add cosmos/propagate_logs Airflow config support for disabling log
propagation by @agreenburg in astronomer#648,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)).
* Add operator_args ``full_refresh`` as a templated field by @joppevos
in astronomer#623
* Expose environment variables and dbt variables in ``ProjectConfig`` by
@jbandoro in astronomer#735
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)).
* Support disabling event tracking when using Cosmos profile mapping by
@jbandoro in astronomer#768,
([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)).

**Enhancements**

* Make Pydantic an optional dependency by @pixie79 in astronomer#736
* Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False
when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in astronomer#730
* Add ``aws_session_token`` for Athena mapping by @benjamin-awd in astronomer#663
* Retrieve temporary credentials from ``conn_id`` for Athena by @octiva
in astronomer#758
* Extend ``DbtDocsLocalOperator`` with static flag by @joppevos  in astronomer#759

**Bug fixes**

* Remove Pydantic upper version restriction so Cosmos can be used with
Airflow 2.8 by @jlaneve in astronomer#772

**Others**

* Replace flake8 for Ruff by @joppevos in astronomer#743
* Reduce code complexity to 8 by @joppevos in astronomer#738
* Speed up integration tests by @jbandoro in astronomer#732
* Fix README quickstart link in by @RNHTTR in astronomer#776
* Add package location to work with hatchling 1.19.0 by @jbandoro in
astronomer#761
* Fix type check error in ``DbtKubernetesBaseOperator.build_env_args``
by @jbandoro in astronomer#766
* Improve ``DBT_MANIFEST`` documentation by @dwreeves in astronomer#757
* Update conflict matrix between Airflow and dbt versions by @tatiana in
astronomer#731 and astronomer#779
* pre-commit updates in astronomer#775, astronomer#770, astronomer#762
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
@DanMawdsleyBA @arojasb3
Symlink dbt packages when RenderConfig.dbt_deps=False (astronomer#730)
fec20ee 
Closes: astronomer#727
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
@jbandoro @arojasb3
Tests: use temp dir for `test_load_via_dbt_ls_without_dbt_deps_and_pr…
f4d771b 
…einstalled_dbt_packages` (astronomer#748)

This is a follow up to astronomer#730 so that the fixture `tmp_dbt_project_dir` is
used for the test instead of the actual project directory since it
installs dbt packages that need to be cleaned up.
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
@tatiana @arojasb3
Release 1.3.0a3 (astronomer#753)
5038c2c 
**Features**

* Add `ProfileMapping` for Snowflake encrypted private key path by
@ivanstillfront in astronomer#608
* Add support for Snowflake encrypted private key environment variable
by @DanMawdsleyBA in astronomer#649
* Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in
astronomer#616
* Add support to select using (some) graph operators when using
`LoadMode.CUSTOM` and `LoadMode.DBT_MANIFEST` by @tatiana in astronomer#728
* Add cosmos/propagate_logs Airflow config support for disabling log
propagation by @agreenburg in astronomer#648
* Add operator_args ``full_refresh`` as a templated field by @joppevos
in astronomer#623
* Expose environment variables and dbt variables in ``ProjectConfig`` by
@jbandoro in astronomer#735

**Enhancements**

* Make Pydantic an optional dependency by @pixie79 in astronomer#736
* Create a symbolic link to `dbt_packages` when `dbt_deps` is False when
using `LoadMode.DBT_LS` by @DanMawdsleyBA in astronomer#730
* Support no `profile_config` for `ExecutionMode.KUBERNETES` and
`ExecutionMode.DOCKER` by @MrBones757 and @tatiana in astronomer#681 and astronomer#731
* Add `aws_session_token` for Athena mapping by @benjamin-awd in astronomer#663

**Others**

* Replace flake8 for Ruff by @joppevos in astronomer#743
* Reduce code complexity to 8 by @joppevos in astronomer#738
* Update conflict matrix between Airflow and dbt versions by @tatiana in
astronomer#731
* Speed up integration tests by @jbandoro in astronomer#732
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
@tatiana @arojasb3
Release 1.3.0 (astronomer#781)
0fe721e 
**Features**

* Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in astronomer#733
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)).
* Add support to select using (some) graph operators when using
``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in astronomer#728
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude))
* Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in
astronomer#755,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)).
* Add ``ProfileMapping`` for Vertica by @perttus in astronomer#540, astronomer#688 and astronomer#741,
as
([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)).
* Add ``ProfileMapping`` for Snowflake encrypted private key path by
@ivanstillfront in astronomer#608, as ([documentation](
https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)).
* Add support for Snowflake encrypted private key environment variable
by @DanMawdsleyBA in astronomer#649
* Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro
in astronomer#616,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)).
* Add cosmos/propagate_logs Airflow config support for disabling log
propagation by @agreenburg in astronomer#648,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)).
* Add operator_args ``full_refresh`` as a templated field by @joppevos
in astronomer#623
* Expose environment variables and dbt variables in ``ProjectConfig`` by
@jbandoro in astronomer#735
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)).
* Support disabling event tracking when using Cosmos profile mapping by
@jbandoro in astronomer#768,
([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)).

**Enhancements**

* Make Pydantic an optional dependency by @pixie79 in astronomer#736
* Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False
when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in astronomer#730
* Add ``aws_session_token`` for Athena mapping by @benjamin-awd in astronomer#663
* Retrieve temporary credentials from ``conn_id`` for Athena by @octiva
in astronomer#758
* Extend ``DbtDocsLocalOperator`` with static flag by @joppevos  in astronomer#759

**Bug fixes**

* Remove Pydantic upper version restriction so Cosmos can be used with
Airflow 2.8 by @jlaneve in astronomer#772

**Others**

* Replace flake8 for Ruff by @joppevos in astronomer#743
* Reduce code complexity to 8 by @joppevos in astronomer#738
* Speed up integration tests by @jbandoro in astronomer#732
* Fix README quickstart link in by @RNHTTR in astronomer#776
* Add package location to work with hatchling 1.19.0 by @jbandoro in
astronomer#761
* Fix type check error in ``DbtKubernetesBaseOperator.build_env_args``
by @jbandoro in astronomer#766
* Improve ``DBT_MANIFEST`` documentation by @dwreeves in astronomer#757
* Update conflict matrix between Airflow and dbt versions by @tatiana in
astronomer#731 and astronomer#779
* pre-commit updates in astronomer#775, astronomer#770, astronomer#762
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jbandoro jbandoro jbandoro left review comments

@tatiana tatiana tatiana approved these changes

Assignees
No one assigned
Labels
area:dependencies Related to dependencies, like Python packages, library versions, etc dbt:deps Primarily related to dbt deps command or functionality size:M This PR changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
1.3.0
3 participants
@DanMawdsleyBA @tatiana @jbandoro