Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix only create task group and test task only if model has test #543

Merged
merged 3 commits into from
Sep 25, 2023

Conversation

raffifu
Copy link
Contributor

@raffifu raffifu commented Sep 15, 2023

Description

This PR will only create task group if dbt test exist by adding propery has_test on DbtNode. The test dependency will be updated after loading the dbt project into DbtGraph

Related Issue(s)

closes #531

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

@netlify
Copy link

netlify bot commented Sep 15, 2023

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name Link
🔨 Latest commit bce3fcf
🔍 Latest deploy log https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/65118c00935a7f0008494df1

@raffifu raffifu temporarily deployed to external September 18, 2023 16:46 — with GitHub Actions Inactive
@raffifu raffifu changed the title [WIP] Fix only create task group and test task only if model has test Fix only create task group and test task only if model has test Sep 18, 2023
@raffifu raffifu marked this pull request as ready for review September 18, 2023 16:47
@raffifu raffifu requested a review from a team as a code owner September 18, 2023 16:47
@raffifu raffifu requested a review from a team September 18, 2023 16:47
@raffifu raffifu temporarily deployed to external September 18, 2023 21:41 — with GitHub Actions Inactive
@codecov
Copy link

codecov bot commented Sep 18, 2023

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.04% 🎉

Comparison is base (502051c) 92.37% compared to head (b86d5fd) 92.41%.

❗ Current head b86d5fd differs from pull request most recent head bce3fcf. Consider uploading reports for the commit bce3fcf to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #543      +/-   ##
==========================================
+ Coverage   92.37%   92.41%   +0.04%     
==========================================
  Files          49       49              
  Lines        1941     1952      +11     
==========================================
+ Hits         1793     1804      +11     
  Misses        148      148              
Files Changed Coverage Δ
cosmos/airflow/graph.py 100.00% <100.00%> (ø)
cosmos/dbt/graph.py 98.76% <100.00%> (+0.08%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks amazing, @raffifu , thank you very much for your contribution!
I added two minor comments, and we can aim to release this as part of 1.2.

cosmos/airflow/graph.py Outdated Show resolved Hide resolved
cosmos/airflow/graph.py Outdated Show resolved Hide resolved
@tatiana tatiana added this to the 1.2.0 milestone Sep 19, 2023
@raffifu raffifu temporarily deployed to external September 23, 2023 03:44 — with GitHub Actions Inactive
@tatiana tatiana merged commit d70e6d7 into astronomer:main Sep 25, 2023
19 of 20 checks passed
tatiana added a commit that referenced this pull request Sep 27, 2023
Bug fixes

* Fix using ``ExecutionMode.KUBERNETES`` by @pgoslatara and @tatiana in #554
* Add support to ``apache-airflow-providers-cncf-kubernetes < 7.4.0`` by @tatiana in #553
* Improve error message in ``config.py`` by @meyobagero in #532
* Use ``returncode`` instead of ``stderr`` to determine dbt graph loading errors by @cliff-lau-cloverhealth in #547
* Fix only create task group and test task only if model has test by @raffifu in #543
* Fix ``on_warning_callback`` behaviour on ``DbtTestLocalOperator`` by @edgga, @marco9663 and @tatiana in #558
* Fix ``DbtTestOperator`` when test does not have ``test_metadata`` by @tatiana in #558
* Fix ``target-path`` not specified issue in ``dbt-project.yml`` by @tatiana in #533

Others

* Docs: add reference links to dbt and Airflow columns by @TJaniF in #542
* pre-commit updates #552 and #546
tatiana added a commit that referenced this pull request Sep 27, 2023
…st (#543)"

This is a temporary measurement for the 1.1.2 release until #561 is addressed.

This reverts commit d70e6d7.
tatiana pushed a commit that referenced this pull request Sep 28, 2023
…CUSTOM) (#563)

This PR fixes parsing test nodes when using LoadMethod.CUSTOM, since it didn't return any test nodes.
This issue surfaced after #543.

Closes: #561
tatiana added a commit that referenced this pull request Sep 28, 2023
…st (#543)"

This is a temporary measurement for the 1.1.2 release until #561 is addressed.

This reverts commit d70e6d7.
tatiana pushed a commit that referenced this pull request Sep 28, 2023
…CUSTOM) (#563)

This PR fixes parsing test nodes when using LoadMethod.CUSTOM, since it didn't return any test nodes.
This issue surfaced after #543.

Closes: #561
(cherry picked from commit 0123c3e)
tatiana pushed a commit that referenced this pull request Sep 28, 2023
This PR will only create task group if `dbt` test exist by adding
propery `has_test` on `DbtNode`. The test dependency will be updated
after loading the dbt project into `DbtGraph`

Closes: #531
tatiana added a commit that referenced this pull request Sep 28, 2023
Bug fixes

* Only create task group and test task only if the model has a test by @raffifu in #543
* Fix parsing test nodes when using the custom load method (LoadMethod.CUSTOM) by @raffifu in #563
* Fix ``DbtTestOperator`` when test does not have ``test_metadata`` by @javihernovoa and @tatiana in #565
* Support dbt 1.6 and apache-airflow-providers-cncf-kubernetes 7.3.0  by @tatiana in #564
@tatiana tatiana mentioned this pull request Sep 28, 2023
tatiana added a commit that referenced this pull request Sep 28, 2023
Bug fixes

* Only create task group and test task only if the model has a test by @raffifu in #543
* Fix parsing test nodes when using the custom load method (LoadMethod.CUSTOM) by @raffifu in #563
* Fix ``DbtTestOperator`` when test does not have ``test_metadata`` by @javihernovoa and @tatiana in #565
* Support dbt 1.6 and apache-airflow-providers-cncf-kubernetes 7.3.0  by @tatiana in #564
@tatiana
Copy link
Collaborator

tatiana commented Sep 29, 2023

tatiana added a commit that referenced this pull request Sep 29, 2023
Bug fixes
    
* Only create task group and test task only if the model has a test by
@raffifu in #543
* Fix parsing test nodes when using the custom load method
(`LoadMethod.CUSTOM`) by @raffifu in #563
* Fix `DbtTestOperator` when test does not have ``test_metadata`` by
@javihernovoa and @tatiana in #565
* Support dbt 1.6 and `apache-airflow-providers-cncf-kubernetes` 7.3.0
by @tatiana in #564
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Do not use task group and create a test task for a model that does not have tests
2 participants