-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix installing deps when using profile_mapping
& ExecutionMode.LOCAL
#659
Fix installing deps when using profile_mapping
& ExecutionMode.LOCAL
#659
Conversation
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was super quick, @joppevos , thank you for fixing this issue.
The only thing missing is a test. I believe we could add a unit test in astronomer-cosmos/tests/operators/test_local.py
, calling run_command
and mocking run_subprocess
, and checking if we're passing the expected arguments.
There is something similar in:
astronomer-cosmos/tests/operators/test_local.py
Lines 369 to 373 in 188fe56
@patch("cosmos.operators.local.DbtLocalBaseOperator.build_and_run_cmd") | |
def test_operator_execute_with_flags(mock_build_and_run_cmd, operator_class, kwargs, expected_call_kwargs): | |
task = operator_class(profile_config=profile_config, task_id="my-task", project_dir="my/dir", **kwargs) | |
task.execute(context={}) | |
mock_build_and_run_cmd.assert_called_once_with(**expected_call_kwargs) |
The motivation to have a test is to make sure we don't re-introduce this bug in the future.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #659 +/- ##
==========================================
+ Coverage 92.71% 92.72% +0.01%
==========================================
Files 54 54
Lines 2223 2227 +4
==========================================
+ Hits 2061 2065 +4
Misses 162 162
☔ View full report in Codecov by Sentry. |
profile_mapping
& ExecutionMode.LOCAL
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for working on this and addressing the feedback so quickly, @joppevos !
We'll merge it once the checks pass.
…AL` (#659) Extends the local operator when running `dbt deps` with the provides profile flags. This makes the logic consistent between DAG parsing and task running as referenced below https://github.com/astronomer/astronomer-cosmos/blob/8e2d5908ce89aa98813af6dfd112239e124bd69a/cosmos/dbt/graph.py#L247-L266 Closes: #658 (cherry picked from commit 6e41471)
Bug fixes * Store `compiled_sql` even when task fails by @agreenburg in #671 * Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in #660 * Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666 * Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in #659 Others * Docs fix: add execution config to MWAA code example by @ugmuka in #674
Bug fixes * Store `compiled_sql` even when task fails by @agreenburg in #671 * Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in #660 * Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666 * Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in #659 Others * Docs fix: add execution config to MWAA code example by @ugmuka in #674 (cherry picked from commit aa9b7bb)
…AL` (#659) Extends the local operator when running `dbt deps` with the provides profile flags. This makes the logic consistent between DAG parsing and task running as referenced below https://github.com/astronomer/astronomer-cosmos/blob/8e2d5908ce89aa98813af6dfd112239e124bd69a/cosmos/dbt/graph.py#L247-L266 Closes: #658 (cherry picked from commit 6e41471)
Bug fixes * Store `compiled_sql` even when task fails by @agreenburg in #671 * Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in #660 * Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666 * Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in #659 Others * Docs fix: add execution config to MWAA code example by @ugmuka in #674
Thanks for the contribution, @joppevos , we released it as part of Cosmos 1.2.4 yesterday: |
**Bug fixes** * Store `compiled_sql` even when task fails by @agreenburg in #671 * Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in #660 * Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666 * Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in #659 **Others** * Docs fix: add execution config to MWAA code example by @ugmuka in #674 (cherry picked from commit aa9b7bb)
…AL` (astronomer#659) Extends the local operator when running `dbt deps` with the provides profile flags. This makes the logic consistent between DAG parsing and task running as referenced below https://github.com/astronomer/astronomer-cosmos/blob/8e2d5908ce89aa98813af6dfd112239e124bd69a/cosmos/dbt/graph.py#L247-L266 Closes: astronomer#658
**Bug fixes** * Store `compiled_sql` even when task fails by @agreenburg in astronomer#671 * Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in astronomer#660 * Fix 'Unable to find the dbt executable: dbt' error by @tatiana in astronomer#666 * Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in astronomer#659 **Others** * Docs fix: add execution config to MWAA code example by @ugmuka in astronomer#674 (cherry picked from commit aa9b7bb)
Description
Extends the local operator when running
dbt deps
with the provides profile flags.This makes the logic consistent between DAG parsing and task running as referenced below
astronomer-cosmos/cosmos/dbt/graph.py
Lines 247 to 266 in 8e2d590
Related Issue(s)
closes #658
Breaking Change?
Checklist