Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix installing deps when using profile_mapping & ExecutionMode.LOCAL #659

Merged
merged 17 commits into from
Nov 10, 2023

Conversation

joppevos
Copy link
Contributor

@joppevos joppevos commented Nov 8, 2023

Description

Extends the local operator when running dbt deps with the provides profile flags.
This makes the logic consistent between DAG parsing and task running as referenced below

self.local_flags = [
"--project-dir",
str(tmpdir),
"--profiles-dir",
str(profile_path.parent),
"--profile",
self.profile_config.profile_name,
"--target",
self.profile_config.target_name,
]
self.log_dir = Path(env.get(DBT_LOG_PATH_ENVVAR) or tmpdir_path / DBT_LOG_DIR_NAME)
self.target_dir = Path(env.get(DBT_TARGET_PATH_ENVVAR) or tmpdir_path / DBT_TARGET_DIR_NAME)
env[DBT_LOG_PATH_ENVVAR] = str(self.log_dir)
env[DBT_TARGET_PATH_ENVVAR] = str(self.target_dir)
if self.render_config.dbt_deps:
deps_command = [self.dbt_cmd, "deps"]
deps_command.extend(self.local_flags)
stdout = run_command(deps_command, tmpdir_path, env)
logger.debug("dbt deps output: %s", stdout)

Related Issue(s)

closes #658

Breaking Change?

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

Copy link

netlify bot commented Nov 8, 2023

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name Link
🔨 Latest commit 58292c7
🔍 Latest deploy log https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/654d08368a19f200086a9a48

@joppevos joppevos marked this pull request as ready for review November 8, 2023 14:18
@joppevos joppevos requested a review from a team as a code owner November 8, 2023 14:18
@joppevos joppevos requested a review from a team November 8, 2023 14:18
Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was super quick, @joppevos , thank you for fixing this issue.

The only thing missing is a test. I believe we could add a unit test in astronomer-cosmos/tests/operators/test_local.py, calling run_command and mocking run_subprocess, and checking if we're passing the expected arguments.

There is something similar in:

@patch("cosmos.operators.local.DbtLocalBaseOperator.build_and_run_cmd")
def test_operator_execute_with_flags(mock_build_and_run_cmd, operator_class, kwargs, expected_call_kwargs):
task = operator_class(profile_config=profile_config, task_id="my-task", project_dir="my/dir", **kwargs)
task.execute(context={})
mock_build_and_run_cmd.assert_called_once_with(**expected_call_kwargs)

The motivation to have a test is to make sure we don't re-introduce this bug in the future.

Copy link

codecov bot commented Nov 8, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (9001c98) 92.71% compared to head (58292c7) 92.72%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #659      +/-   ##
==========================================
+ Coverage   92.71%   92.72%   +0.01%     
==========================================
  Files          54       54              
  Lines        2223     2227       +4     
==========================================
+ Hits         2061     2065       +4     
  Misses        162      162              
Files Coverage Δ
cosmos/operators/local.py 91.74% <100.00%> (+0.09%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tatiana tatiana added this to the 1.3.0 milestone Nov 8, 2023
@tatiana tatiana added the area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc label Nov 9, 2023
@tatiana tatiana changed the title extend deps command with profile flags Fix installing deps when using profile_mapping & ExecutionMode.LOCAL Nov 9, 2023
Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this and addressing the feedback so quickly, @joppevos !
We'll merge it once the checks pass.

@joppevos
Copy link
Contributor Author

Thanks for working on this and addressing the feedback so quickly, @joppevos ! We'll merge it once the checks pass.

Sounds good! Thanks for the support @tatiana :)

@tatiana tatiana modified the milestones: 1.3.0, 1.2.4 Nov 10, 2023
@tatiana tatiana merged commit 6e41471 into astronomer:main Nov 10, 2023
40 checks passed
tatiana pushed a commit that referenced this pull request Nov 15, 2023
…AL` (#659)

Extends the local operator when running `dbt deps` with the provides
profile flags.

This makes the logic consistent between DAG parsing and task running as
referenced below

https://github.com/astronomer/astronomer-cosmos/blob/8e2d5908ce89aa98813af6dfd112239e124bd69a/cosmos/dbt/graph.py#L247-L266

Closes: #658
(cherry picked from commit 6e41471)
tatiana added a commit that referenced this pull request Nov 15, 2023
Bug fixes

* Store `compiled_sql` even when task fails by @agreenburg in #671
* Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in #660
* Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666
* Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in #659

Others

* Docs fix: add execution config to MWAA code example by @ugmuka in #674
@tatiana tatiana mentioned this pull request Nov 15, 2023
tatiana added a commit that referenced this pull request Nov 15, 2023
Bug fixes

* Store `compiled_sql` even when task fails by @agreenburg in #671
* Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in #660
* Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666
* Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in #659

Others

* Docs fix: add execution config to MWAA code example by @ugmuka in #674

(cherry picked from commit aa9b7bb)
@tatiana tatiana mentioned this pull request Nov 15, 2023
tatiana pushed a commit that referenced this pull request Nov 15, 2023
…AL` (#659)

Extends the local operator when running `dbt deps` with the provides
profile flags.

This makes the logic consistent between DAG parsing and task running as
referenced below

https://github.com/astronomer/astronomer-cosmos/blob/8e2d5908ce89aa98813af6dfd112239e124bd69a/cosmos/dbt/graph.py#L247-L266

Closes: #658
(cherry picked from commit 6e41471)
tatiana added a commit that referenced this pull request Nov 15, 2023
Bug fixes

* Store `compiled_sql` even when task fails by @agreenburg in #671
* Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in #660
* Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666
* Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in #659

Others

* Docs fix: add execution config to MWAA code example by @ugmuka in #674
@tatiana
Copy link
Collaborator

tatiana commented Nov 16, 2023

Thanks for the contribution, @joppevos , we released it as part of Cosmos 1.2.4 yesterday:
https://github.com/astronomer/astronomer-cosmos/releases/tag/astronomer-cosmos-v1.2.4

tatiana added a commit that referenced this pull request Nov 16, 2023
**Bug fixes**

* Store `compiled_sql` even when task fails by @agreenburg in #671
* Refactor `LoadMethod.LOCAL` to use symlinks instead of copying
directory by @jbandoro in #660
* Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666
* Fix installing deps when using `profile_mapping` &
`ExecutionMode.LOCAL` by @joppevos in #659

**Others**

* Docs fix: add execution config to MWAA code example by @ugmuka in #674

(cherry picked from commit aa9b7bb)
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
…AL` (astronomer#659)

Extends the local operator when running `dbt deps` with the provides
profile flags.

This makes the logic consistent between DAG parsing and task running as
referenced below

https://github.com/astronomer/astronomer-cosmos/blob/8e2d5908ce89aa98813af6dfd112239e124bd69a/cosmos/dbt/graph.py#L247-L266

Closes: astronomer#658
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
**Bug fixes**

* Store `compiled_sql` even when task fails by @agreenburg in astronomer#671
* Refactor `LoadMethod.LOCAL` to use symlinks instead of copying
directory by @jbandoro in astronomer#660
* Fix 'Unable to find the dbt executable: dbt' error by @tatiana in astronomer#666
* Fix installing deps when using `profile_mapping` &
`ExecutionMode.LOCAL` by @joppevos in astronomer#659

**Others**

* Docs fix: add execution config to MWAA code example by @ugmuka in astronomer#674

(cherry picked from commit aa9b7bb)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix running install_deps=True when using profile_mapping and ExecutionMode.LOCAL
3 participants