[Bug] Couldn't use virtualenv_dir with Google Cloud Composer

[anai-s]

Astronomer Cosmos Version

Other Astronomer Cosmos version (please specify below)

If "Other Astronomer Cosmos version" selected, which one?

1.5.1

dbt-core version

1.7.18

Versions of dbt adapters

dbt-bigquery==1.7.4
dbt-adapters==1.4.0 (package in composer 2.9.1-airflow 2.9.1 image)

LoadMode

AUTOMATIC

ExecutionMode

VIRTUALENV

InvocationMode

None

airflow version

2.9.1

Operating System

Windows 10

If a you think it's an UI issue, what browsers are you seeing the problem on?

No response

Deployment

Google Cloud Composer

Deployment details

1. PYPI Packages

• astronomer-cosmos==1.5.1
• dbt-bigquery==1.7.4

2. Composer Environment Configuration

• Composer 2 - Image composer-2.9.1-airflow-2.9.1
• location europe-west1
• others configurations are the default one

What happened?

I expected to be able to test virtualenv_dir with Cloud Composer by reproducing the example dag here https://github.com/astronomer/astronomer-cosmos/blob/main/dev/dags/example_virtualenv.py but I got the error below

Relevant log output

Broken DAG (dags/test_dag.py):
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/airflow/gcs/dags/test_dag.py", line 28, in <module>
    execution_config=ExecutionConfig(
                     ^^^^^^^^^^^^^^^^
TypeError: ExecutionConfig.__init__() got an unexpected keyword argument 'virtualenv_dir'

How to reproduce

Upload the following file my_cosmos_dag.py into Composer DAG Folder

Anything else :)?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Contact Details

No response

[dosubot]

Hey, @anai-s! I'm here to assist you with any issues, questions, or contributions. Let's work together to resolve the bug you've encountered.

I found a similar closed issue that might be relevant to your problem. The issue was related to Cosmos 1.4.1 not creating a virtual environment when using ExecutionMode.VIRTUALENV ^[1]. The solution involved configuring the ExecutionConfig class appropriately.

However, in your case, it seems like the virtualenv_dir argument is not recognized by the ExecutionConfig constructor in Astronomer Cosmos version 1.5.1. This suggests that there might be a version compatibility issue or a recent change in the ExecutionConfig class.

To use the virtualenv_dir argument with ExecutionConfig in Google Cloud Composer when using Astronomer Cosmos version 1.5.1 and dbt-core version 1.7.18, you need to set the execution_mode to ExecutionMode.VIRTUALENV and provide the path to the virtual environment directory via the virtualenv_dir parameter. Here is an example configuration:

from cosmos.config import ExecutionConfig, ExecutionMode

execution_config = ExecutionConfig(
    execution_mode=ExecutionMode.VIRTUALENV,
    virtualenv_dir='/path/to/your/virtualenv'
)

This configuration ensures that the specified virtual environment directory is used for execution when the execution mode is set to ExecutionMode.VIRTUALENV ^[2][3][4].

If you continue to encounter issues, please ensure that you are using the correct version of Astronomer Cosmos and that the virtualenv_dir parameter is supported in that version.

To continue talking to Dosu, mention @dosu.

[pankajkoti]

hi @anai-s, the argument virtualenv_dir in ExecutionConfig was added in PR #1079 . This PR was included in the latest release of astronomer-cosmos==1.6.0. So, we suggest you to please upgrade your Cosmos version to the latest released version 1.6.0 and try out. Looking forward to hear how it goes then.

[anai-s]

Hi @pankajkoti , thanks for the quick answer! It fixed my problem.
It would be cool if you specify the version compatibility as you did for the new feature Source Nodes Rendering :)

[pankajkoti]

Thanks @anai-s for confirming.

Yes, in fact, we were missing to document this new param. I have opened a PR to document this #1173 and also specify that this argument is newly added in 1.6