Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mention in documentation that the callback functionality is supported in ExecutionMode.VIRTUALENV #1401

Merged
merged 2 commits into from
Dec 18, 2024
Merged

Mention in documentation that the callback functionality is supported in ExecutionMode.VIRTUALENV #1401

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
042551b
Mention in docs callback support for ExecutionMode.VIRTUALENV
pankajkoti Dec 18, 2024
012b191
Add comment on why callback functions are commented in the DAG
pankajkoti Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions dev/dags/example_virtualenv.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,19 @@ def example_virtualenv() -> None:
"py_requirements": ["dbt-postgres"],
"install_deps": True,
"emit_datasets": False, # Example of how to not set inlets and outlets
# --------------------------------------------------------------------------
pankajkoti marked this conversation as resolved.
Show resolved Hide resolved
# For the sake of avoiding additional latency observed while uploading files for each of the tasks, the
# below callback functions to be executed are commented, but you can uncomment them if you'd like to
# enable callback execution.
# Callback function to upload files using Airflow Object storage and Cosmos remote_target_path setting on
# Airflow 2.8 and above
# "callback": upload_to_cloud_storage,
# --------------------------------------------------------------------------
# Callback function if you'd like to upload files from the target directory to remote store e.g. AWS S3 that
# works with Airflow < 2.8 too
# "callback": upload_to_aws_s3,
# "callback_args": {"aws_conn_id": "aws_s3_conn", "bucket_name": "cosmos-artifacts-upload"}
# --------------------------------------------------------------------------
},
)

Expand Down
9 changes: 5 additions & 4 deletions docs/configuration/callbacks.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
Callbacks
=========

Cosmos supports callback functions that execute at the end of a task's execution when using ``ExecutionMode.LOCAL``.
Cosmos supports callback functions that execute at the end of a task's execution when using ``ExecutionMode.LOCAL`` and
``ExecutionMode.VIRTUALENV``.
These callbacks can be used for various purposes, such as uploading files from the target directory to remote
storage. While this feature has been available for some time, users may not be fully aware of its capabilities.

Expand Down Expand Up @@ -46,6 +47,6 @@ metadata.
Limitations and Contributions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Currently, callback support is available only when using ``ExecutionMode.LOCAL``. Contributions to extend this
functionality to other execution modes are welcome and encouraged. You can reference the implementation for
``ExecutionMode.LOCAL`` to add support for other modes.
Currently, callback support is available only when using ``ExecutionMode.LOCAL`` and ``ExecutionMode.VIRTUALENV``.
Contributions to extend this functionality to other execution modes are welcome and encouraged. You can reference the
implementation for ``ExecutionMode.LOCAL`` to add support for other modes.
Loading