Add helper functions for uploading target directory artifacts to remote cloud storages

[pankajkoti]

This PR introduces helper functions that can be passed as callable callbacks for Cosmos tasks to execute post-task execution. These helper functions enable the uploading of artifacts (from the project's target directory) to various cloud storage providers, including AWS S3, Google Cloud Storage (GCS), Azure WASB, and general remote object stores using Airflow’s ObjectStoragePath.

Key Changes

Adds a cosmos/io.py module that includes the following helper functions

1. upload_artifacts_to_aws_s3

   • Uploads artifact files from a task’s local target directory to an AWS S3 bucket.
   • Supports dynamically appending DAG metadata (e.g., dag_id, task_id, run_id, and try number) to the uploaded file paths.
   • Utilizes S3Hook from the airflow.providers.amazon.aws module.

2. upload_artifacts_to_gcp_gs

   • Uploads artifact files from a task’s local target directory to a Google Cloud Storage (GCS) bucket.
   • Appends DAG-related context to the GCS object names for better traceability.
   • Leverages GCSHook from airflow.providers.google.cloud.

3. upload_artifacts_to_azure_wasb

   • Uploads artifact files from a task’s local target directory to an Azure Blob Storage container.
   • Automatically structures blob names with metadata, including dag_id, task_id, and execution details.
   • Utilizes WasbHook from the airflow.providers.microsoft.azure module.

4. upload_artifacts_to_cloud_storage

   • A generic helper function that uploads artifacts from a task’s local target directory to remote object stores configured via Airflow’s ObjectStoragePath (Airflow 2.8+ feature).
   • Supports custom remote storage configurations such as remote_target_path and remote_target_path_conn_id.
   • Dynamically constructs file paths that include DAG metadata for clear organization.

These helpers functions can be passed as the callback argument to DbtDAG or to your Dag instance as demonstrated in the example DAGs dev/dags/cosmos_callback_dag.py and dev/dags/example_operators.py correspondingly. You can also pass callback_args as shown in the example DAGs. These helper functions are mere examples of how callback functions can be written and passed to your operators/DAGs to be executed after task completions. Taking reference of these helper functions, you can write your own callback function and pass those.

Limitations

1. This PR has been tested and is currently supported only in ExecutionMode.LOCAL. We encourage the community to contribute by adding callback support for other execution modes as needed, using the implementation for ExecutionMode.LOCAL as a reference.

closes: #1350
closes: #976
closes: #867
closes: #801
closes: #1292
closes: #851
closes: #1351
related: #1293
related: #1349

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[cloudflare-workers-and-pages]

Deploying astronomer-cosmos with    Cloudflare Pages

Latest commit:	ae83d6b
Status:	✅  Deploy successful!
Preview URL:	https://46387f83.astronomer-cosmos.pages.dev
Branch Preview URL:	https://artifacts-upload-helpers.astronomer-cosmos.pages.dev

View logs

[tatiana]

Hi @pankajkoti , it's amazing to see the progress here, we'll be able to close almost 10 tickets of our backlog. A minor feedback, while this is in draft mode

[codecov]

Codecov Report

Attention: Patch coverage is 98.75000% with 1 line in your changes missing coverage. Please review.

Project coverage is 96.28%. Comparing base (cbd8622) to head (ae83d6b).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
cosmos/io.py	98.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1389      +/-   ##
==========================================
+ Coverage   96.21%   96.28%   +0.07%     
==========================================
  Files          67       68       +1     
  Lines        4071     4149      +78     
==========================================
+ Hits         3917     3995      +78     
  Misses        154      154              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tatiana]

Thank you very much for addressing all the feedback, @pankajkoti !

This is probably a breaking record for a Cosmos PR: to close seven tickets in one go. Really excited to share this with the community in 1.8