Skip to content
New issue

Cosmos 1.6.0

Closed Aug 20, 2024 100% complete

Many users from OSS and Astronomer raised issues related to disk utilization. In the past (until Cosmos 1.3.x), these were exclusive to LoadMode.VIRTUALENV (#958). Still, more recently (since Cosmos 1.4.x), with the introduction of local cache, we intentionally started caching in the local temporary directory, leading to issues (#1042). From an Astro clou…

Many users from OSS and Astronomer raised issues related to disk utilization. In the past (until Cosmos 1.3.x), these were exclusive to LoadMode.VIRTUALENV (#958). Still, more recently (since Cosmos 1.4.x), with the introduction of local cache, we intentionally started caching in the local temporary directory, leading to issues (#1042). From an Astro cloud perspective, the introduction of Ephemeral storage has caused this issue to become even more evident. For this reason, actions that aim to reduce the local cache are a priority in 1.6.

We aim to address these by closing the following tickets, which we consider critical:

  • Support caching remotely (#927, #894)
  • Offer tooling for users to delete stale cache from disk #1078
  • [Bug] Caching filling up Airflow nodes disk #1042
  • [Bug] /tmp/ File Not Found Error Causing Task Failure for dbt Cosmos Tasks #1075

In the meantime, as of Cosmos 1.5, we introduced DBT_LS cache into Airflow variables. We have customers using this feature in production, but some of them would like to store this cache in a remote location. We agreed with them to address this by 1.6:

  • Allow storing dbt ls cache into Object Store #1072

One of the most popular community and customer requests is:

  • Support loading manifest from S3/GCS/Azure BlobStorage #448
    Which will help simplify users' CI/CD pipelines significantly. We're aiming to implement this feature as well.

Finally, another extremely popular request has been for Cosmos to have a default way to handle dbt Source nodes. There are several open tickets related to this, and also a PR from a contributor who happens to be an Astronomer customer:

Now is the right time to talk about this.

Our integration tests do not run DAGs using the ExecutionMode.KUBERNETES. This means we rely on human testing, which sometimes may be forgotten and can incur into issues. The following ticket aims to address this:
#535

Finally, there are some other tasks that we're aiming to wrap up:

  • Support persisting the LoadMode.VIRTUALENV directory (#611)
  • New TestBehavior.BUILD (#892)
  • Support "standalone" test tasks when they are not attached to models, seeds, or snapshots #959
  • Cache dbt deps lock file #930
Loading