[Feature] Store dbt artifacts (namely, manifest.json, catalog.json and run_results.json) after every dbt run.

[victormacaubas]

Description

Hi all,

I’ve noticed some similar requests, but I wanted to ask if it’s possible to persist dbt artifacts (namely, manifest.json, catalog.json and run_results.json) permanently after every run, regardless of whether we use TestBehavior.AFTER_ALL or TestBehavior.AFTER_EACH.

Use case/motivation

I’d like to be able to send these run results to both Metaplane and Atlan for data observability analytics.

Related issues

#1253
#801

Are you willing to submit a PR?

• Yes, I am willing to submit a PR!

[pankajkoti]

hi @victormacaubas , thanks for requesting this feature. Would it help if we start uploading these to a remote_target_path? We already introduced this config in PR #1224 but at the moment we only upload files in the target -> compileddirectory to the remote_target_path & only in case of ExecutionMode.AIRFLOW_ASYNC. We just logged an issue #1293 and would appreciate if you'd have any comment there or here highlighting how these could help you & if we start supporting these what files are more particularly of importance. Would also be nice if you would have the time to help contribute supporting one or two of these :)

[victormacaubas]

Hello @pankajkoti

It would be even better if these artifacts could be persisted in a remote_target_path, like S3! I’m currently running in ExecutionMode.LOCAL and using the parsing method with manifest.json (since my project is quite large, using dbt_ls was problematic). I need to send these results to Atlan and Metaplane for alerting and data observability, and storing them externally would be ideal.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[darennathanielviki]

Hello @pankajkoti, I would also like to propose to have this feature on ExecutionMode.KUBERNETES. We would like to upload run_results.json file to Datahub for observability of the data quality. On top of that, since dbt test is ran per model basis in Cosmos, would be great to have the run_results to be based off of the model name, something like run_results_<model_name> perhaps?

[tatiana]

@victormacaubas @darennathanielviki, to give visibility, we're actively working on this task, and we should release it by Cosmos 1.8. Since different flavours of this feature have been requested, we've created a label to track them:
https://github.com/astronomer/astronomer-cosmos/issues?q=is%3Aissue%20state%3Aopen%20label%3Aexecution%3Acallback

Once this is out there, we'd love feedback. Thanks for your patience.

[victormacaubas]

@tatiana thank you for your work on this! excited to see it live

[pankajkoti]

hi @victormacaubas , we recently merged PR #1389, which introduces minor changes to the existing callback functionality and will be included in the upcoming Cosmos 1.8.0 release.

To allow users to try out these changes ahead of the official release, we have prepared an alpha release. You can install it using the following link: astronomer-cosmos 1.8.0a3. PR #1389 also provides examples showcasing how to use this callback functionality.

For additional guidance, refer to the documentation on leveraging callbacks: Callback Configuration. The helper functions demonstrated in the examples can be found here: cosmos/io.py. However, you are not limited to these; you can create your own custom callback functions using these examples as a reference and pass them via the callback argument.

We would greatly appreciate any feedback you have after testing this alpha release!

[victormacaubas]

Howdy! That's great, I'll look into it and see if we can test this out next sprint =) thank you all