Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate dbt resource identifier to see if it is a valid airflow task id during conversion from dbt to Airflow #657

Closed
tatiana opened this issue Nov 8, 2023 · 4 comments
Labels
area:parsing Related to parsing DAG/DBT improvement, issues, or fixes area:rendering Related to rendering, like Jinja, Airflow tasks, etc enhancement New feature or request
Milestone

Comments

@tatiana
Copy link
Collaborator

tatiana commented Nov 8, 2023

While parsing the dbt project, we should confirm if the node names we'll use to define Airflow task IDs are valid.
If they are not valid, we should at least raise a warning (probably an error message) explaining the issue to users.

We can accomplish this by using airflow.utils.helpers.validate_key.

This task is related to #636, but they are not duplicates and should not conflict.

This improvement came from a conversation with @MrBones757 , regarding Cosmos 1.2.2. More details:

When we create dbtNode objects in https://github.com/astronomer/astronomer-cosmos/blob/main/cosmos/dbt/graph.py#L367

We set the nodeName == alias, with a fallback to name if it doesn't exist.

We then create the airflow graph with that value here: https://github.com/astronomer/astronomer-cosmos/blob/main/cosmos/airflow/graph.py#L125

The issue: dbt doesn't impose limitations on node names, but airflow does as such. When we go to render the airflow dag in the UI, we get:

The key 'my$cool_alias' has to be made of alphanumeric characters, dashes, dots and underscores exclusively
@tatiana tatiana added the enhancement New feature or request label Nov 8, 2023
@tatiana
Copy link
Collaborator Author

tatiana commented Nov 8, 2023

For some reason, I'm not being able to assign this issue to @MrBones757 in the Github UI, but he is planning to work on this!

@tatiana tatiana added this to the 1.3.0 milestone Nov 8, 2023
@tatiana tatiana added area:parsing Related to parsing DAG/DBT improvement, issues, or fixes area:rendering Related to rendering, like Jinja, Airflow tasks, etc labels Nov 8, 2023
@MrBones757
Copy link
Contributor

Before implementing this, i will wait on #636 to ensure there are no clashing changes.

@tatiana
Copy link
Collaborator Author

tatiana commented Nov 17, 2023

@MrBones757 #662 has a first level of "validation" (conversion from dots to underscore)

@tatiana tatiana modified the milestones: 1.3.0, 1.4.0 Dec 7, 2023
@dosubot dosubot bot added the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Mar 9, 2024
Copy link

dosubot bot commented Mar 9, 2024

Hi, @tatiana,

I'm helping the Cosmos team manage their backlog and am marking this issue as stale. The issue involves validating dbt resource identifiers to ensure they comply with Airflow task ID requirements during the conversion process. It seems that there are plans to work on this, with @MrBones757 expressing intent to address it and waiting for progress on another related issue to avoid conflicts. Additionally, there's a preliminary level of "validation" related to this issue in another open matter (#662).

Could you please confirm if this issue is still relevant to the latest version of the Cosmos repository? If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your understanding and cooperation. If you have any questions or need further assistance, feel free to reach out to me directly.

@dosubot dosubot bot closed this as not planned Won't fix, can't repro, duplicate, stale Mar 16, 2024
@dosubot dosubot bot removed the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Mar 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:parsing Related to parsing DAG/DBT improvement, issues, or fixes area:rendering Related to rendering, like Jinja, Airflow tasks, etc enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants