Pydantic

[ykuc]

Hi!
Why does astronomer-cosmos package have pydantic in dependencies?
I didn’t find it in the code, but airflow 2.7.2 upgrade to Pydantic 2

[ykuc]

https://apache-airflow.slack.com/archives/C059CC42E9W/p1699029324253389

[tatiana]

@ykuc Thanks for raising this!

This dependency was introduced 4 months ago and documented in the code:

astronomer-cosmos/pyproject.toml

Line 37 in 1fb3e23

# Airflow & Pydantic issue: https://github.com/apache/airflow/issues/32311

8cd9ff3

It was originally introduced because of this Airflow issue:
apache/airflow#32311

We'll remove it once our checks pass, thanks @jlaneve for the quick fix!

[tatiana]

@ykuc @jlaneve It seems the Airflow 2.6 tests are failing if we remove the Pydantic dep:
#654
https://github.com/astronomer/astronomer-cosmos/actions/runs/6776766746/job/18418976543?pr=654

= 23 failed, 152 passed, 18 deselected, 2 xfailed, 1 warning, 75 errors in 21.08s =

We could consider only installing this version of Pydantic for this particular version of Airflow when running its tests.

[bolkedebruin]

This should indeed be a conditional dependency or preferably a constraint. This issue was resolved in Airflow since quite some time and Airflow itself is moving rapidly to Pydantic 2. Including cosmos downgrades pydantic at the moment while it doesn't have any dependencies on it.

[pixie79]

Is there any progress on this, it would appear MWAA with 2.7.2 is failing with astronomer-cosmos due to constraints on needing pydantic 2.4.2 which is conflicting with the lock in cosmos.

[seeholza]

I came here for the same reason as @pixie79, since including cosmos pydantic is downgraded to <2, which breaks other dependencies for me (pydanctic >= 2, on airflow 2.7.* and airflow 2.8.0).

Glad to see its going to be included in 1.3.0 via #654 -- Thanks so much @jlaneve ! I'll be working off v1.3.0a3 in the meanwhile.

[dosubot]

Hi, @ykuc,

I'm helping the Cosmos team manage their backlog and am marking this issue as stale. From what I understand, you raised concerns about the presence of Pydantic in the dependencies of the astronomer-cosmos package, which was causing issues with Airflow 2.7.2. The maintainers have acknowledged the issues caused by the upgrade to Pydantic 2 and have discussed the need for conditional dependency or constraint. They have also expressed gratitude for the upcoming inclusion of Pydantic in version 1.3.0, indicating that the issue is being addressed in the next release.

Could you please confirm if this issue is still relevant to the latest version of the Cosmos repository? If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the Cosmos repository.

Dosu