datahub dependencies are staged incorrectly

[ryanlovett]

Bug description

Python packages from environment.yml, infra-requirements.txt, and requirements.txt are installed in different stages in Dockerfile. As a result, a package included in, and installed by environment.yml may be upgraded by requirements.txt later, even if it is not explicitly included in the latter.

For example numpy is set to 1.21.* in environment.yml but is being upgraded to 1.25 when requirements.txt is evaluated. We should integrate infra-requirements.txt and requirements.txt directly into environment.yml so that the resolver sees everything at once.

Environment & setup

• Hub: datahub
• Language: python

How to reproduce

1. Read environment.yml
2. Check the version of numpy installed in datahub
3. Look at CI and see how numpy is being upgraded later.

[shaneknapp]

yeah, i was thinking the same thing on friday... super annoying.

…

On Sat, Oct 7, 2023 at 5:57 PM Ryan Lovett ***@***.***> wrote: Bug description Python packages from environment.yml, infra-requirements.txt, and requirements.txt are installed in different stages in Dockerfile. As a result, a package included in, and installed by environment.yml may be upgraded by requirements.txt later, even if it is not explicitly included in the latter. For example numpy is set to 1.21.* in environment.yml but is being upgraded to 1.25 when requirements.txt is evaluated. We should integrate infra-requirements.txt and requirements.txt directly into environment.yml so that the resolver sees everything at once. Environment & setup - Hub: datahub - Language: python How to reproduce 1. Read environment.yml 2. Check the version of numpy installed in datahub 3. Look at CI and see how numpy is being upgraded later. — Reply to this email directly, view it on GitHub <#5117>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMIHLCVBQHPMXLAUY67WP3X6H3BPAVCNFSM6AAAAAA5XJINYSVHI2DSMVQWIX3LMV43ASLTON2WKOZRHEZTCNJXHEZDGNI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[shaneknapp]

fwiw this just bit us (me) hard on the publichealth hub... #5190