Failed to update dependencies via bot due to transitive pre-release dependency

[tumido]

Describe the bug
We have a dependency installed via Pipenv. This dependency depends on a package which has only pre-releases. We don't want to specify allow_prereleases = true in our Pipfile since that would turn pre-release resolution globally for all dependencies and we don't want that.

Notes:

• We have no control over the required transitive dependency version.
• This dependency without a proper release is not specified in our Pipfile at all, therefore we can't use the "lock to a specific pre-release trick" as described here new 'pre' syntax in Pipfile pypa/pipenv#1760 (comment)

To Reproduce
Steps to reproduce the behavior:

1. Have a Pipfile which installs a dependency that depends on a pre-release itself
2. Lock it locally using pipenv lock --pre
3. Commit
4. Let bots auto update dependencies - it fails

In our case (aicoe-aiops/mailing-list-analysis-toolkit#36) the troubled dependencies are:

• papermill which is our direct dependency
• black which is a dependency of papermill

Expected behavior
Auto updater should be able to resolve pre-releases if they were already resolved to a pre-release for packages in Pipfile.lock file. This is different to allowing to resolve all pre-releases for all packages (the allow_prereleases flag).

Screenshots
n/a

Additional context
aicoe-aiops/mailing-list-analysis-toolkit#36
pypa/pipenv#1760
pypa/pipenv#965

[fridex]

This is something Pipenv community complains about as well. I understand the use of this and the consequences it brings (some software stacks are not resolvable at all). We can open this discussion in Pipenv community. If they provide a "standard way" how to approach this (e.g. in pipfile, specify acceptable pre-releases for a package, similarly as specifying the index to be used to install the package) we are happy to include it in our recommendations - it's not that big change on our side.

[tumido]

There's this upstream issue, suggesting a per-package pre-release spec Pipfile syntax extension
pypa/pipenv#1760

I've faced the same issue already, but in that case it was our direct dependency so the workaround to locking it to an exact pre-release version worked in that case. However in this case, when it's a transitive dependency I'm clueless... Like.. the same workaround might help for the time being, but it may lead version mismatches if the transitive dependency version requirement is changed... Which would just postpone the same problem, it's not a real solution.

So my ask is, if there's a possibility for the auto-update job to facilitate this. I don't know, maybe a config flag allowing pre-release resolution on packages which are already locked to a pre-release? Or something like that. I know this is far fetched though, but any idea would be welcomed. We certainly want our stacks to be resolvable by the auto-updater so I'm looking for a way how to achieve that. 🙂

[fridex]

There's this upstream issue, suggesting a per-package pre-release spec Pipfile syntax extension
pypa/pipenv#1760

Proposed pypa/pipenv#4607 based on that. An extension could be created which would act on trasitive packages considered (but I would not bring that to the referenced PEEP).

So my ask is, if there's a possibility for the auto-update job to facilitate this. I don't know, maybe a config flag allowing pre-release resolution on packages which are already locked to a pre-release? Or something like that. I know this is far-fetched though, but any idea would be welcomed. We certainly want our stacks to be resolvable by the auto-updater so I'm looking for a way how to achieve that.

The auto-update bot uses Pipenv under the hood (as of now), so we rely on Pipenv logic. Hopefully, we will soon switch to Thoth based updates in which case we can support whatever will be standardized by Pipenv/Python community. As we offer thamos install, we can bring solutions possibly faster but users of our recommendations will need to use our tooling (instead of Pipenv).

[goern]

Shouldnt the advisor-based update manager of kebechet address this issue?

[fridex]

Shouldnt the advisor-based update manager of kebechet address this issue?

Selectively enable pre-releases for just some packages is not something Pipenv can do. It is possible to turn on enabling pre-relesases globally, for all the packages in the software stack, or turn it off completely. If there would be a standardized way how to enable pre-releases just for some packages in Pipfile (e.g. as described in pypa/pipenv#4607), we can provide such a mechanism. As of now, we are compatible with Pipenv and Pipfile/Pipfile.lock capabilities it offers.

[fridex]

BTW we could introduce [thoth] section to Pipfile, I guess it would work with Pipenv as it ignores unknown entries in the Pipfile. We could introduce some experimental features there (such as this one).

[sesheta]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[fridex]

Thoth now supports enabling pre-releases for certain packages, see:

https://thoth-station.ninja/docs/developers/adviser/experimental_features.html#selectively-enabling-pre-releases

I will close this one, feel free to report any issues if any spotted. Thanks for the report 👍🏻

/close

[sesheta]

@fridex: Closing this issue.

In response to this:

Thoth now supports enabling pre-releases for certain packages, see:

https://thoth-station.ninja/docs/developers/adviser/experimental_features.html#selectively-enabling-pre-releases

I will close this one, feel free to report any issues if any spotted. Thanks for the report 👍🏻

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.