[FEATURE] make pypi recipes more robust by using the JSON API for retrieving download URL

[dlaehnemann]

Is your feature request related to a problem? Please describe.

The following motivating example is one way that auto-generation of pypi recipe urls recently broke for me, but I guess other things might break this in the future (for example, URL redirects might change or break, or be discontinued, or ...):

poetry recently moved to enforcing that sdist files (tar.gz) created by its build command conform to the respective naming specs:
The sdist naming spec points to the wheel / binary naming specs, which state that all -_. symbols are to be replaced by _ (underscore).

For packages that contain - or . in their package name and are packaged via poetry for upload to pypi, this means that their package name and their sdist file name will differ and the standard pypi.io link generated by grayskull will not work (and the respective (bio-)conda packages silently broke). Take this example:

{% set name = "snakemake-wrapper-utils" %}
{% set version = "0.5.2" %}

source:
  url: "https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz"

Instead, this now has to be:

{% set name = "snakemake-wrapper-utils" %}
{% set version = "0.5.2" %}

source:
  url: "https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name|replace("-", "_") }}-{{ version }}.tar.gz"

Also see this respective pull request in the package used as an example here:
bioconda/bioconda-recipes#39979

Describe the solution you'd like

The official guidance of pypi for determining download URLs is to "query PyPI’s JSON API". Thus, I would like to suggest to make this the new default behaviour for pypi packages, in the hope that this will make the resulting recipes more robust.

In the above example, this would lead to:

{% set name = "snakemake-wrapper-utils" %}
{% set version = "0.5.2" %}

source:
  url: "https://files.pythonhosted.org/packages/52/e1/449dcfcba437f9bda6cd232664182dcf7acee78c3fe77a238078c84ec7b7/snakemake_wrapper_utils-0.5.2.tar.gz"

The auto-update bots for conda-forge and bioconda would need to be updated accordingly, to also query the pypi JSON API for the url. As a result, version update commits would have a larger footprint of change in the URL, but everything should nevertheless be automated.

Describe alternatives you've considered

The above example gives a workaround solution, using the jinja2 filter |replace("-","_"). However, this is not a generic solution and instead of covering such a specific case with code in grayskull, I would instead suggest to go for the more generic solution via the JSON API.

Additional context

Code that might be helpful in the implementation of this can be found here:

• a stackoverflow answer giving example python code for querying the pypi JSON API for a download URL
• this link to how the pypi redirect links are generated, using the JSON API

[bollwyvl]

See also #460.

[marcelotrevisani]

For now I would rather go for the jinja replace, because if I put the full url to the python hosted url that will generate more problems in Conda-forge.

@ocefpaf