Improve setup.py parsing

[andrew]

Find dependencies when specified as an array:

required_packages = [
    'coverage==3.7.1',
    'Mako==1.0.3',
    'PyYAML==3.11',
]

[brainwane]

Hey there -- since I'm working on the new PyPI and getting it closer and closer to launch, I wanted to check whether you're planning on using the API instead of parsing setup.py -- or perhaps I am misunderstanding your needs?

[andrew]

@brainwane thanks for getting in touch, at the moment we don't want to execute any arbitrary python code so parsing with a regex is the best way for us to handle it until we have a safe way of using each package managers tooling directly, then we'll use the API, thanks!

[andrew]

We are using the pypi rest api in the main Libraries.io codebase over here: https://github.com/librariesio/libraries.io/blob/master/app/models/package_manager/pypi.rb but that doesn't help for this project which parses any setup.py, usually from applications rather than pypi modules.

[brainwane]

https://github.com/pypa/twine/blob/fb062b77c7ef38c28430dd4c8edaa592c4b59b07/setup.py#L21:L27 is an example here:

install_requires = [
    "tqdm >= 4.14",
    "pkginfo >= 1.0",
    "requests >= 2.5.0, != 2.15, != 2.16",
    "requests-toolbelt >= 0.8.0",
    "setuptools >= 0.7.0",
]

[andrew]

@brainwane this library is also used for parsing dependency information from repositories that aren't pushed to pypi, including public and private github repositories, forks etc

[brainwane]

Could I please ask for this and the other PyPI issues -- #414, #459, #415 -- to get higher priorities on Tidelift's roadmap? The current gaps in coverage leave a lot of real dependencies untracked.

[havocp]

@brainwane noted!

One thing to be sure is clear to anyone who might stumble on the discussion, these are all for repository dependencies, so https://libraries.io/github/pytest-dev/pytest vs package deps like https://libraries.io/pypi/pytest/5.0.1/tree ... hopefully we're getting the package deps right.

Team Tidelift is very development-team-size-constrained right now, to set expectations, but I do appreciate the bump on these and we've added them to our internal backlog too.

[abitrolly]

You won't be able to parse setup.py with Ruby. There are no even AST parsers for Python written in Ruby, and no those that can detect syntax for Python 2 and various 3.x versions without errors.

[brainwane]

Related conversation on Mastodon the last few days.

[jonnymaserati]

Do you also plan to parse the conditional or extras dependencies in setup.py? Or should I get setup.py to generate the requirements.txt file in order for my dependencies to be accounted for here?