Cython dependency not installed before executing setup.py when installing with PyPI

[stefan-baumann]

Problem

When installing madmom via pip on Python 3 (e.g. pip3 install madmom), Cython is missing from the dependencies (correction: Cython is in the dependencies, but not installed before executing setup.py), causing the package install to fail when it is not installed already.

Steps needed to reproduce the behaviour

Just install madmon via pip with Cython not installed.

[superbock]

cython is listed as a requirement. This issue seems to be related to how pip checks the dependencies and when it installs them, see pypa/pip#25 for more details. I am not sure if reordering the entries in requirements.txt or setup.py could possibly resolve the issue or not. Do you have a chance to try this?

[stefan-baumann]

Okay, admittedly I don't know much about creating pip packages. Looking back more closely at the terminal output I got when installing it, it seems like it's actually included, just not installed before trying to execute the setup.py:

  Downloading madmom-0.16.1.tar.gz (20.0 MB)
     |████████████████████████████████| 20.0 MB 4.2 MB/s 
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-u04m6iju/madmom/setup.py'"'"'; __file__='"'"'/tmp/pip-install-u04m6iju/madmom/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-u04m6iju/madmom/pip-egg-info
         cwd: /tmp/pip-install-u04m6iju/madmom/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-u04m6iju/madmom/setup.py", line 12, in <module>
        from Cython.Build import cythonize, build_ext
    ModuleNotFoundError: No module named 'Cython'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Output from when it installed successfully afterwards after installing Cython manually for reference:

  Using cached madmom-0.16.1.tar.gz (20.0 MB)
Requirement already satisfied: cython>=0.25 in /media/.../env-linux/lib/python3.8/site-packages (from madmom) (0.29.21)
Collecting mido>=1.2.8
  Downloading mido-1.2.9-py2.py3-none-any.whl (52 kB)
     |████████████████████████████████| 52 kB 1.6 MB/s 
Requirement already satisfied: numpy>=1.13.4 in /media/.../env-linux/lib/python3.8/site-packages (from madmom) (1.19.4)
Requirement already satisfied: scipy>=0.16 in /media/.../env-linux/lib/python3.8/site-packages (from madmom) (1.5.3)
Building wheels for collected packages: madmom
  Building wheel for madmom (setup.py) ... done
  Created wheel for madmom: filename=madmom-0.16.1-cp38-cp38-linux_x86_64.whl size=21666166 sha256=4ad295ead762aa3ffe3c3dcea15d1a2b7733d4dd9077004645cf8bfe99f3c093
  Stored in directory: /home/stefan/.cache/pip/wheels/a2/fc/0f/2d5569ea771ee70a9bf321a72744b4507eb85ff540d82b54eb
Successfully built madmom
Installing collected packages: mido, madmom
Successfully installed madmom-0.16.1 mido-1.2.9

Apparently the issue is that pip/setuptools does not install all of the dependencies before calling the setup.py.
Looking into this problem, I was able to find a PR from another project that apparently fixed the same problem: automl/auto-sklearn#611. Maybe this same approach could be applied for the setup.py here as well?

[stefansjs]

Looks like the pyproject.toml is missing from the .tar.gz on pypi. I guess the solution might be to add it to the manifest file.

jazzband/pip-tools#1296 (comment)

[stefansjs]

Oh, I see now. It looks like the pyproject.toml was added since the last release. The whole purpose of PEP 517 was to address this situation with build-time requirements. My hope is that it should be fixed with the next release.

[TGabor]

Is this already fixed?
Installing from github directly works flawlessly for me:

pip install git+https://github.com/CPJKU/madmom

[stefansjs]

It's fixed in master by 3b60128, but it hasn't been released on pypi yet.

[tae-jun]

@superbock Hi, thanks for building an awesome package. I've been using madmom in my MIR research happily.

And now I'm building a music structure analysis package and trying to publish it.

The package depends on madmom and due to this build-time dependency issue, it can only be installed as:

pip install numpy cython
pip install allinone

but I hope it can be simply pip install allinone.

Because of the policy of PyPI, I can't publish a package with a direct dependency such as:

madmom @ git+https://github.com/CPJKU/madmom

Could you please publish 0.17? It will be highly appreciated. Thanks!

[itsbrex]

+1 for @tae-jun's request. 😁

Thanks for a great package @superbock! Having some trouble with this in not just allinone, so hopefully a patch is on the way?

[yocontra]

Just updating that this is still present as of April 2024. I'm working around it by installing cython and numpy in a separate command prior to running the pip install for madmom, having them all in the same requirements.txt still has the same issue oddly.

[sjain24]

When using pip install git+https://github.com/CPJKU/madmom I am getting the following error :

[10 lines of output]
      Collecting setuptools
        Using cached setuptools-68.0.0-py3-none-any.whl (804 kB)
      Collecting wheel
        Using cached wheel-0.42.0-py3-none-any.whl (65 kB)
      Collecting cython>=0.25
        Using cached Cython-3.0.11-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
      ERROR: Could not find a version that satisfies the requirement numpy>2 (from versions: 1.3.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, 1.6.1, 1.6.2, 1.7.0, 1.7.1, 1.7.2, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.10.0.post2, 1.10.1, 1.10.2, 1.10.4, 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 1.13.3, 1.14.0, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5, 1.14.6, 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.15.4, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.16.4, 1.16.5, 1.16.6, 1.17.0, 1.17.1, 1.17.2, 1.17.3, 1.17.4, 1.17.5, 1.18.0, 1.18.1, 1.18.2, 1.18.3, 1.18.4, 1.18.5, 1.19.0, 1.19.1, 1.19.2, 1.19.3, 1.19.4, 1.19.5, 1.20.0, 1.20.1, 1.20.2, 1.20.3, 1.21.0, 1.21.1, 1.21.2, 1.21.3, 1.21.4, 1.21.5, 1.21.6)
      ERROR: No matching distribution found for numpy>2
      WARNING: You are using pip version 22.0.4; however, version 24.0 is available.