installing vsc-base with easy_install (from PyPI) picks up vsc-install from GitHub (rather than PyPI)?

[boegel]

I'm not 100% sure about this, but I see no other way in which vsc-install 0.10.20 was picked up during the EasyBuild bootstrap while it was not released yet on PyPI (it is now, since I published the latest master of vsc-install to deal with the issue), cfr. https://api.travis-ci.org/jobs/177114943/log.txt?deansi=true

@stdweird Is that intentional? Shouldn't packages published on PyPI be using the vsc-install that was used to package it, or am I missing something?

If it is intentional, we need a way to ensure that the latest vsc-install is always published on PyPI too?
Or I need to make some changes to how the bootstrap picks a vsc-install version to install from PyPI during stage2; it now checks what was installed during stage1, and assumes that version is available on PyPI.

[boegel]

This seems to confirm my suspicion:

$ easy_install vsc-base
Searching for vsc-base
Reading https://pypi.python.org/simple/vsc-base/
...
Best match: vsc-base 2.5.5
Downloading https://pypi.python.org/packages/73/24/e4244a743e0cfc4af1f3b3b772698e2f4dafc7052bc006a1b829b66f7a3a/vsc-base-2.5.5.tar.gz#md5=de68a64cead111b169933354d47ac9b9
Processing vsc-base-2.5.5.tar.gz
Writing /tmp/easy_install-DH3sqY/vsc-base-2.5.5/setup.cfg
Running vsc-base-2.5.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-DH3sqY/vsc-base-2.5.5/egg-dist-tmp-DTDQgG
INFO: This is (based on) vsc.install.shared_setup 0.10.17
...
INFO: Searching for vsc-install>=0.10.1
INFO: Best match: vsc-install 0.10.1
INFO: Doing git clone from https://github.com/hpcugent/vsc-install.git to /tmp/easy_install-DH3sqY/vsc-base-2.5.5/temp/easy_install-3uWBEy/vsc-install.git
INFO: Processing vsc-install.git
INFO: Writing /tmp/easy_install-DH3sqY/vsc-base-2.5.5/temp/easy_install-3uWBEy/vsc-install.git/setup.cfg
INFO: Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-DH3sqY/vsc-base-2.5.5/temp/easy_install-3uWBEy/vsc-install.git/egg-dist-tmp-8583Vc
INFO: This is (based on) vsc.install.shared_setup 0.10.20

[stdweird]

i'm not sure what is going on here, but i also don't see anything going wrong. i assume the bootstrap script assumed that whatever version was picked up by the vsc-base install, it should on pypi?

the installation will or should always use latest whatever is required. the shared_setup does not force a version as the dependency, and certainly not the version it was packaged with. but there are some other things going on here.

altough someone pushed 0.10.20 to pypi, before that latest was 0.10.19, so it is very strange that you get Best match: vsc-install 0.10.1
@JensTimmerman i guess the next part that git is used to fetch vsc-install while easy_install vsc-base is used, is something you added wrt the install from private git magic? will it walwasy do that and fallback to pypi?

in particular, the travis report has

Installed /home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/vsc_base-2.5.5-py2.6.egg
/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/pkg_resources/__init__.py:2510: PEP440Warning: 'vsc (install-0.10.1)' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions.
  PEP440Warning,
Processing dependencies for vsc-base
Searching for vsc-install>=0.10.1

[JensTimmerman]

this is because the vsc-base setup.py includes:
'setup_requires': ['vsc-install >= %s' % VSC_INSTALL_REQ_VERSION],

this will prompt vsc-install (that you obviously already have, of you would never get to this part) to install the latest vsc-install, which it now can find on github.

I guess we should just remove this setup_requires, it's superfluous, since we can never get to it if we don't have vsc-install anyway.

[JensTimmerman]

if this is not the default behaviour you want we should also merge hpcugent/vsc-install#61

[JensTimmerman]

actually, this will make no difference at all, you still would need to have it in install_requires for the rpm's to work correctly, so it would still be pulled in

[JensTimmerman]

so yes, ensuring latest master is always on pypi is a good idea and probably the best solution.
I guess we can let jenkins do this? If you all agree I can look into it.

[boegel]

There are more problems with this: the auto-magic update to the latest vsc-install being pulled from GitHub implies a hard dependency on git, even though there's no real reason for it?

See easybuilders/easybuild-framework#1962

[JensTimmerman]

yes, this was a bug in setuptools, I've fixed this in pypa/setuptools#609 so try using a newer setuptools.

so this ticket is just about agreeing that jenkins should always push to pypi, for which I haven't gotten a reply yet, do you agree there @boegel ? Or do you wish another solution?

[boegel]

Auto-publishing to PyPI makes sense I guess, just as long as we make sure not to introduce regressions...

Publishing a broken release on PyPI has a bigger impact than just having it merged in master.

[JensTimmerman]

ok, follow up in hpcugent/vsc-install#63