Incompatible versions of numpy in Quantum Mobile

[sphuber]

Thanks a lot @chrisjsewell . I have good news and bad news: the good news is that with the new QM release I can run the abinit example for Si fast without issues 👍 ! However, there was an unrelated problem that had to be fixed first. When trying to submit I was faced with the following exception:

Traceback (most recent call last):
  File "/home/max/.virtualenvs/aiida/bin/aiida-common-workflows", line 11, in <module>
    load_entry_point('aiida-common-workflows', 'console_scripts', 'aiida-common-workflows')()
  File "/home/max/.virtualenvs/aiida/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/max/.virtualenvs/aiida/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/max/.virtualenvs/aiida/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/max/.virtualenvs/aiida/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/max/.virtualenvs/aiida/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/max/.virtualenvs/aiida/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/max/codes/aiida-common-workflows/aiida_common_workflows/cli/launch.py", line 48, in cmd_relax
    process_class = load_workflow_entry_point('relax', plugin)
  File "/home/max/codes/aiida-common-workflows/aiida_common_workflows/plugins/entry_point.py", line 42, in load_workflow_entry_point
    return entry_point.load_entry_point('aiida.workflows', prefix)
  File "/home/max/codes/aiida-core/aiida/plugins/entry_point.py", line 208, in load_entry_point
    loaded_entry_point = entry_point.load()
  File "/home/max/.virtualenvs/aiida/lib/python3.7/site-packages/reentry/entrypoint.py", line 38, in load
    module = import_module(self.module_name)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/max/codes/aiida-common-workflows/aiida_common_workflows/workflows/relax/abinit/__init__.py", line 4, in <module>
    from .generator import *
  File "/home/max/codes/aiida-common-workflows/aiida_common_workflows/workflows/relax/abinit/generator.py", line 10, in <module>
    from pymatgen.core import units
  File "/home/max/.virtualenvs/aiida/lib/python3.7/site-packages/pymatgen/__init__.py", line 45, in <module>
    from .core.composition import Composition
  File "/home/max/.virtualenvs/aiida/lib/python3.7/site-packages/pymatgen/core/__init__.py", line 11, in <module>
    from .lattice import Lattice
  File "/home/max/.virtualenvs/aiida/lib/python3.7/site-packages/pymatgen/core/lattice.py", line 23, in <module>
    from pymatgen.util.coord import pbc_shortest_vectors
  File "/home/max/.virtualenvs/aiida/lib/python3.7/site-packages/pymatgen/util/coord.py", line 17, in <module>
    from . import coord_cython as cuc
  File "coord_cython.pyx", line 1, in init pymatgen.util.coord_cython
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

We have seen this before but I never figured out an actual fix that is reproducible. It seems that there is an incompatibility between the numpy that was installed and the one that was used during the build process of pymatgen 😞 I tried removing and reinstalling the numpy==1.19.5 which was installed, but to no avail. I then tried to install numpy==1.20.1 which solved the problem, however, for this I had to remove aiida-fleur as dependency from aiida-common-workflows and reinstall, as aiida-fleur explicitly specifies numpy<1.20 and so it would fail with a dependency conflict. I think they may have added this because numpy==1.20 removed Python 3.6 support and they wanted to still support it. @Tseplyaev @broeder-j can you confirm this? If that is the case, I don't think this is the way to do this. Simply adding python_requires>=3.6 in your setup will inform pip that you support Python 3.6 and so it shouldn't install incompatible numpy versions. It would be good if you could remove this explicit limitation.

Then @chrisjsewell , do you have any idea how we can prevent this numpy problem in QM?

Originally posted by @sphuber in #159 (comment)

[chrisjsewell]

with the new QM release

Firstly, can you clarify what version of QM you used?
I assume you mean marvelnccr/quantum-mobile:24.03.21-abinit for the abinit fix, but then this will not have originally included aiida-fleur and e.g. for from pymatgen.core.composition import Composition has no problems. For completeness:

$ pip freeze
aiida-abinit==0.2.0a1
-e git+https://github.com/aiidateam/aiida-core.git@7b8a400bb9549748171b514a0a2a2e68e177cdac#egg=aiida_core
aiida-pseudo==0.5.0
aiida-quantumespresso==3.4.0
aiidalab==20.11.0b1
aiidalab-widgets-base==1.0.0b15
alabaster==0.7.12
aldjemy==0.9.1
alembic==1.5.8
aniso8601==9.0.1
appdirs==1.4.4
archive-path==0.2.1
ase==3.21.1
attrs==20.3.0
Babel==2.9.0
backcall==0.2.0
bcrypt==3.2.0
bleach==3.3.0
bokeh==1.4.0
CacheControl==0.12.6
cachetools==4.2.1
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
circus==0.16.1
click==7.1.2
click-completion==0.5.2
click-config-file==0.6.0
click-spinner==0.1.10
configobj==5.0.6
cryptography==3.4.6
cycler==0.10.0
decorator==4.4.2
defusedxml==0.7.1
Django==2.2.19
dnspython==2.1.0
docutils==0.15.2
dulwich==0.19.16
elementpath==2.2.0
email-validator==1.1.2
entrypoints==0.3
ete3==3.1.2
fastapi==0.63.0
fastentrypoints==0.12
fileupload==0.1.5
Flask==1.1.2
Flask-Cors==3.0.10
Flask-RESTful==0.3.8
frozendict==1.2
furl==2.1.0
future==0.18.2
graphviz==0.16
idna==2.10
imagesize==1.2.0
importlib-metadata==3.7.3
ipykernel==5.5.0
ipython==7.21.0
ipython-genutils==0.2.0
ipywidgets==7.6.3
ipywidgets-extended==1.0.5
itsdangerous==1.1.0
jedi==0.18.0
Jinja2==2.11.3
joblib==1.0.1
jsonschema==3.2.0
jupyter-client==6.1.12
jupyter-core==4.7.1
jupyterlab-widgets==1.0.0
kiwipy==0.5.5
kiwisolver==1.3.1
lark-parser==0.11.2
lockfile==0.12.2
Mako==1.1.4
Markdown==3.3.4
MarkupSafe==1.1.1
matplotlib==3.3.4
mistune==0.8.4
monty==2021.3.3
more-itertools==8.7.0
mpmath==1.2.1
msgpack==1.0.2
nbconvert==5.6.1
nbformat==5.1.2
networkx==2.5
nglview==2.7.7
notebook==5.7.10
numpy==1.20.1
optimade==0.13.3
optimade-client==2021.3.16
orderedmultidict==1.0.1
packaging==20.9
palettable==3.3.0
pandas==1.2.3
pandocfilters==1.4.3
paramiko==2.7.2
parso==0.8.1
pathtools==0.1.2
pexpect==4.8.0
pgsu==0.1.0
pickleshare==0.7.5
pika==1.2.0
Pillow==8.1.2
plotly==4.14.3
plumpy==0.15.1
prometheus-client==0.9.0
prompt-toolkit==3.0.18
psutil==5.8.0
psycopg2-binary==2.8.6
ptyprocess==0.7.0
PyCifRW==4.4.2
pycparser==2.20
pydantic==1.8.1
pydata-sphinx-theme==0.4.3
Pygments==2.8.1
pymatgen==2022.0.5
PyMySQL==0.9.3
PyNaCl==1.4.0
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
python-editor==1.0.4
python-memcached==1.59
pytz==2019.3
PyYAML==5.1.2
pyzmq==22.0.3
qe-tools==2.0.0rc2
reentry==1.3.2
requests==2.25.1
requests-cache==0.5.2
retrying==1.3.3
ruamel.yaml==0.16.13
ruamel.yaml.clib==0.2.2
scikit-learn==0.24.1
scipy==1.6.1
seekpath==1.9.7
Send2Trash==1.5.0
shellingham==1.4.0
shortuuid==1.0.1
simplejson==3.17.2
six==1.15.0
snowballstemmer==2.1.0
spglib==1.16.1
Sphinx==3.5.3
sphinx-copybutton==0.3.1
sphinx-notfound-page==0.6
sphinx-panels==0.5.2
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-details-directive==0.1.0
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
sphinxext-rediraffe==0.2.6
SQLAlchemy==1.3.23
SQLAlchemy-Utils==0.34.2
sqlparse==0.4.1
starlette==0.13.6
sympy==1.7.1
tabulate==0.8.9
terminado==0.9.3
testpath==0.4.4
threadpoolctl==2.1.0
toml==0.10.2
topika==0.2.2
tornado==4.5.3
tqdm==4.59.0
traitlets==5.0.5
typing-extensions==3.7.4.3
tzlocal==2.1
uncertainties==3.1.5
upf-to-json==0.9.3
urllib3==1.26.4
voluptuous==0.12.1
watchdog==0.10.4
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==1.0.1
widget-periodictable==2.1.5
widgetsnbextension==3.5.1
wrapt==1.11.2
xmlschema==1.5.3
zipp==3.4.1

So I assume, you must have installed aiida-fleur etc, independently at which point this issue has been introduced?

Indeed if I activate the aiida environment then run pip install aiida-fleur==1.1.2, afterwards numpy==1.17.5/pymatgen==2020.12.31 and I start having issues:

max@269fea80010f:/$ ipython
Python 3.7.10 (default, Feb 20 2021, 21:17:23) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.22.0 -- An enhanced Interactive Python. Type '?' for help.
[TerminalIPythonApp] WARNING | Error in loading extension: aiida.tools.ipython.ipython_magics
Check your config files in /home/max/.ipython/profile_default
Traceback (most recent call last):
  File "/home/max/.local/lib/python3.7/site-packages/IPython/core/shellapp.py", line 301, in init_extensions
    self.shell.extension_manager.load_extension(ext)
  File "/home/max/.local/lib/python3.7/site-packages/IPython/core/extensions.py", line 80, in load_extension
    mod = import_module(module_str)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/max/.local/lib/python3.7/site-packages/aiida/tools/__init__.py", line 25, in <module>
    from .data.array.kpoints import *
  File "/home/max/.local/lib/python3.7/site-packages/aiida/tools/data/array/kpoints/__init__.py", line 14, in <module>
    from aiida.orm import KpointsData, Dict
  File "/home/max/.local/lib/python3.7/site-packages/aiida/orm/__init__.py", line 17, in <module>
    from .groups import *
  File "/home/max/.local/lib/python3.7/site-packages/aiida/orm/groups.py", line 80, in <module>
    class Group(entities.Entity, entities.EntityExtrasMixin, metaclass=GroupMeta):
  File "/home/max/.local/lib/python3.7/site-packages/aiida/orm/groups.py", line 56, in __new__
    entry_point_group, entry_point = get_entry_point_from_class(namespace['__module__'], name)
  File "/home/max/.local/lib/python3.7/site-packages/aiida/plugins/entry_point.py", line 287, in get_entry_point_from_class
    for group in ENTRYPOINT_MANAGER.get_entry_map().keys():
  File "/home/max/.local/lib/python3.7/site-packages/reentry/default_manager.py", line 69, in get_entry_map
    return self._backend.get_map(dist=dist_names, group=groups, name=ep_names)
  File "/home/max/.local/lib/python3.7/site-packages/reentry/jsonbackend.py", line 173, in get_map
    dist_list = self._dist_list_from_arg(dist)
  File "/home/max/.local/lib/python3.7/site-packages/reentry/jsonbackend.py", line 268, in _dist_list_from_arg
    dist_list = _listify(dist_arg)
  File "/home/max/.local/lib/python3.7/site-packages/reentry/jsonbackend.py", line 285, in _listify
    from six.moves import collections_abc
ImportError: cannot import name 'collections_abc' from 'six.moves' (unknown location)

In [1]: from pymatgen.core.composition import Composition
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-929d9a721734> in <module>
----> 1 from pymatgen.core.composition import Composition

~/.local/lib/python3.7/site-packages/pymatgen/__init__.py in <module>
     43 SETTINGS = _load_pmg_settings()
     44 
---> 45 from .core.composition import Composition
     46 from .core.lattice import Lattice
     47 from .core.operations import SymmOp

~/.local/lib/python3.7/site-packages/pymatgen/core/__init__.py in <module>
      9 
     10 from .composition import Composition
---> 11 from .lattice import Lattice
     12 from .operations import SymmOp
     13 from .periodic_table import DummySpecies, Element, Species

~/.local/lib/python3.7/site-packages/pymatgen/core/lattice.py in <module>
     21 from numpy.linalg import inv
     22 
---> 23 from pymatgen.util.coord import pbc_shortest_vectors
     24 from pymatgen.util.num import abs_cap
     25 from pymatgen.util.typing import Vector3Like

~/.local/lib/python3.7/site-packages/pymatgen/util/coord.py in <module>
     15 import numpy as np
     16 
---> 17 from . import coord_cython as cuc
     18 
     19 # array size threshold for looping instead of broadcasting

coord_cython.pyx in init pymatgen.util.coord_cython()

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

As mentioned in the original discussion, the problem is that numpy is first installed by pymatgen, which it requires for its compilation procedure, and by default just installs the latest numpy version (currently 1.20.1), then later aiida-fleur"forces" numpy to be downgraded to<1.20, at which point numpy is no longer compatible with pymatgen. I'd also note that the pip 2020-resolver is already used, which is not picking up this incompatibility, nor does running pip check`.

This issue then may not necessarily occur when building the full QM, which installs all dependencies at once and so is less prone to overriding of previous installs. But obviously I will check this when I build a "full" release and have a think if there is other ways to test this (more so than just pip check)

Indeed though, it would be good if aiida-fleur could remove this pinning which, if nothing else is forcing a numpy downgrade, should "fix" the problem. Although this is not neccessarily a robust solution against it happening in the future.

Also of note, recent updates to pymatgen may or may not have fixed this in some way.
Also a previous "solution" in aiida-core CI is to pre-install numpy at a specific version, i.e. numpy<1.20, although again this is not an ideal long-term solution

cc also @aiidateam/dependency-manager

[sphuber]

Firstly, can you clarify what version of QM you used?

Both marvelnccr/quantum-mobile:20.11.2a and marvelnccr/quantum-mobile:24.03.21-abinit.

I assume you mean marvelnccr/quantum-mobile:24.03.21-abinit for the abinit fix, but then this will not have originally included aiida-fleur and e.g. for from pymatgen.core.composition import Composition has no problems

This indeed only occurred when installing aiida-common-workflows which is what I am testing, and this installs aiida-fleur as one of its requirements.

[csadorf]

Just to verify, are you using a recent version of pip (>=42)? Because I was able to show that this problem was likely resolved in version 42, see this comment.

[chrisjsewell]

This indeed only occurred when installing aiida-common-workflows

That is somewhat of a problem, that with aiida-common-workflows you are potentially overriding the initial working environment I build in Quantum Mobile, and obviously not something I can necessarily account for 😬
In my experience pip, even with the new resolver, still does not ensure version compatibility across multiple installs into the same environment.
I would note though, as partial mitigation (and how aiidalab dependencies are installed) that there is a ~/local/share/aiida/constraints.txt file on the VM, generated from the aiida-core's install dependencies. You can use this to install with: pip install -c ~/local/share/aiida/constraints.txt aiida-common-workflows, such that it ensures that all the aiida-core dependency version are still with their compatible ranges.

are you using a recent version of pip

You mean setuptools 😉, but yeh well that is 47.1.0 before installing aiida-fleur and 39.0.1 after, so make of that what you will 🤷

[csadorf]

are you using a recent version of pip

You mean setuptools 😉,

😛

but yeh well that is 47.1.0 before installing aiida-fleur and 39.0.1 after, so make of that what you will 🤷

I can't find an obvious reason for why it would downgrade setuptools, but this could be related to the issue.

[chrisjsewell]

For this issue, and for #162, does it not make more sense to create a new virtualenv within which to install/run common workflows, rather taking the working aiida virtualenv, then "destroying" it with the aiida-common-workflow install lol.

Also I think your install_requires requires some work:

aiida-common-workflows/setup.json

Line 17 in a3aa64b

"install_requires": [

I would suggest it should include a pinning for aiida-core (probably to 1.5.2 for now) and also ensure all the plugins have some form of upper pinning, so the package won't break later on if plugin make any breaking changes.

[sphuber]

I think that creating an entire new virtual environment because it breaks another might be the wrong way to go about it. Isn't aiida-fleur already part of the aiida env in QM? I think it is part of the MaX codes isn't it? It seems that at least part of the problem here is related to their explicit upper limit on numpy which is not necessary. I will open a PR there and ask if they can make a release with the change. Then for the compatibility of aiida-common-workflows. All plugins seem to use semantic versioning and I hope they respect this as much as possible. Wouldn't using the compatibility operator be sufficient? I don't want the versions to be completely locked down either.