Skip to content

SIRF and Python

Jump to bottom
Kris Thielemans edited this page Dec 28, 2020 · 21 revisions

SIRF can be called from Python. This page documents some issues that you might encounter.

Version conflicts

If you have more than 1 Python version installed, you could encounter conflicts when installing or using SIRF. It is important that you use the same version of python throughout the installation, and of course also when running SIRF.

For instance, many people have both python2.7 and python3, or have different virtualenv's. You will have to create a separate SIRF build for every version of Python that you want to use. Obviously, you will also have to install Python packages for each. Using an environment such as Anaconda might help.

If you have conflicts during installation, or CMake picked up the wrong version, we recommend to first try to set the CMake variable PYTHON_EXECUTABLE (use an absolute path!). For instance,

    # find out where python3 sits
    which python3
    cmake -DPYTHON_EXECUTABLE=/wherever/it/was/python3

If that fails, set both PYTHON_LIBRARY and PYTHON_INCLUDE_DIR. On many systems python-config or python3-config can help (but not when using Anaconda).

Linux

As an example, you could have

$ python3-config --ldflags
-L/usr/lib/python3.4/config-3.4m-x86_64-linux-gnu -L/usr/lib -lpython3.4m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
$ python3-config --cflags
-I/usr/include/python3.4m -I/usr/include/python3.4m  -Wno-unused-result -g -fstack-protector-strong -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

This tells you that appropriate settings are

PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.4m.so
PYTHON_INCLUDE_DIR=/usr/include/python3.4m

MacOS

You could have a look at the Python Guide for OSX.

Python2

If you wish to use python2, the following should work:

    cmake -DPYTHON_EXECUTABLE=/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

If that fails, specifying everything explicitly might work, e.g. to use the system Python

    cmake -DPYTHON_EXECUTABLE=/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 \
          -DPYTHON_LIBRARY=/System/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib \
          -DPYTHON_INCLUDE_DIR=/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7

Python3

If you prefer to use python3 (which is recommended these days), you will need to install it with

    brew install python

and then find the relevant location of your Python executable (CMake will then attempt to find your Python library and include directories). If you installed python3 with brew, the following might work:

    PY_INST=$(ls -d1 /usr/local/Cellar/python/3.*/Frameworks/Python.framework/Versions/3.*|tail -n 1)
    PY_VER=$(basename ${PY_INST})
    PY_EXE=$PY_INST/bin/python$PY_VER
    cmake -DPYTHON_EXECUTABLE=$PY_EXE <normal_options>

(This tries to find the (alphabetically) last Python 3 version, and then assigns its executable to PY_EXE).

Note that specifying PYTHON_EXECUTABLE=/usr/local/bin/python for CMake will sadly not work. If the above fails, you could check our Travis file which also tests SIRF on OSX (search for osx).

Please make sure that you always use the Python version that you used to build SIRF. Either save the PY_EXE variable somewhere, or see below re SIRF_PYTHON_EXECUTABLE.

SuperBuild installation

After building and installing SIRF with the SuperBuild, two environment files for (bash and csh) are created. These files contain all the necessary information about the binary installation to avoid conflicts of versions of software.

Before using SIRF, these files should be sourced in a shell session. In particular, they set the variables

  • SIRF_PYTHON_EXECUTABLE which points to the python executable that was used during the build.
  • PYTHONPATH which points to the location of the python modules for SIRF

Python packages

Here's a list of Python packages which are needed for SIRF to work

numpy 
scipy 
matplotlib 
docopt  
h5py 
libxml2-python
psutil 
nose
coverage

We also recommend a Python IDE, e.g.

spyder

Using your OS package manager

On a debian based system you would issue

sudo apt-get install python-scipy  python-docopt python-numpy python-h5py python-matplotlib python-libxml2 python-psutil spyder python-tk python-nose python-coverage

Using different python package managers

The current list of required python packages can be found above. In the next sections you can find a (possibly incomplete) set of instruction for conda and pip.

Using Anaconda

Try the following

conda update conda
conda update --all
conda create --name sirf
source activate sirf
conda install cmake numpy scipy matplotlib docopt h5py libxml2 psutil nose coverage
pip install sphinx_bootstrap_theme

Using pip

If you are using pip install (either with the SuperBuild or manually installing the dependencies of SIRF), you need to ensure that the same version of python is used for this as will be used throughout the installation.

You can/should specify the python executable that you want to use for pip by replacing the pip install command with /path/to/desired/python/executable -m pip install. Assuming that this executable is stored in SIRF_PYTHON_EXECUTABLE, this might work

$SIRF_PYTHON_EXECUTABLE -m pip install --upgrade pip setuptools wheel
$SIRF_PYTHON_EXECUTABLE -m pip install --only-binary=numpy,scipy,matplotlib numpy scipy matplotlib
$SIRF_PYTHON_EXECUTABLE -m pip install docopt nose codecov coveralls

Add the --user option if you don't have administrator permissions.

Run-time conflicts with shared libraries

Despite best efforts, it is possible that SIRF is compiled using a version of a library that is different from what is used by Python itself, for example if you compile SIRF with a more recent compiler than your normal system compiler. CMake should normally warn you about this with something like

  Cannot generate a safe runtime search path for target _pysirfreg because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libz.so.1] in /usr/lib64 may be hidden by files in:
      /opt/Python/Python-3.6/lib

This might lead to run-time errors. Below is one example that we encountered on CentOS 7.

OMP library conflicts

We did encounter a corner case on CentOS 7 when using Python 2.7 with gcc 4.9 (possibly because the Python version was out-of-date). There was a conflict that the libgomp.so.1 version used by Python was too old. As more recent versions of libgomp should contain older versions as well, this can be resolved by using LD_PRELOAD.

LD_PRELOAD=/lib64/libgomp.so.1 python
Clone this wiki locally