Skip to content

SIRF and Python

Jump to bottom
Kris Thielemans edited this page Jun 6, 2021 · 21 revisions

SIRF can be called from Python. This page documents some issues that you might encounter.

Python versions

Up to SIRF version 2.2.0 we supported both Python 2 and 3. SIRF 3.0 dropped Python 2 support (it might still work, but don't try!).

Version conflicts

If you have more than 1 Python version installed, you could encounter conflicts when installing or using SIRF. It is important that you use the same version of python throughout the installation, and of course also when running SIRF.

For instance, many people have both python2.7 and python3, or have different virtualenv's. You will have to create a separate SIRF build for every version of Python that you want to use. Obviously, you will also have to install Python packages for each. Using an environment such as Anaconda might help.

If you have conflicts during installation, or CMake picked up the wrong version, we recommend to first try to set the CMake variable PYTHON_EXECUTABLE (use an absolute path!). For instance,

    # find out where python3 sits
    which python3
    cmake -DPYTHON_EXECUTABLE=/wherever/it/was/python3

If that fails, set both PYTHON_LIBRARY and PYTHON_INCLUDE_DIR. On many systems python-config or python3-config can help (but not when using Anaconda).

Please make sure that you always use the Python version that you used to build SIRF.

Linux

As an example, you could have

$ python3-config --ldflags
-L/usr/lib/python3.4/config-3.4m-x86_64-linux-gnu -L/usr/lib -lpython3.4m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
$ python3-config --cflags
-I/usr/include/python3.4m -I/usr/include/python3.4m  -Wno-unused-result -g -fstack-protector-strong -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

This tells you that appropriate settings are

PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.4m.so
PYTHON_INCLUDE_DIR=/usr/include/python3.4m

MacOS

You could have a look at the Python Guide for OSX. For instance, use

    brew install python

and then find the relevant location of your Python executable (CMake will then attempt to find your Python library and include directories). If you installed python3 with brew, the following might work:

    PY_INST=$(ls -d1 /usr/local/Cellar/python/3.*/Frameworks/Python.framework/Versions/3.*|tail -n 1)
    PY_VER=$(basename ${PY_INST})
    PY_EXE=$PY_INST/bin/python$PY_VER
    cmake -DPYTHON_EXECUTABLE=$PY_EXE <normal_options>

(This tries to find the (alphabetically) last Python 3 version, and then assigns its executable to PY_EXE).

Note that specifying PYTHON_EXECUTABLE=/usr/local/bin/python for CMake will sadly not work. If the above fails, you could check our Travis file where we used to on OSX (search for osx).

SuperBuild installation

After building and installing SIRF with the SuperBuild, two environment files for (bash and csh) are created. These files contain all the necessary information about the binary installation to avoid conflicts of versions of software.

Before using SIRF, these files should be sourced in a shell session. In particular, they set the variables

  • SIRF_PYTHON_EXECUTABLE which points to the python executable that was used during the build.
  • PYTHONPATH which points to the location of the python modules for SIRF

Python packages

Use the SIRF requirements.txt to install packages. Note that if you used the SIRF-SuperBuild, you can find it in /wherever/you/built/sources/SIRF/requirements.txt.

We also recommend a Python IDE, e.g. spyder.

Using pip

If you are using pip install (either with the SuperBuild or manually installing the dependencies of SIRF), you need to ensure that the same version of python is used for this as will be used throughout the installation.

You can/should specify the python executable that you want to use for pip by replacing the pip install command with /path/to/desired/python/executable -m pip install. Assuming that this executable is stored in SIRF_PYTHON_EXECUTABLE, the following might work:

$SIRF_PYTHON_EXECUTABLE -m pip install --upgrade pip setuptools wheel
$SIRF_PYTHON_EXECUTABLE -m pip install -U -r requirements.txt

## Using Anaconda/miniconda
Try the following

conda update conda conda update --all conda create --name sirf source activate sirf conda install cmake conda install -c conda-forge -y --file requirements.txt

(you might have to edit the `requirements.txt` somewhat).

## Using your OS package manager
(This option is not recommended and the text is out-of-date).

On a debian based system you would issue

sudo apt-get install python-scipy python-docopt python-numpy python-h5py python-matplotlib python-libxml2 python-psutil spyder python-tk python-nose python-coverage


# Run-time conflicts with shared libraries

Despite best efforts, it is possible that SIRF is compiled using a version of a library that is 
different from what is used by Python itself, for example if you compile SIRF with a more recent
compiler than your normal system compiler. CMake should normally 
warn you about this with something like

Cannot generate a safe runtime search path for target _pysirfreg because files in some directories may conflict with libraries in implicit directories:

runtime library [libz.so.1] in /usr/lib64 may be hidden by files in:
  /opt/Python/Python-3.6/lib

This might lead to run-time errors. Below is one example that we encountered on CentOS 7.

## OMP library conflicts
We did encounter a corner case on CentOS 7 when using Python 2.7 with gcc 4.9
(possibly because the Python version was out-of-date). There was
a conflict that the `libgomp.so.1` version used by Python was too old.
As more recent versions of `libgomp` should contain older versions as well, this can be
resolved by using [LD_PRELOAD](http://man7.org/linux/man-pages/man8/ld.so.8.html).
```sh
LD_PRELOAD=/lib64/libgomp.so.1 python
Clone this wiki locally