Note for Ray backend developers on the Python toolchain

[pcmoritz]

This concerns people who are building Ray from source, i.e. Ray backend developers.

TL;DR: If you are building Ray from source, you should use Anaconda packages soon instead of pip.

There are known problems with the compatibility of TensorFlow, pyarrow and other pip packages (see the discussion in https://groups.google.com/forum/#!search/manylinux1/ray-dev/gbYKK9Gy_ns/PYuai11YAgAJ). This is due to the fact that C++11 is not supported in a forward compatible way on the manylinux1 platform used by pip. So far we have used a patch and carefully chosen build flags to avoid certain C++11 features for pyarrow to circumvent these issues and prevent clashes with other packages. However with the inclusion of LLVM into pyarrow, going forward this won't be an option any more. As soon as #3898 is merged, we will be using a mostly unmodified pyarrow wheel that will clash with various pip packages (namely scipy, opencv and tensorflow).

The simplest solution for Ray backend developers for now is to use conda packages for the dependencies, as the C++ standard library shipped with conda doesn't have the problems mentioned above.

For Ray users we will keep shipping the patched pyarrow wheels for now. A possible longer term solution might be to ship pyarrow as a manylinux2010 wheel once that is ready (see pypa/manylinux#252). Any other ideas to resolve this are greatly appreciated!

[byronyi]

This should not be an issue as pre-built TF (>= 1.15) wheels on PyPI are built with gcc 7.3 and manylinux2010. See here. A relevant discussion thread could be found here.