Setup Issues with Capreolus on Canada Compute

[jasper-xian]

I ran into mostly the same errors when setting up Capreolus on CC (Cedar) with setup-cc.md as this issue here. Thus, I followed the setup described in this comment within that issue.

Some of the new issues I noticed were that scipy-stack now uses python 3.8, so the python version changes to 3.8.10 when running module load scipy-stack. I also noticed that the TensorFlow version installed was 2.8.0, despite the requirements.txt listing it to be between 2.3 and 2.5.

---

Additionally, when running pip install tensorflow-ranking==0.4.2 --no-cache-dir I ran into the error:

ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly

This was solved by manually importing pyarrow with the commands below:

module load gcc/9.3.0 arrow python scipy-stack
python -c "import pyarrow"

---

Despite this, I was able to verify my installation as per these commands. However, when running

python -m capreolus.run rerank.train with file=docs/reproduction/config_msmarco.txt

to fine-tune a mini MS MARCO, I run into this set of warnings/errors:

2022-04-12 17:37:07.215423: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-04-12 17:37:09.466046: F tensorflow/core/platform/default/env.cc:73] Check failed: ret == 0 (11 vs. 0)Thread tf_numa_-1_Eigen creation via pthread_create() failed.
Aborted

Am a bit stuck here (perhaps it is a mismatching TensorFlow version issue?), so any help would be much appreciated!