How to enable plugin when using Google Colab notebook

[XuShuqi7]

Hi everyone,

I am currently working on the neutron magnetic scattering for the HighNESS project. I have implemented the magnetic model in a plugin named "MagScat" which is compatible with the latest NCrystal 3.0.0 version:

https://github.com/XuShuqi7/ncplugin-MagScat

The verification is presented in the .ipynb file. Now I would like to use the Google Colab notebook to share these results. While I confront some troubles when trying to compile the plugin in NCrystal:

%cd -q /content/
!git clone https://github.com/mctools/ncrystal &> /dev/null
!git clone https://github.com/XuShuqi7/ncplugin-MagScat &> /dev/null
%cd -q ncrystal
!git checkout v3.0.0 &> /dev/null
%cd -q ..
!mkdir ncrystal-build
!mkdir ncrystal-install
%cd -q ncrystal-build
!cmake -D CMAKE_INSTALL_PREFIX=../ncrystal-install/ ../ncrystal/ -DBUILTIN_PLUGIN_LIST="../ncplugin-MagScat" &> /dev/null
!make &> /dev/null
!make install &> /dev/null
%cd -q /content

These commands are based on a code provided by Ignacio. My modified !cmake command line is probably wrong. Does someone know how to correct it?

Thank you very much for your help.

Best regards,
Shuqi

[tkittel]

Dear Shuqi,

Thanks for your question. I am not an expert in google-collab, but here are a few observations:

• If you are debugging a problem, do not remove all the potentially useful error message by redirecting to /dev/null.
• My cd command does not recognise a -q option, removing it would make your code more portable.
• You don't have to manually clone a ncplugin repo in order to use it (read on).
• The cmake command you run gives the error (from the NCrystal CMake code):

  ---- Trying to install plugin via local path: ../ncplugin-MagScat
  CMake Error at CMakeLists.txt:361 (message):
    Local path in BUILTIN_PLUGIN_LIST is not an absolute paths:
    ../ncplugin-MagScat
  

  if I replace the last argument with -DBUILTIN_PLUGIN_LIST="$PWD/../ncplugin-MagScat" it seems to help.
  However, if you are anyway cloning the ncplugin-MagScat repo just to use it with the -DBUILTIN_PLUGIN_LIST argument, you can instead simply ask the NCrystal CMake code to get the code with -DBUILTIN_PLUGIN_LIST=XuShuqi7:MagScat. So you can do (remove the set -e line if it doesn't work, it is there to make sure the commands ends as soon as there is a failure, it might not work or be needed on google collab):

  set -e
  cd /content/
  mkdir ncrystal-build ncrystal-install
  git clone -b v3.0.0 https://github.com/mctools/ncrystal 
  cd ncrystal-build
  cmake -DCMAKE_INSTALL_PREFIX=../ncrystal-install/ ../ncrystal/ -DBUILTIN_PLUGIN_LIST=XuShuqi7:MagScat
  make -j2 install
  cd ..
  

Afterwards you should activate the installation by sourcing setup.sh, and asking ncrystal-config to provide a summary in order to verify everything makes sense:

. /content/ncrystal-install/setup.sh
ncrystal_inspectfile --plugins
ncrystal-config --summary

I hope this helps. I verified the above commands locally, but not on google collab. Please let us know if it works (or if there are things that needs to be changed) here, for the benefit of other google-collab + NCrystal users.

Cheers,
Thomas

[XuShuqi7]

Dear Thomas,

thank you for your prompt reply! I have tried to put your suggesting commands in Colab

!set -e
%cd /content/
!mkdir ncrystal-build ncrystal-install
!git clone -b v3.0.0 https://github.com/mctools/ncrystal
%cd ncrystal-build
!cmake -DCMAKE_INSTALL_PREFIX=../ncrystal-install/ ../ncrystal/ -DBUILTIN_PLUGIN_LIST=XuShuqi7:MagScat
!make -j2 install
%cd ..

While it raises:

/content
Cloning into 'ncrystal'...
remote: Enumerating objects: 3573, done.
remote: Counting objects: 100% (51/51), done.
remote: Compressing objects: 100% (39/39), done.
remote: Total 3573 (delta 13), reused 19 (delta 10), pack-reused 3522
Receiving objects: 100% (3573/3573), 2.99 MiB | 14.50 MiB/s, done.
Resolving deltas: 100% (2544/2544), done.
Note: checking out '54860a9dd91ef7eda37e0c97ac63c599ca875ddb'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

git checkout -b

/content/ncrystal-build
-- The CXX compiler identification is GNU 7.5.0
-- The C compiler identification is GNU 7.5.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Trying to add plugin: XuShuqi7:MagScat
---- Trying to install plugin via remote git repo: https://github.com/XuShuqi7/ncplugin-MagScat.git (gitref main)
---- Fetched sources to /content/ncrystal-build/_deps/pld3b955785f36c934162297e1533cf110_1-src
CMake Error at /content/ncrystal-build/_deps/pld3b955785f36c934162297e1533cf110_1-src/CMakeLists.txt:7 (cmake_minimum_required):
CMake 3.15.3 or higher is required. You are running version 3.12.0

-- Configuring incomplete, errors occurred!
See also "/content/ncrystal-build/CMakeFiles/CMakeOutput.log".
make: *** No rule to make target 'install'. Stop.
/content

The problem seems to be the version of cmake. So I tried to upgrade it by:

!apt install cmake --upgrade

But the cmake is already the latest version supported by the system:

Reading package lists... Done
Building dependency tree
Reading state information... Done
cmake is already the newest version (3.10.2-1ubuntu2.18.04.2).
0 upgraded, 0 newly installed, 0 to remove and 41 not upgraded.

I have tried to modify the initial commands but failed.

I put the link to the colab here in case you want to verify:

https://colab.research.google.com/drive/12mWhrTlQzCt9YafDcNLtYCl_3dzET1K0?usp=sharing

Best regards,
Shuqi

[tkittel]

UPDATE SEPTEMBER 2024: There is now an example notebook which compiles the texture plugin in the https://github.com/mctools/ncrystal-notebooks repo .

Indeed, the plugins need cmake-3.15.

I am not sure how the google collab environment works, or what platform it is emulating, but you can try to add the following commands at the top:

curl -OLf https://github.com/Kitware/CMake/releases/download/v3.23.1/cmake-3.23.1-Linux-x86_64.tar.gz
tar xf cmake-3.23.1-Linux-x86_64.tar.gz
export CMAKE_ROOT="$PWD/cmake-3.23.1-linux-x86_64"
export PATH="$PWD/cmake-3.23.1-linux-x86_64/bin:$PATH"

That works locally on my linux machine.

[XuShuqi7]

Dear Thomas,

thank you for your suggestions! I have tried your commands but they didn't help to upgrade the cmake in the colab. While they guided me into a right direction for solving the problem. I found a tutorial to upgrade the cmake:

https://www.youtube.com/watch?v=dlmaacb-kyw

By following these commands, yours and Ignacio's, eventually I was capable of installing the NCrystal with the developed plugin. I put all the commands here in case anyone may face the same situation:

!wget https://github.com/Kitware/CMake/releases/download/v3.23.1/cmake-3.23.1-linux-x86_64.sh
!sh ./cmake-3.23.1-linux-x86_64.sh --prefix=/usr/local/ --exclude-subdir
!cmake --version

!set -e
%cd /content/
!mkdir ncrystal-build ncrystal-install
!git clone -b v3.0.0 https://github.com/mctools/ncrystal
%cd ncrystal-build
!cmake -DCMAKE_INSTALL_PREFIX=../ncrystal-install/ ../ncrystal/ -DBUILTIN_PLUGIN_LIST=XuShuqi7:MagScat
!make -j4 install
%cd ..

import sys
import os
sys.path.append('/content/ncrystal-install/share/NCrystal/python/')
os.environ['NCRYSTALDIR'] = "/content/ncrystal-install/"
os.environ['NCRYSTAL_DATADIR'] = "/content/ncrystal-install/share/NCrystal/data"
os.environ['NCRYSTALDIR'] = "/content/ncrystal-install/"
os.environ["PATH"] += os.pathsep + "/content/ncrystal-install/bin"
os.environ["LD_LIBRARY_PATH"] += os.pathsep + "/content/ncrystal-install/lib" + os.pathsep + "/content/ncrystal-install/lib64"

Once again, thank you for helping me figure out this issue!

Best regards,
Shuqi

[tkittel]

Great that you got it to work, and thanks for reporting back here with the final solution. The last snippet (with os.environ...) makes me wonder if perhaps I should make some sort of convenience module for easy setup in e.g. google collab (then again, perhaps we should just eventually add NCrystal to PyPI...)