-
Notifications
You must be signed in to change notification settings - Fork 15
Compiler deployment on BB5
These are brief instructions to deploy commercial compiler toolchain on BB5.
First clone the Spack repository and stable branch for deployment:
git clone https://github.com/BlueBrain/spack.git
cd spack
git pull
git checkout bbpv-0.4.3
We will use SPACK_ROOT environmental variable that point to spack directory:
export SPACK_ROOT=`pwd`
To have spack in PATH, add following to .bashrc (replace SPACK_ROOT by absolute path)
export PATH=$SPACK_ROOT/bin:$PATH
source $SPACK_ROOT/share/spack/setup-env.sh
With this, Spack should be usable now. We can check architetcure of the system as:
→ spack arch
linux-rhel7-x86_64
And installed packages as:
→ spack find
==> 0 installed packages.
Commercial softwares like Intel, PGI and Allinea need licenses. These are usually simple text files with license key or license server details. To allow Spack to automatically configure licenses, we can license files in spack/etc/spack/licenses directory. We have licenses stored in below mentioned repository. We can clone it under Spack as:
git clone ssh://bbpcode.epfl.ch/user/kumbhar/spack-licenses $SPACK_ROOT/etc/spack/licenses
The directory has following layout:
$ tree $SPACK_ROOT/etc/spack/licenses
.
├── allinea-forge
│ └── Licence
├── intel
│ └── license.lic
└── pgi
└── license.dat
We will use APPS_DIR_PATH environmental variable to point to apps directory. Lets say we define it as:
export APPS_DIR_PATH=/gpfs/bbp.cscs.ch/apps
For testing we can set it as: export APPS_DIR_PATH=$HOME/apps
We will install compilers and tools under following directory hierarchy. This can be changed and we will discuss it later.
$APPS_DIR_PATH
.
├── compilers
│ └── install
│ └── modules
│ └── .cache
│ └── .mirror
├── tools
│ └── install
│ └── modules
│ └── .cache
│ └── .mirror
Some software tarballs are very large and often time consuming to download (e.g. Intel Parallel Studio is about ~3 GB). In order to avoid download on every new installation, we can create a mirror where software tarballs could be stored. This is also helpful when we have clusters where network connection is slow or network access is not available (on certain systems). You can check spack mirror documentation for details and extra options. In below script we are simply downloading software tarballs one-by-one:
spack mirror add bbpv_spack_mirror $APPS_DIR_PATH/compilers/.mirror
packages_to_mirror=(
'[email protected]'
'[email protected]'
'[email protected]'
#'[email protected]'
)
for package in "${packages_to_mirror[@]}"
do
spack mirror create -d $APPS_DIR_PATH/compilers/.mirror --dependencies $package
done
Some softwares do need manual download from registered account (e.g. PGI compiler). Once we download the tarball, we can manually copy them to mirror directory. For example, PGI compiler 2017.10 is downloaded in below mentioned location. Also, Intel Parallel Studio takes long time to install and hence better to use pre-downloaded tarball as:
mkdir -p $APPS_DIR_PATH/compilers/.mirror/pgi
cp /gpfs/bbp.cscs.ch/home/kumbhar/DOWNLOADS/pgilinux-2017-1710-x86_64.tar $APPS_DIR_PATH/compilers/.mirror/pgi/pgi-17.10.tar.gz
mkdir -p $APPS_DIR_PATH/compilers/.mirror/intel-parallel-studio
cp /gpfs/bbp.cscs.ch/home/kumbhar/DOWNLOADS/intel-parallel-studio-cluster.2018.1.tgz $APPS_DIR_PATH/compilers/.mirror/intel-parallel-studio/
Note : Based on version name, you have to rename tarball while copying to mirror i.e. pgilinux-2017-174-x86_64.tar.gz to pgi-17.4.tar.gz
By default Spack installs packages inside $SPACK_ROOT/opt/. We can provide following configuration files to change location, module naming scheme and system provded packages:
config:
install_tree: $APPS_DIR_PATH/compilers/install
source_cache: $APPS_DIR_PATH/compilers/.cache
module_roots:
tcl: $APPS_DIR_PATH/compilers/modules/tcl
lmod: $APPS_DIR_PATH/compilers/modules/lmod
install_path_scheme: '${PACKAGE}-${VERSION}'
modules:
enable::
- lmod
- tcl
prefix_inspections:
bin:
- PATH
bin64:
- PATH
lib:
- LD_LIBRARY_PATH
lib64:
- LD_LIBRARY_PATH
lib/pkgconfig:
- PKG_CONFIG_PATH
lib64/pkgconfig:
- PKG_CONFIG_PATH
'':
- CMAKE_PREFIX_PATH
tcl:
all:
filter:
environment_blacklist: ['CPATH', 'LIBRARY_PATH']
naming_scheme: '${PACKAGE}/${VERSION}'
hash_length: 0
whitelist:
- 'gcc'
- 'pgi'
- 'intel-parallel-studio'
- 'llvm'
blacklist:
- '%gcc'
NOTE : INTEL_ROOT is special environmental variable for Intel compilers! So don't set PACKAGE_ROOT into modules.yaml!
packages:
autoconf:
paths:
[email protected]: /usr
version: [2.63]
automake:
paths:
[email protected]: /usr
version: [1.13.4]
bison:
paths:
[email protected]: /usr
version: [3.0.4]
cmake:
paths:
[email protected]: /usr
version: [2.8.12]
...
Lets copy the provided system configurations in $HOME/.spack where Spack will search by default:
mkdir -p $HOME/.spack
rm -rf $HOME/.spack/*
cp $SPACK_ROOT/sysconfig/bb5/compilers/* $HOME/.spack/
# mirror config gets deleted, lets re-create
spack mirror add bbpv_spack_mirror $APPS_DIR_PATH/compilers/.mirror
One can look at configurations that spack is going to use as:
→ spack config get config
config:
install_tree: $APPS_DIR_PATH/compilers/install
template_dirs:
- $spack/templates
module_roots:
tcl: $APPS_DIR_PATH/compilers/modules/tcl
lmod: $APPS_DIR_PATH/compilers/modules/lmod
dotkit: $spack/share/spack/dotkit
build_stage:
- $tempdir
- /nfs/tmp2/$user
- $spack/var/spack/stage
source_cache: $APPS_DIR_PATH/compilers/.cache
misc_cache: ~/.spack/cache
verify_ssl: true
checksum: true
dirty: false
install_path_scheme: ${PACKAGE}-${VERSION}
By default Spack will find compilers available in $PATH. We can see available compilers using :
→ spack compiler find
→ spack compilers
==> Available compilers
-- clang rhel7-x86_64 -------------------------------------------
[email protected]
-- gcc rhel7-x86_64 ---------------------------------------------
[email protected]
These are default compilers installed on system. For software development we often need to install multiple compilers to meet requirements of different users. Also, compilers are expensive to install considering long build time. Often we don't need to reinstall compilers from scratch. In production environment we typically preserve compiler installations and re-compile all other software stack (e.g. mpi and mpi-dependent packages). In this case it is good practice to install compilers in separate directory.
We can achieve this by using sample config.yaml that we have already copied to $HOME/.spack/. Here is the script to install all compilers in one go:
compilers=(
'[email protected] +advisor+clck+daal+gdb+inspector+ipp+itac+mkl+mpi+rpath+shared+tbb+vtune'
'[email protected]'
'pgi+network+nvidia'
'[email protected]'
'[email protected]'
'[email protected]'
)
core_compiler='[email protected]'
spack compiler find
for compiler in "${compilers[@]}"
do
spack spec $compiler %$core_compiler
spack install $compiler %$core_compiler
done
Spack automatically creates lmod as well as tcl modules. We can use them using MODULEPATH or as:
module use $APPS_DIR_PATH/compilers/modules/tcl/linux-rhel7-x86_64/
And now module avail will show installed compilers:
$ module avail
--------------------------------------- $APPS_DIR_PATH/compilers/modules/tcl/linux-rhel7-x86_64/ ---------------------------------------
gcc/5.4.0 gcc/6.4.0 gcc/7.3.0 (D) intel-parallel-studio/cluster.2018.1 pgi/17.10 intel/18.0.1
Alright! All our compilers are ready to use!
If you change naming scheme in modules.yaml then we can re-generate all modules as:
spack module refresh -m tcl -m lmod -y --delete-tree
For a network installation of PGI compilers, we must have to run below installation script on each system on the network where the compilers and tools will be available for use. For example, we installed PGI compilers on bbpviz1. If we try to use PGI compiler on bbpviz2, we get :
$ pgcc hello.c
pgcc-Error-Please run makelocalrc to complete your installation
In this case we have to run following on every node where users are going to use PGI compiler to compile the code:
$ module load pgi/17.10
$ which makelocalrc
$APPS_DIR_PATH/compilers/install/pgi-17.10/linux86-64/17.10/bin/makelocalrc
$ makelocalrc -x $APPS_DIR_PATH/compilers/install/pgi-17.10/linux86-64/17.10 -net $APPS_DIR_PATH/compilers/install/pgi-17.10/linux86-64/17.10
Once compielrs are installed, we can deploy common tools/debugger with Spack. It is best practice to separate compiler installation from other tools so that in case of system upgrade we can keep installed compilers and re-build all other tools/mpi etc.
# make sure following variables are set:
export APPS_DIR_PATH=/gpfs/bbp.cscs.ch/apps/
export SPACK_ROOT=/gpfs/bbp.cscs.ch/apps/spack/
# update repository
cd $SPACK_ROOT && git pull
git checkout bbpv-0.4.3
# likwid installation group
export LIKWID_GROUP=likwid
# initialise Spack environment as:
export PATH=$SPACK_ROOT/bin:$PATH
source $SPACK_ROOT/share/spack/setup-env.sh
# lets copy the spack configurations for tools:
mkdir -p $HOME/.spack
rm -rf $HOME/.spack/*
cp $SPACK_ROOT/sysconfig/bb5/tools/* $HOME/.spack/
# we can now install all tools as:
spack install [email protected]
spack install [email protected]
spack install [email protected]
spack install [email protected]
spack install likwid +setgid
spack install [email protected]
And if we want to re-generate all modules, do:
spack module refresh --module-type tcl --delete-tree --yes
We can use installed modules as:
module use $APPS_DIR_PATH/tools/modules/tcl/linux-rhel7-x86_64/