OSSS UCX

Implementation Overview

Name:	OSSS-UCX
Vendor/Implementor:	OSSS/LANL/SBU
Open Source:	yes
Website:	https://github.com/openshmem-org/osss-ucx
User Guide:	https://github.com/openshmem-org/osss-ucx
Version Supported:	1.4
Release Date:	2011
Platforms:	x86, ARM, POWER
OS Support:	Linux
Transports:	see https://github.com/openucx/ucx
SHMEMX support:	if requested

User Installation Experiences

summit

Prereqs Not Installed

Component	From	Version
libevent	https://libevent.org/	2.1.11

Main Bits

Component	From	Version
PMIx	https://github.com/openpmix/openpmix	3.1.4
PRRTE / Open-MPI	not needed, use LSF directly
UCX	https://github.com/openucx/ucx
OSSS-UCX	https://github.com/openshmem-org/osss-ucx

Build

N.B. the configure commands here are all run from a separate build directory created as sibling of the source. All of these imply make and make install of course, and also an autogen for git-clones.

libevent

#!/bin/sh

../libevent-2.1.11-stable/configure \
    --prefix=$HOME/opt/libevent/2.1.11 \
    --disable-samples \
    --disable-debug-mode

PMIx

LSF seems to be averse to PMIx > 3.1.4 so go with that. Also seems to work with the PMIx 3.1.4 in Spectrum MPI (IBM rebrand of Open-MPI).

#!/bin/sh

../pmix-3.1.4-source/configure \
    --prefix=$HOME/opt/pmix/3.1.4 \
    --disable-debug \
    --with-libevent=$HOME/opt/libevent/2.1.11

UCX

"knem" throws warnings during execution, suspect related to CUDA memory, so disabling for now (investigating). I get (known) errors during compilation with XL compilers (https://www.ibm.com/support/pages/apar/LI74419), so falling back to GCC for now.

#!/bin/sh

../source/configure \
    --prefix=$HOME/opt/ucx/git \
    --enable-mt \
    --enable-optimizations \
    --enable-cma \
    --without-knem \
    --without-cuda --without-java

OSSS-UCX

#!/bin/sh

../source/configure \
    --prefix=$HOME/opt/osss-ucx \
    --with-pmix=$HOME/opt/pmix/3.1.4 \
    --with-ucx=$HOME/opt/ucx/git

Test Installation

$ PATH=$HOME/opt/osss-ucx/bin:$PATH

$ which oshcc
~/opt/osss-ucx/bin/oshcc

$ osh_info
# OpenSHMEM Package name:      osss-ucx
# OpenSHMEM Package version:   1.0
...
# Using UCX from:              /ccs/home/tonyc/opt/ucx/git
# UCX Build Version:           1.9
# Using PMIx from:             /ccs/home/tonyc/opt/pmix/3.1.4
# PMIx Build Version:          3.1.4
...

Running Programs

Summit's LSF has the launcher jsrun which is PMIx-aware, so we can launch directly.

login$ oshcc helloworld.c
login$ bsub -Is -q batch -W 2:00 -nnodes 2 -P $project /bin/bash
... wait for allocation...
batch$ jsrun -r 2 ./a.out
h22n13: Hello from PE    3 of    4
h22n13: Hello from PE    2 of    4
h22n12: Hello from PE    1 of    4
h22n12: Hello from PE    0 of    4

frontera

blah