CFD Proxy
                          Version 1.0.1
                             README

                          December 2014

==============================================================================
Table of contents
==============================================================================

 1.  Overview
 
 2.  Contents of this Distribution
 3.  Hardware and Software Requirements
 4.  Configuration
 5.  Related Documentation - MPI (Message Passing Interface)
 6.  Documentation - GASPI (Global Address Space Programming Interface)
 7.  Implementation details
 8.  Results
 9.  Community involvement

==============================================================================
1. Overview
==============================================================================

This is the release of the hybrid.f6 CFD Proxy.  The CFD proxy kernel 
implements and evaluates the overlap efficiency for halo exchanges in 
unstructured meshes. 

The application first reads one of the multigrid mesh levels of a 
graph-partitioned and preprocessed F6 airplane mesh (dualgrid.[12-192].tgz). 

The actual CFD kernel we have implemented is a multi-threaded green gauss
gradient calculation with a subsequent halo exchange.  We believe that -
as far as the ratio of computation and communication is concerned - this 
kernel is representative for a much broader class of applications which
make use of unstructured meshes.

This benchmark aims at strong scaling scenarios with ~ 100 mesh points per 
(x86 Ivy Bridge) core or less. For more details please see the documentation. 
For even more details please have a look at the code.

==============================================================================
2.  Contents of this Distribution
==============================================================================

This release includes the actual mesh data in ./f6 directory and source 
code for various MPI and GASPI implementations of the halo exchange pattern
in ./src directory.

==============================================================================
3. Hardware and Software Requirements
==============================================================================

1) Server platform with InfiniBand HCA or ROCE Ethernet.

2) Linux operating system 

3) MPI. Download and install. This kernel requires MPI to be interoperable 
   with GASPI. So far this has been verified to work with with Openmpi, 
   Mvapich, Mvapich2, Intel MPI and IBM Platform MPI. 

4) GASPI. For installation and download instructions of GPI2 (currently 
   the only full implementation of GASPI) please see 
   https://github.com/cc-hpc-itwm/GPI-2

5) For reading the NetCDF dualgrid meshes you will need the NetCDF library
   http://www.unidata.ucar.edu/netcdf

==============================================================================
4. Configuration
==============================================================================

1) Download and install the required software packages, netcdf, MPI, GPI2.
   Note : GPI2 needs to be compiled against your MPI library of choice in 
   order to use the  MPI interoperabilty mode. 
   (./install.sh --with-mpi=/../../foo -p PREFIX)

2) Adapt the Makefile in src. (NETCDF_DIR, MPI_DIR, GPI2_DIR, CFLAGS, 
   (USE_MPI_MULTI_THREADED, USE_GASPI, USE_NTHREADS)). The MPI lib will 
   require support for either MPI_THREAD_MULTIPLE or MPI_THREAD_SERIALIZED.
   There are quite a few possible combinations of the provided various 
   settings. Pick the one which works best for your architecture, report 
   back and/or provide your own implementation. 
 
3) Unpack the dualgrid meshes in f6. Github restrictions currently only allows 
   for rather moderate mesh size, so we are in the process of looking for a 
   better solution.

3) GASPI can make use of the startup mechanisms of MPI. Start the
   hybrid.f6.exe hybrid MPI/GASPI executable as a regular hybrid OpenMP/MPI 
   application, e.g  mpirun -np 12 -machinefile machines -perhost 2 
   ../src/hybrid.f6.exe -lvl 2 dualgrid

==============================================================================
5. MPI
==============================================================================

   For MPI Documentation
   http://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf

==============================================================================
6. GASPI
==============================================================================

   For GASPI Documentation
   http://www.gpi-site.com/gpi2/gaspi/

==============================================================================
7. Implementation details
==============================================================================

Threading model
---------------
We have adopted a threading strategy which allows read access only to points
belonging to other thread domains and read/write access to the points the
thread actually owns.  The CFD proxy hence implements 3 different face types:
faces which reside entirely within the thread domain (and which are allowed 
to update both attached data points, ftype 1), faces which only write to the
left data point (ftype 2) and faces which only write to the right data point 
(ftype 3). 
In order to maximize scalar efficiency we have split the thread domains into 
sub domains (colors) which fit in the L2 cache.  The threads then  iterate 
over these colors until the respective thread domain has been  completed. 
We perform strip mining both with respect to initialized and finalized 
points. The colors in this implementation hence require a substantial 
amount of meta data about e.g about which data points are visited for the
first time (for each color), or which points are visited for the last time.  

Overlapping communication and computation
-----------------------------------------
For an efficient overlap of computation with communication we need to trigger
the communication as early as possible. When preprocessing the mesh we hence 
mark up all finalized points per color which belong to the mesh halo. 
The mesh faces are reordered such that halo points are updated first during
the gradient reconstruction. The thread which completes the final update 
(for a specific communication  partner) on these halo points then triggers 
the communication – either via  MPI_Isend, MPI_Put or gaspi_write_notify. 
In order to achieve this, the code additionally keeps track of the status 
of the inner halo (ghost cell) points for every color and maintains a list 
which halo points (per color) have to be send to which neighboring rank. 
We note that while this method allows for a maximal overlap of communication 
and computation, it either requires a full MPI_THREAD_MULTIPLE or a 
MPI_THREAD_SERIALIZED MPI version.  For the latter version we have 
encapsulated the actual MPI_Isend and MPI_Put in an OpenMP critical section.
We have enhanced the CFD proxy with a multithreaded gather/scatter operation
from/into the MPI/GASPI buffers. For this all colors use additional meta 
data with respect to required points (per color) in their outer halo. 
We note that for high scalablity on Xeon Phi  (and probably most other 
many-core architectures) this kind of optimization (multihreaded pack/unpack)
appears to be mandatory.

==============================================================================
8. Results
==============================================================================

For current results on a Fat Tree FDR/Ivy Bridge please have a look at the 
documentation. New, shiny and much better results for Xeon Phi have been  
included in the wiki.

https://github.com/PGAS-community-benchmarks/CFD-Proxy/wiki#results

==============================================================================
9. Community involvement
==============================================================================

We encourage the HPC community to provide new patterns or improve existing
ones. No restrictions apply. Instead we encourage you to improve
this kernel via better threadings models, and/or better communication and/or
more scalable communication API’s and/or better programming languages. Bonus
points are granted for highly scalable implementations which also feature
better programmability.