Testing pipeline (gitlab)

Notes on gitlab pipeline

Regression repository

• The regression repository contains a submodule pointing to version of MOM6-examples.
• A directory structure mirroring MOM6-examples experiments.
• The regression files (ocean.stats.gnu) corresponding to the particular version of MOM6-examples.
• The regression files are unique to compiler versions, system environment and platform so need not be publicly shared and are thus hosted at GFDL behind the firewall ([Gaea-stats-MOM6-examples].(https://gitlab.gfdl.noaa.gov/ogrp/Gaea-stats-MOM6-examples)). Being behind the firewall also allows us to use the front-end (gitlab) to manage tests on our HPC.
• A illustrative copy of the regression repo is on GitHub but is not guaranteed to be up to date.

Gitlab pipeline

• The list of tests to run is defined by the "stats" files (ocean.stats.gnu, seaice.stats.intel, ...) recorded in the regression repository.
• There are 59 experiment configurations (as of Sept 2017) which leads to ~400 runs. When the queue wait time is minimal, the testing on c4 takes approximately 45 minutes end-to-end.

The pipeline is implemented via:

• a YAML file in MOM6, .gitlab-ci.yml;
• a repository of Makefiles that manages the cloning, compiling, running and testing: MOM6 Regression System (MRS).

The process is readily extracted by reading the YAML file but in words is:

1. Setup stage
   • Recursive clone of regressions repository dev/gfdl
   • Merge of MOM6 pull request with dev/gfdl
2. Build stage
   • Compiles a symmetric and non-symmetric executables for each configuration:
     • ocean-only
     • ice-ocean (SIS2)
     • land-ice-ocean
     • fully coupled
   • For each of the above using each of the following compiler vendor/mode combinations:
     • Gnu debug
     • Gnu repro
     • Intel repro
     • PGI repro
     • Gnu static-memory repro
3. Run stage Note: The run stage does not generate an error code unless the job fails to submit.
   • For all experiment configurations:
     • Run with symmetric executable for each of Gnu, Intel and PGI repro.
     • Run with non-symmetric executable for each of Gnu, Intel and PGI repro with an alternative layout.
   • Run with static-memory Gnu repro
   • Run restart tests with symmetric Gnu repro executable.
4. Test stage
   • Check that non-symmetric tests ran and reproduce regression results.
   • Check that symmetric tests ran and reproduce non-symmetric output.
   • Check that static tests ran and reproduce regression results.
5. Cleanup stage
   • Used to cleanup tar files used for caching executables and output between stages.

Passing of data between stages is implemented via tar files with names including the environment variable $CI_PIPELINE_ID which is unique to each pipe. This avoids a bottleneck when dealing with very large artifacts.