Fixes towards achieving bit-for-bit reproducibility in GC-Classic upon restart

[lizziel]

Name and Institution (Required)

Name: Lizzie Lundgren
Institution: Harvard University

Describe the update

This PR applies changes in GC-Classic needed to achieve bit-for-bit reproducibility when breaking up a run in time. Please note that with these updates out-of-the-box GC-Classic will still not have reproducibility upon restart due to a limitation of HEMCO. HEMCO reads inputs, including GEOS-Chem restart file variables, as single precision when most of these fields in GEOS-Chem are double precision.

Separately I implemented an update that reads the GEOS-Chem restart file locally, preserving native precision found in the file. With this PR and that update on top of it I am able to achieve bit-for-bit reproducibility in GC-Classic full chemistry benchmark simulation when breaking up the run in time. We are not going to bring in the update to read the restart file locally because we need to use HEMCO for its flexgrid capability. However, I will make that update available as a dev branch should anyone want to use it (see here). An update to HEMCO to read native precision will come later.

Updates within GEOS-Chem that change benchmark output:

1. Fix a parallelization issue in GEOS-Chem's computation of State_Chm%DryDepNitrogen used in the HEMCO soil NOx extension and saved out to the GEOS-Chem restart file. I believe this was previously undetected because the differences are at the level of double precision numerical noise and thus were previously obscured by outputting to file as single precision. This update includes writing to the restart and diagnostic files using native precision which exposed the issue.
2. Fix bug in Chem_ORVCsesq restart file entry in GC-Classic fullchem HEMCO_Config.rc.
3. Register State_Met and State_Chm arrays in GC-Classic History using the native precision used in the model. Previously the precision was unspecified and outputs thus used the default of single precision.
4. Specify REAL8 as the output precision of State_Diag%SpeciesRst which is the species concentration array written to the restart file GC-Classic. Previously it was set to REAL4.
5. Convert species concentration units of mol/mol <-> kg/kg in main.F90. This update ensures error is not introduced due to unit conversions used for saving species to the restart file in mol/mol. Conversion to mol/mol must occur every timestep as part of updating diagnostics used for restart variables so that GEOS-Chem Classic can achieve bit-for-bit reproducibility upon breaking up runs. We still use kg/kg dry air throughout main.F90. Conversion from mol/mol to kg/kg occurs at the start of every timestep. This also means that mol/mol is not converted to kg/kg in the subroutine that retrieves restart variables from HEMCO during initialization since it will occur later.
6. Move setting prescribed CH4 to just after emissions in GC-Classic. It is necessary to move where prescribed CH4 is set because it is dependent on PBL height and previously PBL height was not updated until after Set_CH4 was called. This introduced small differences in GC-Classic when breaking up runs. CH4 is now set using updated PBL height and is grouped with emissions where other prescribed surface VMRs are applied. It is still applied outside of emissions.

This PR also contains several no-diff updates:

• Fix minor error in checking variables after State_Chm array allocation
• Remove unused local variables in ucx_mod
• Move getting restart var DELPDRY to subroutine GC_Get_Restart. This is a no diff update but now allows print of min, max, sum of restart variable Met_DELPDRY to be included with all of the other restart variables retrieved from HEMCO into GEOS-Chem for use upon startup.
• Add GeosUtil/print_mod.F90 for specialized model print options. This module currently has one subroutine only, to print min, max, and mean of species concentrations to log. It currently only prints if root thread and does not convert units. Optional arguments allow printing one species or a consecutive sequence of species indexes. The current units are printed to the log.
• Clean up prints in Get_GC_Restart subroutine so that all species initial concentrations are printed to log, including if background values are used (with note about that), as well as initial concentrations of other arrays in the restart file that are read into the model.
• No diff fixes to SOIL_WETDEP and SOIL_DRYDEP subroutines used to update State_Chm%WetDepNitrogen and State_Chm%DryDepNitrogen used in the Soil NOx extension in HEMCO. Changes include removing the unused level argument and uncommenting a commented out Soil NOx extension logical used to determine whether to call SOIL_WETDEP in Do_Washout_as_Sfc.

Expected changes

This update will cause small differences in all GEOS-Chem Classic simulations.

Reference(s)

None

Related Github Issue

geoschem/HEMCO#18
#111
#2014

[yantosca]

@lizziel, I think I know what this may be. The N deposition code (I think GET_N_DEP) is trying to sum data into a 2-D array within a 3-D parallel loop. This might lead to numerical differences if the loop over levels (L) is parallelized.

You could get around that by declaring the variable to hold the level sum with !$OMP REDUCTION(+: varname). But it might just be more elegant to store the N deposition in a 3-D field of State_Chm which could be summed in the vertical outside of a parallel loop. I think the N deposition code pre-dated the State objects in GC v11-01.