Skip to content

Notes from NorESM CTSM Friday meetings.

Jump to bottom
Rosie Fisher edited this page Dec 3, 2024 · 204 revisions

Dec 6th 2024

WINS

  • Update on puthon dependencies in FATES tests: we discussed this at the software meeting and Greg is going to make a PR that moves the FATES parameters that are switches into the namelist variables.

trying to get rid of python dependencies in testing routines, which means that we would not do FATES tests of things that are turined on by the parameter file? FATES needs NumPy and SciPy. The parameter file in FATES is gneerated by a python script.

Nov 29nd 2024

WINS

  • INES2 budget news
  • MEGAN running and sending non-absurd isoprene to NorESM (which sends back v odd climate but...)

AGENDA

  • Issue with git clone. Is v slow. Maybe to do with 12k references? Or stale branches?

  • Matvey trying to get rid of python dependencies in testing routines, which means that we would not do FATES tests of things that are turined on by the parameter file? FATES needs NumPy and SciPy. The parameter file in FATES is gneerated by a python script.

  • Plan for PR acceptance and modifications. (if Matvey & Mariana are both here)

    • HYPOTHETICAL CASE

    • Rosie makes PR and runs test suite.

    • Matvey has changes to make to make code nicer from code review - does he:

    • a) add changes to existing PR and go back through testing cycle or

    • b) make a new subsequent PR?

    • Matvey has changes to make to make THAT ARE CRITICAL

    • a) add changes to existing PR and go back through testing cycle or

  • Think about how to run Marit's diagnostics on megan and dry dep testig runs.

  • Update on dry deposition code status.

    • Need to think about benchmarks (maybe in OsloAero if it behaves itself.)

  • 78 PFT surface dataset changes to parameter file. How to handle NorESM default parameter file? (i.e. for drydep, megan, twostream, crops, calibration). Make xmlchange scripts to patch onto changing FATES defaults?

TO DO FROM LAST MEETING

  • rosie to make a new param file with MEGAN changes AND twostream model. And add to the notes for the new run. DONE

  • Matvey to make changes to the way leaf C_i (internal co2 concentration in the mesophyll) is moved between FATES and CTSM. NOT DONE YET.

  • AFTER THAT Submit PR on MEGAN back to main CTSM and FATES repos. NOT DONE YET

MEGAN global totals. image DRYDEP image

Nov 22nd 2024

https://github.com/NorESMhub/noresm3_dev_simulations/issues/61 <fates_paramfile>lnd/clm2/paramdata/fates_params_api.36.1.0_14pft_c241003_megan.nc</fates_paramfile>

Nov 15th 2024

  • Status of MEGAN testing
    • Mariana have done quite a bit of testing. Conclusion: NorESM2.5_alpha08 will need to have Rosie's PR on MEGAN.
    • TODO: Mariana has made a PR to Rosie's branch. Will need to be accepted (by Rosie). Mariana will then be merged into NorESMhub.
  • Heiko's request for CLM fields (available with FATES?)
  • Discuss any issues with testing protocol
    • MEGAN PR also includes the new test suite. 24 test. 20 are currently passing, two don't pass because of we don't have numpy, needed for fixedBiogeography. Not sure about the last two. But take home is that this is quite good (we test much more now than earlier). We run with updated CIME (in NorESM2.5). Mariana have changed the tests that used months to use days.
  • Albedo issue solved in tuning task force.
    • Win!!!
  • little bit of work in drydep PR still tbd.
    • Season indexing needs to be finished. Should be ready by next Friday. (This will go into alpha09).
  • Status of marit’s disgnostics. If/how to integrate into the NorESM workflow.
    • We will come back to this next week when Marit is available.
  • CMIP7 data request, which variables can we provide.
  • CMIP6+ forcing data are available (I think). Should we test these.

Nov 1st 2024

  • First meeting of tuning task force. https://github.com/NorESMhub/CTSM/wiki/Useful-NorESM-github-links

  • Stuff to work on in hackathon

  • Rosie : Report on status of MEGAN and drydep PRs. DONE

  • ALL : Discussion of diagnostic strategy.

  • ALL : Discussion of how to document offline simulations DONE

  • ALL : Figure out how to run diagnostics package(s) (to socialize this skillset).

Here is a possible suggestion for division of labour in things that need to go into alpha09.

  • @marit : diagnostics. Add MEGAN and DRYDEP fields.
  • @matvey : testing or PRs. variables, setup, analysis, etc.
  • @kjetil : code review?
  • @rosie : query dry deposition differences (why are they almost 0? maybe cpl.hist velocity is right variable). Document the SP cases. Alter the parameter file to include crop PFT area in hlm_pft_map. Sort out 2FA!
  • @mariana : Finidat issues. Make issues to create coupled simulations with drydrp and megan branches (and CLM-SP for comparison)? Testlist & testmods.
  • @alok : Run test coupled simulations with drydrp and megan branches?
  • @ane : learn how to do testing from mariana.

Oct 25th 2024

  • alpha tag/betzy updates. 07 is done. 08 is in progress. 09 is FATES updates.
    • We will have a full-day hackaton Nov 1.
  • CTSM FATES board (this is a thing, that will be discussed at the SE meeting). https://github.com/orgs/ESCOMP/projects/10/views/1
    • Rosie will try to join escomp/ctsm SE meeting in two weeks.
  • Diagnostics
    • Land diagnostics plots look weird.
  • Megan outputs.
  • Megan and Fire updates after alpha08.

Oct 18th 2024

  • Discuss alpha tags plan.
  • Question about accounts for debugging NorESM runs and Betzy queues.
  • Megan PR
  • Drydep PR
  • Land diagnostics?

Notes on testlist commands. ./query_testlists --list categories

The simulations go here https://github.com/NorESMhub/noresm3_dev_simulations

The discussion of tags is here: https://github.com/NorESMhub/NorESM/discussions/575

Oct 11th 2024

  • ACTIONS.

    • Make noresm branch of FATES. done
    • Rosie: make changes to config to allow runs to crash on Betzy.
    • Rosie: Add changes to externals.cfg to point to rosie's drydep branch. done
    • Rosie: Make the drydep code work! done
    • Rosie: read system testing in noresm documentatation

  • Update on dry deposition PR status/testing.

  • Strategy for PR for FATES-side changes.

  • Strategy for debug-testing? Do we need a less expensive case?

Oct 4th 2024

  • CTSM modules - matvey has a branch, update_login3. Needs to be turned on and turned off... (only login3 is working on Betzy right now).
  • Improved formatting of notes :)
  • Update on simulations: Status of new surface dataset testing. FWIW the calibrated parameter file to use for additional simulations is here. Alpha06 tag. Points to new surface dataset without the SE grid error. *. Where to put the CLM53-BGC initial condition data? In a 'do not use' folder in main inputdata directory?
  • Drydep and megan PRs: Testing of drydep code. If and how to submit to CTSM/NGEE repos. Mariana will send instructions on how to run coupled mode. Rosie will try and run. Copy branch and merge the main repo into it should fix this...
  • Mats' issue with runoff discussion. Mariana's idea to run CLM-BGC and FATES-SP to compare runoff into labrador sea. Maybe invite Mats and Ada to discuss tuning strategy for this.
  • FATES SPITFIRE OAAT ensemble: Ran over the weekend. Need to analyse.
  • 16pft surface dataset: still need this for the SE grids AFAIK. There is an issue to make 4x5 degree dataset, and so we should also ask them to make an ne30pg3 grid https://github.com/ESCOMP/CTSM/issues/2810

Sept 27th 2024

  • Wins: lots of projects funded, including NorESM4CMIP7 and INES2.
  • drydep code
  • New fire parameters in the making based on a fire PPE (Rosie)
  • Matvey testing and making NorESM working on Betzy after updates on nodes.
  • CTSM5.3 brought in (including new surface datasets and dust parameterization). Required updates to CIME ++. Should be ready by next Thursday (when new simulations will be started).
    • Marit will run CTSM5.3-FATES-SP spin-up with default setup. Needed before next Thursday (Oct 3.).
    • Rosie will update FATES parameter file. Potentially rerun spin-up again with this if time (good to do, but not critical)
    • Mariana will verify functionality in coupled NorESM2.5.

Sept 20th 2024

  • Think about what happens in FATES with 78pft surface dataset.
  • Thinking about FATES-SP runs. Ok?
  • Status of drydep issue
    • Rosie have made a draft version. Need to look up Wesely 1989 to understand how the season indexes should work, to correctly calculate this in Fates.
  • Regridding and plotting
  • Mariana have written info on: :
  • Timeline (hopefully):
    • New CTSM5.2 SE dataset (with corrected soil properties) available:
    • CTSM5.3 hopefully ready around end of next week
    • For now, we could do a short spin-up with fixed 5.2 surface data, to run coupled NorESM. (CAM-Nor people might want to look at this).
    • We probably need a couple of weeks before we can do i) CTSM5.3 simulations and ii) NorESM-FATES-Nocomp simulations. (Need to keep in mind that NorESM-FATES-SP probably also have problems with drydep)

Sept 13th 2024

  • System updates
    • Matvey will test new modules on betzy on Monday.
  • Dust
    • New initial condition files (for CLM53) and surface datasets being made at NCAR with the new non broken surface datasets.
  • Spinup:
  • NorESM meetings will shift to being development meetings.
  • Fire code: need to write last part of fire emissions code. Maybe not required to make FATES NOCOMP work in coupled mode.
  • MEGAN
    • https://github.com/NorESMhub/fates/pull/8
    • FATES-SP mode produces VOC values presumably using ghost pft indices.
    • It is plausble that FATES NOCOMP mode will do the same thing, and produce the same erroneouosly mapped VOC emissions, in the absence of getting the MEGAN FATES code working.
  • Dry deposition
    • Drydep scheme needs 1) wesley vegetation type and 2) season index.
      1. Wesley vegetation type can be created by FATES
      1. Season is a strange beast. Need to look at the original paper to figure out how to calculate it. Potentially send the season index from FATES

Sept 6th 2024

  • Action items and updates from NorESm2.5 meeting

    • NorESM2.5 code 'chill' at the end of the year.
    • means we need to prioritize outstanding chemistry coupling issues and nitrogen coupling.
    • There will be a spinup and calibration 'task team' lead by Jerry. Need to think about the 'equilibrium' state of FATES wrt LULUC.

  • Next step is to run a simulation which has FATES-SP and compare with the CLM51-SP simulations.

    • Alok will run this and Mariana will write an issue with instructions in it.
    • This will be 'out of the box' FATES per the latest NorESM-CTSM tag
    • AFTER that, Rosie to point to new FATES parameter file, assuming that nothing goes wrong.
    • AFTER that, we will check that we have all the most up to date snow schemes etc.

  • Discussion of dry deposition issue.

    • Next friday 13th we will do 9.30-11.30ish mini hackathon on this.

  • Status of fire and VOC PRs.

  • Dust emissions and surface file in SE vs FV

    • Marit to make issue on ESCOMP about the strange PCT_CLAY strangeness.

Aug 16th 2024

  • Status of code after Betzy module debacle.
  • Status of FATES calibration and spin-up.
  • Recap of discussion of diagnostics package.
  • FATES Initial condition interpolation for SE grid

Aug 9th 2024

June 28th 2024

June 21st 2024

  • Updates
    • Calibration
    • Trigrid update
    • Ada discussion r.e. august runs Configuration notes: noresm2_5_alpha03_v1 in my fork git clone https://github.com/mvertens/NorESM.git cd NorESM git checkout feature/noresm2_5_alpha03_v1

TRIGRID case: ./create_newcase --case /cluster/home/mvertens/noresm/test_nf1850oslo_trigrid --compset NF1850oslo --res a%ne30np4.pg3 _l%0.9x1.25_oi%ne30np4.pg3_r%r05_w%null_z%null_g%null_m%tnx1v4 --project nn9039k --run-unsupported

SE grid case: ./create_newcase --case /cluster/home/mvertens/noresm/test_nf1850oslo_trigrid --compset NF1850oslo --res a%ne30np4.pg3 _l%ne30np4.pg3_oi%ne30np4.pg3_r%r05_w%null_z%null_g%null_m%tnx1v4 --project nn9039k --run-unsupported

Status of the FATES code on the CTSM repo?

  • Discussion of Betty's PhD project.

May 31st 2024

Agenda

  • Updates
    • CICERO 'NextGenCarbon' project funded :)
    • calibration
    • trigrid
    • land use discussion
    • N cycle plans
  • Compsets discussion
  • Need to crate a protocol for NorESM3 coupled spinup. Perhaps in September. Both for :
    • the standard coupled model spinup
    • Emission-driven spinup
    • FATES spinup (with land use change)

May 24th 2024

Agenda

  • Fire and BVOC code (Matvey doing coldstart temperature initialization for CTSM. Needed to make excess ice the default option.)
  • Calibration updates
  • BECCS v0 implementatioh. Does not pass balance check yet. https://github.com/rosiealice/ctsm/tree/BECCS_v0
  • Meeting with Charlie (and Ryan) on Wed at 5pm CET to talk about land-use spinup (& nutrients). -- Mariana, Ada and Mats gave talk to the CESM co-chairs meeting on Tuesday on NorESM plans.
  • Tri-grid and diagnostics. -- Mariana and Bill have done loads of work dealing with the complexities of the mapping, building on some previous work in E3SM.

May 3rd 2024

Agenda

Action:

  • Explore trigrid in CMEPS. Make sure it works
  • Test trigrid:
    • F compset, 2deg atm, 1deg land. Both FV
    • F compset, 1deg atm, 1deg land. Atm SE, land FV
    • F compset, 1deg atm, 1deg land. Both SE
    • B compset, 1deg atm, 1deg. SE/FV (atm/land). Check conservation

April 26th 2024

Agenda

  • Model calibration update
  • Discussion of ensemble diagnostics conversation with NCAR.
  • Update on to do list for NorESM.

Action- Rosie to arrange discussion on SE diagnostics package with Will, Daneiele, Mariana...

April 12th 2024

  • Matvey working on the ex cess ice in CTSM. The new lower (frozen) initial soil temperature breaks a FATES-hydro test which was previously passing but liekly would have failed in a longer test and exposes the inability of FATES-HYDRO to deal with totaly frozen soils, which we should probably fix independently. We can discuss this in the FATES SE meeting on Monday.

  • Report from the CMCC-NorESM meeting and follow-up items. (we didnt talk about this but the meeting was great and CMCC are still using CLM4.5/5 and so there are not any real imminent land related collaboration). There was discussion of how to have common github repos (ongoing), and how to do CMIP7 spinup, and how to do SE diagnostics.

  • Status of fire emissions coupling. Made PRs to both NorESM/ctsm and NorESM/fates branches. Still missing the part to deal with the emission species. Matvey will work on this potentially next week.

  • Status of MEGAN coupling, per https://github.com/NorESMhub/CTSM/issues/36

  • For the PRs to the FATES and CTSM main repositories, we should 1) wait for PRs to be accepted into the main FATES forks. 2) rebase onto the main FATES/CTSM repos. 3) Make new PR from branch into those repos.

  • CH4 NorESM doesn't have emission driven methane. But the reasons why seem to be that CH4 does not come out of the land, despite being calculated there. Mariana does not think that moving CH4 fluxes to the atmosphere might not be too difficult. Dirk might be doing interactive CH4 in NorESM as part of CN-coESM project. There are no hooks in mediator to accept it from the land, and no hooks in CAM to do with it so that would need fixing. In CLM5.2, wetland extent is going to be lower (from Will). Action is to email Dirk (Rosie). Zosia (CICERO) is potentially interested medium-term in using a fully coupled emissions-driven methane model and greater emission driven capacity is desirable. A coupled of people on the CLM side are also potentially interested in this.

  • Discussion of manage_externals change to git submodule. Possibly Matvey and/or Mariana to push back to NCAR on the need for community engagement on change tp basic workflow and dependency on external organizations system python settings.

March 14th 2024

Meetings in March 2024, including 2 day hackathon, were dedicated to fire emissions coding. Project board link

Feb 7th 2024

  • NorESM2.5 vs NorESM3: Fire emissions, bvoc's, nitrogen in FATES. Hackathon for fixing these issues before September?
    • Fire emissions plumbing next week (every two weeks friday 9:30-11:30)?
    • NorESMhub/fates branch and NorESMhub/CTSM branch for fire emissions development. Give Rossie/Kjetil rights to these.
  • NorESMhub/CTSM tag with FATES-two-stream parameter file: feature/two-stream. rosie's branch with two-stream on
  • INES interim meeting in September.
  • Frost mortality: low prior, no plant hydro for now.

Feb 2nd 2024

  • FATES and Nutrients moved to norESM3?
  • Outcome of frost mortality discussion.
  • Update on coupled FATES-SP testing strategy.
  • Update on Adrianna's parameters.

Jan 26th 2024

Suggested compsets and parameter files for coupled simulations (to be filled in, pointing to separate tags in NorESMhub/FATES):

  • FATES-SP twostream with default parameters.

  • FATES-SP new configuration/parameters (Rosie and Adrianna)

  • FATES-NOCOMP existing parameterization from Jessie (with or without the modifications to leaf layers). Needs to be spun up (1850, transient etc.)

  • FATES-NOCOMP/CN.

  • FATES-NOCOMP/CN with land use.

  • FATES with nutrient coupling goals for NGEE/FATES and NorESM (timeline, demand ...)

Jan 25th 2024

  • Defining configuration for the first tests of NorESM2.5. Waiting for some OsloAERO tunings from NCAR. Mariana has some issues with restarts in 1850.
  • We need to define a configuration in FATES for testing.

Hierarchy of FATES complexity runs.

  1. FATES-SP twostream with default parameters. The twstream model is turned on in the parameter file by changing the fates_rad_model variable to "2"
  2. FATES-SP new configuration/parameters (Rosie and Adrianna)
  3. FATES-NOCOMP existing parameterization from Jessie (with or without the modifications to leaf layers). Needs to be spun up (1850, transient etc.)
  4. FATES-NOCOMP/CN. CAN WE MAKE THIS PART OF NORESM3?
  5. FATES-NOCOMP/CN with land use.

Summary of Spectral Element diagnostics

  • SE runs do not work in the normal diagnostics package.
  • xESMF can be used in python to plot data, but isn`t part of a running diagnostics package yet.
  • Tri-grid capacity can in principle run atmosphere on SE and land on FV but this is scientifically wierd.
  • Keith O is using an old NCL diagnostics package with FV coupled versions to debug the CLM5.1 coupled issues (effort is mostly around snow and albedo and shrubs).
  • The existing diagnostics package is actually a bit hard to integrate into an efficient workflow as it needs to be moved outside the HPC firewall to view the results and that can be clunky.
  • Adrianna Foster (NCAR) and Jessie Needham (LBL) have been running ILAMB on ensembles of FATES. This does not as far as I know run with SE grids, but is useful in terms of hitting the available benchmark data.
  • Best approach might be to 1) operationalize the ILAMB approach and 2) test whether it can run with SE output and 3) discuss SE diagnostics with NCAR and 4) create nimble skeleton plotting tool for FATES output to assess the realistic workflow requirements.

Jan 19th 2024

  • NorESM AMIP run with SE NorESM
  • Same but with FATES
  • Diagnostics for SE. CAM diagnostic package with xESMF by Jessie (but it's slow).
    • Use Keith NCL scripts
    • Output at 1 degree at run-time
      • Loose relationships within a gridcell for land variables
    • Run on different grids for the atmosphere and land.
      • More runtime
      • Changing coupling variables when regriding. ATMxLND will not be as straightforward as in 1to1
    • Make a package out of CAM package for CLM.

Jan 12th 2024.

As soon as two stream tag is in at ESCOMP.

  • Mariana will do 2deg 10-20-year FateNoComp and BgcCrop degree runs with NorESM2.5-dev - CAM on the finite volume grid on the piControl.
  • First, offline spinup with CPL history from CMIP6. 2deg FV resolution. Use DATM forcing from CMIP6 and run 100 years forcing CLM/FATES nocomp with the DATM forcing. Will see how well the plants are evolving. Need to ask Dirk for location of forcing files for piControl.
  • Rosie will provide the parameter file tag. Make a tag on NorESM-tage for checking out the new parameter file.
  • Make a CTSM tag that corresponding to the two-stream tag at ESCOMP.
  • Make another tag of the one above which points to fates tag Rosie will provide.
  • Make a NorESM tag to point to CTSM tag above.

Jan 4th 2024

Issues to make FATES operational for NorESM2.5

  • Init_interp issues. Will NorESM spin-up different resolutions independantly?
  • Ways to visualize spectral element (ask Keith).
  • CMORization for FATES variables.

FATES gantt chart

Project board for NorESM-FATES lives on the NorESM repo here

Nov 24th 2023.

CLM-FATES is three times slower than CLM51 after 35 years. Mariana to make the FBG true for the timing test. This should improve timing. We discusses the sunderlying setup for NOCOMP mode in light of the potential to improve the speed with fewer cohorts etc. link to charlie's slide deck here image

Nov 17th 2023

  1. Issue in call t_start() in CNDriver
  2. Compset for use_fates_nocomp
  3. Output for performance testing:
    • Runs with standard output + PFS tests
    • Fates hist PR
    • Output for spinup: FATES_VEGC, LAI +???
  4. CTSM5.2 project CTSM5.2 branch
  5. Timeline for CTSM6.0 and CESM3:
    • Preliminary: Winter 2024: includes CTSM5.2, Matrix CN, Dust emissions, Tillage ...
    • Code freeze: Summer 2024
    • Tunning: Winter - pre-tunning, Summer: actual tunning.
  6. ILAMB and diagnostics, PPE diagnostics. NorESM tunning: which diagnostic package to use?
  7. Landuse and preproc tools for mksruf_*

Nov 10th 2023

  1. Performance Test for FATES. Use the current out of the box. Need to define the compset, tag resolution etc. Will wait for the restart bug to come into CTSM. Issue here
  2. INES2 - need to clearly state what will happen without more funding.
  3. Need to decide whether FATES tags live inside the NorESM repo or the NGEET repo.

Nov 3rd 2023.

  1. Issue with input data disk space. Do we want to be testing etc on Fram as well as Betsy? Fram mostly used for offline land configurations. Do we need coupled model to be tested there? Mat and Mariana to discuss Betsy vs Fram usage. UiO people mostly using Betsy with release tags. Other developers using Fram. NorESM-3alpha is current development branch.

  2. Discussion with Betty on MIMICS+

  3. Next steps with FATES once restart bug is fixed.

  • Wait until the radiation scheme and the bugfix for FATES restart are available to do performance tests. Do these with NOCOMP at nominal 1 degree resolution (also FV 2 degree). FATES performance should spin up in 20 years - also verify this.

  • Spin-up script sorted for FATES. Issue added to project board

  • Need for N cycle to be linked from FATES to CTSM. Plan to do the CTSM side changes on the NorESM branch, .

Oct 27th 2023

  1. Discussion of Fram CESM compilation issues. Discussion here
  2. Restart bug for FATES.
  3. NES2 proposal

Sept 29nd 2023

Discussion of FRAM:

  1. Discussion with Greg and Ryan on FATES NOCOMP bug will happen on friday at 5pm
  2. INES2 land model work package in preparation. Rosie & Heiko leading WP2. Please get in touch if you want to discuss anything.
  3. Adele Zaini will start PhD on Oct 16th working on BVOCs in NorESM/CTSM/FATES
  4. Interesting discussion of modularity on ILMF forum. Model parameterization forum on Oct 10th.
  5. New progress on calibration of tree allometry and respiration in FATES (Jessie Needham and Adrianna Foster). Much healthier boreal forest! (was a big bias in last weeks calibration).
  6. Need to decide on he CTSM version for NorESM2.1. The current tag is v5.0.14

Sept 22nd 2023

  1. We have created this board to track issues with FATES that need to be completed for the sake of the coupled NorESM FATES_NOCOMP

  2. @rosiealice gave a talk on the latest calibration efforts which are looking good. here

  3. We will have a meeting about the NOCOMP restart bug next week. Greg will organise the time.

  4. New updated issue on fire emissions logic: https://github.com/NGEET/fates/issues/1088

  5. We had a CLM meeting on theESCOMP CTSM-FATES tasks board , which are heavily parallel with our board. The notes should be in the minutes (but are not yet..

  6. INES2 planning underway. WP2 (land) and WP4(coupling) most relevant here.

Actions:

  • Rosie to send Mariana restart dataset . Mariana will run a restart test.
  • Add issue to the board to cover nutrient calibration
Clone this wiki locally