-
Notifications
You must be signed in to change notification settings - Fork 0
2023 CTSM PPE Meeting Notes
Katie Dagon edited this page Oct 22, 2024
·
1 revision
Announcements
- Schedule for the next few PPE meetings with upcoming holidays/conferences
- AGU sessions on PPEs! GC12D - Oral, GC21M - Poster
Agenda
- Hossein Kaviani (UVA) presents "Quantifying parameter uncertainty in environmental models" --PhD Student at UVA doing an internship with Katie Dagon
Notes
Bayesian inference: incorporate prior information into our predictions
- Can be used for estimating posterior and constraining range of parameter values MCMC: Markov chain Monte Carlo
- Markov chain is an explorer, following a gradient
- Needs a 'burn-in' period (adjustable parameter)
- Many methods use multiple chains
Developed new diagnostic assessment of MCMC algorithms
- MCMC needs to be effective (converge to true posterior)
- efficient (in time)
- reliable (consistent across random seeds)
- controllable (insensitive to hyper-parameters)
Goal: to come up with diagnostics that ensure an algorithm is insensitive to "unimportant" hyper parameters
-
Comparing three existing MCMC methods: -- DREAM, metropolis-hastings (MH) and Adaptive Metropolis (AM)
-
Test problems: 10D bimodal mixed gaussian distribution
-
Hyper parameters of interest: number of chains & number of function evaluations
-
KL divergence tells us how close the estimated posterior is to our true posterior (true posterior is known)
-
Diagnostics developed can a priori tell a researcher what algorithms are appropriate for different problems (due to their insensitivity to choice of hyper parameters).
Research question 2: Integrating multi-objective, Real-Time control into storm water management
- implement BORG algorithm in SWM moswl (storm water management)
- Objectives: minimizing floods (overflows) & minimizing sum of squared errors between simulated flows and pre-development flows
- Many of the target objectives, in practice have conditional constraints, BORG can capture the objective goals with constraints.
Internship at NCAR:
- Coupling Bayesian inference with emulators of CLM
- Tuning 32 CLM parameters
- Develop likelihood function to capture a probabilistic space between observations and model simulations
- Using DREAM algorithm
Announcements
- The Parameter Estimation Interest Group will be having its first meeting next week, Wednesday October 4 at 9am MT. You can join the email list and slack channel to stay up to date on group activities.
Agenda
- Nina Raoult (Univ. Exeter) presents: "Parameter perturbation experiments in land surface modelling"
Notes
- Working with JULES and ORCHIDEE
- Parameter sensitivity, history matching (mouse w/ cheese!), and emergent constraints
- Parameter sensitivity: Morris and Sobol
- Morris to find most sensitive
- Sobol to capture interactions; needs many more simulations which can be computationally challenging
- sensitivity at different timesteps
- Bayesian framework: optimized parameters
- History matching: ruling out unlikely parameters using implausibility function
- Sampling for history matching framework? Latin hypercube for initial sampling and emulators (LHC for dense sampling too)
- Advantage to tune against multiple metrics (vs. traditional Bayesian)
- Can discover parameters with double minimums -- don't change cut off too soon (i.e., how to define implausibility)
- Comparing to gradient based descent: can get stuck (many local mins)
- Generate ensembles from posterior distributions to test the model
- Parameter spread in future projections (e.g., Booth et al. 2012)
- Use emergent constraints to combine parameter constraint (via calibration) with future relationship
Discussion
- Emergent constraint method: using posterior distribution of one parameter (Topt), combine with linear regression from future projection experiments
- Ecological plausibility of optimized parameters: constraining the prior distributions; parameters are making up for compensating errors, how many parameter do you want/need?
- Improving structural biases: when parameter optimization can't get close to obs, likely pointing to a structural process (e.g., missing process)
- History matching vs. behavioral/non-behavioral terminology like GLUE (Beven) in hydrology
- Standard "cutoff" for history matching is 3 based on standard deviations, can think about how to adjust this during process
- Assessing emulator performance during waves
- Emergent constraints: need to understand the physical mechanisms behind them, otherwise can be noise
- Computational constraints of Sobol: approach it as a chain of methods with Morris first, limit the number of parameters; use ensembles, emulators with or without history matching; in reality really only using Sobol on site examples due to computational limitations
- What are the limitations of gradient descent? Not really giving us uncertainty
Announcements
- Introduce NCAR visitor Hossein Kaviani (UVA)
- The ILMF is organizing a series of webinars including one on parameter estimation on October 10, 9-11am MT. Organized by Rosie, Daniel, and Nina Raoult. Linnia will be speaking about the CLM PPE work! Register here.
- Next CLM PPE meeting will be at a different day/time: Wednesday September 27th at 10am MT, Nina Raoult (Univ. Exeter) will be presenting.
Agenda
- Daniel: OAAT manuscript update and SSP extension runs
- Linnia: Updates on LAI calibration
Link to PFT parameter sheet: https://docs.google.com/spreadsheets/d/1ZIM5ZT6DLdWm9s2uLXqd9YUbYME4Qw6p65S92gG6r4o/edit?usp=drive_link
Notes
- Will/Rosie: there is a FUN parameter name switch, we can test out how that impacts simulations with the sparse grid. It is baked into all the PPE simulations.
- Daniel: OAAT manuscript updates
- CTSM/CLM naming convention for publications, a bit open ended at this point. How to best track the literature / be consistent?
- Data available on request
- Hosting the CLM PPE output on
/glade/campaign/cgd/tss(~3 TB), plan to create a DOI/link e.g., through Climate Data Gateway; backup smaller dataset ready for OAAT publication and analysis. - Comments welcome on the OAAT manuscript, email Daniel.
- Daniel: SSP3-7 extensions update
- Runs to 2100 are complete
- Huge spread in land carbon sink
- Can compare to CESM LE, TRENDY, ISIMIP, CMIP
- Rosie: Color-code the lines in the spread plot with dominant mode of variability in carbon loss?
- Will/Charlie: ILAMB scores to weight likelihood of various parameter sets
- Andy: Behavioral vs. non-behavioral outcomes, matching physically realistic metrics
- Sanjiv: What is the largest hammer in the system? Should we have a poll to guess? Likely leafcn, slatop, jmax (continent on the LAI-selected parameters which is a subset of all CLM parameters)
- Linnia: LAI calibration slides
- Lessons learned: compound cost functions are hard to emulate, design emulation for physical relationships, use ensemble of emulators
- Calibrating PFT-specific parameters independently without running a huge number of simulations
- Universal vs. PFT-varying parameters
- Need help defining relationships for these parameters
- Gaussian process emulators for each PFT, PFT-level calibration target is CLM-SP, spatial PFT mean of annual max LAI
- Deciduous emulators aren't as good, overall performance is very good
- Charlie: Deciduous phenology parameters may be partially represented in PPE (e.g., crit_dayl)
- Hierarchy of sampling: 10^6 samples per PFT, univariate sampling - could we do this better?
- Plausibility criteria: universal sets with at least one PFT set within observational tolerance of CLM-SP target, very narrow to start (+/- 1%), 15% of original sample is plausible, then passing sets for PFT (still hundreds), then randomly sample 25 for running CLM
- Within testing, can improve many PFT biases in LAI; broadleaf deciduous temperate tree is struggling due to emulator uncertainty
- Still neglecting interactions between PFT parameters
- GP emulator uncertainty seems robust
- Take-away: need to take plant trait relationships into account
- Guoqiang: they have used an optimization workflow that updates the emulator (iterative)
- Sanjiv: emulating with meteorological information, will work on this for next steps (see Baker et al. 2022)
- Charlie: to tackle within PFT relationships could use something in the cost function, or take into account trait relationships in the initial ensemble design (i.e., don't vary the parameters independently). We can try both.
- Ben: what's the lowest dimensionality to represent forcing data?
- Followup: set up a call to brainstorm trait relationships, see spreadsheet
Announcements
- CESM Workshop and Parameter Estimation cross-working group session was a success!
- Presentation slides are here
- Follow-up parameter estimation interest group in development - look out for an email survey
- PPE session at AGU: abstract deadline is Aug 2
Agenda
- Guoqiang Tang (NCAR) presents "Improving the hydrological performance of CTSM through parameter optimization and large-sample watershed modeling”
- Slides here
NOTES
- Hydrology calibration toolboxes (e.g., OSTRICH) have been implemented for CTSM
- Multiple optimization algorithms can be implemented (e.g., DDS, MO-SMO)
- CAMELS dataset used as calibration target (expanding to CARAVAN).
- Configuration for NCAR HPC system has required infrastructure development.
MO-ASMO example:
- initially need 15-20X parameters simulations,
- trains a surrogate (random forest)
- finds new parameter sets (pareto front, NSGA-II multiobjective optimization)
- Runs the forward model with sample
- repeat until criteria is met (14 iterations used in example)
Announcements
- CESM Workshop and Parameter Estimation cross-working group session
- Analysis of PPEs in Atmospheric Research (APPEAR) Virtual Seminar Series
- Linnia ran a new mini-ensemble - nice work!
- Katie and Daniel received funding from NCAR to work on PPE this summer
- Volunteers/suggestions for June 8 CLM PPE meeting?
- Guoqiang can't make June 8 but would be interested in presenting at a later date
- Ben: interest in CMIP activity regarding PPEs? Email Ben for more info / to get involved
Agenda
- discuss the CLM5 OAAT paper
- figures are coming along really well
- preview the OAAT diagnostics page
- https://webext.cgd.ucar.edu/I2000/PPEn11_OAAT/
- interested in feedback!
- list by parameter?
- links to parameter spreadsheet and CLM variable names
- should follow-up with CISL on cloud deployment
- also some potential for UCSB data science students to contribute here
- discuss next steps for CLM5-PPE
- software and datasets
- tutorial once we have a calibration example?
- storing parameter information in parameter files (e.g., ranges)
- follow-on experiments
- SSP extensions: 2 scenarios?
- DART CAM reanalysis product
- analyzing CLM PPE SP simulations
- software and datasets
Agenda
- Short status update on the various CLM-PPE projects
- one-at-a-time paper 'in prep'
- can share figures, or can wait and share full draft
- will be seeking input on diagnostics suite to put online
- conversation started with CISL re: interactive visualization hosting
- two follow-on experiments in the pipeline
- extending current LHC to 2100
- running default params for 80-member DART ensemble (Raeder et al. 2021)
- proposing a parameter estimation cross-working group session for CESM workshop
- one-at-a-time paper 'in prep'
- Exploring parametric dependence of climate feedbacks using a Perturbed Parameter Ensemble (PPE)
- Saloua Peatier from CERFACS
- Results from an atmospheric PPE, examining equilibrium climate sensitivity (ECS)
NOTES
Exploring the influence of parameters on equilibrium climate sensitivity in the CNRM-CM5-1 model.
Experimental Design:
- Examine parameters related to cloud micropohysics, convection, and cloud radiation (ice cloud microphysics were most influential)
- LHC sampling - 30 parameters - 102 ensemble members
- 2 experiments (control and future) - 3 year simulations
Performance metrics:
- LW,SW, T, P annual mean combined into an error metric
- Used CFMIP as uncertainty range.
- Used EOF analysis to reduce spatial dimensionality and reconstruct RMSE
- Climate feedback parameter
- CNRM-CM6-1 LHC ensemble spread in climate feedback parameter was larger than multimodal ensemble (CFMIP), but ECS was shifted higher.
Emulation & Calibration:
- Multi-linear regression used as emulator to predict error and climate feedback parameter.
- Used emulator and gradient reduction optimization to identify “candidate” parameterizations of the model
- Minimize the Total Error metric, for evenly spaced values of the climate feedback parameter.
- Generated ~15 “candidate” parameterizations
- Ran the atmospheric model with “candidate” parameterizations
Results:
- Most candidate parameterizations were within the error range for error but had a wide spread in the climate feedback parameter lambda.
- None of the parameter sets in the Latin-hypercube ensemble, or the calibrated candidates, were able to beat the default hand tunded model.
- Most “candidates” improved temperature but did not improve precipitation.
- Candidates with high ECS were often way out of top of atmosphere energy balance.
Takeaways:
- Model parameterizations that were within the plausible uncertainty range of observational error simulated a wide range of equilibrium climate sensitivities.
- Model developers are remarkably skilled at hand tuning!
Agenda
- (Re)Introduction to this group and the purpose of these meetings
- Share updates on this project
- Hear from others on PPE-related projects
- LMWG recordings are online
- Land DA Town Hall on Machine Learning for Land Data Assimilation on Feb 22
- Brief simulation/analysis update
- OAAT (one-at-a-time) paper in prep
- LHC (latin hypercube) analysis underway
- Linnia Hawkins presents on LAI calibration approaches for the LHC ensemble. Slides are available here.
Notes
- Establish an ML-methodology for LAI calibration
- Hope to improve fluxes through vegetation structure
- ML-based emulators to interpolate beyond PPE simulations
- Thinking about what metrics to use (e.g., global mean, multiple objectives)
- How to incorporate spatial variability
- GP emulators have training considerations (e.g., covariance structure, hyperparameter tuning), but you also get measure of uncertainty
- Choosing metrics and tools for hyperparameter tuning
- How sensitive is emulator response to hyperparameters?
- Using covariance structure in hyperparameter to sample PFT-dependent parameters
- ILAMB has a new scoring methodology that tries to keep weighting away from the tropics
- Accounting for structural limitations
- Combining different metrics for multi-objective calibration
- Equifinality challenges
- How to define uncertainty in observations?
- Use obs uncertainty to weigh targets
- Stability of MCMC sampler chain is something to look at
- Pareto sampling for multi-objective optimization
- How to define parameter ranges or re-define based on optimization results
- Coordination among traits in plants, can we use that to define covariance
- Biases in atmospheric forcing data
- Use precip and temp as predictors in emulation
- This workflow has a lot of variation and possibility!