9.configure script update #37

singhd789 · 2024-02-21T22:19:27Z

No description provided.

- add yaml validation - add click usage - include all source information for components in regrid rose-app (until rose is retired)

…script-update

- configure script validates yaml and writes more information in rose-apps - edits yaml updated to feel more like previous experiment XMLS - schema.json updated to validate new yaml format

- accidentally added these files from a merge conflict that was fixed previously on branch

…script-update

singhd789 · 2024-02-23T21:28:17Z

Note: the configUpdates folder is not needed and will be taken out. The reason it is still in there in because of something Ian and I were looking at in our refactoring task, but it is solely for my development purposes.

ceblanton · 2024-02-27T20:16:35Z

fre/frepp/configUpdates/pp.yaml

+configuration_paths: 
+  rose-suite: "/home/Dana.Singh/pp/ppyaml/fre-cli/fre/frepp/configUpdates/rose-suite-am5_c96L33_amip.conf" 
+  rose-remap: "/home/Dana.Singh/pp/ppyaml/fre-cli/fre/frepp/configUpdates/remap-rose-app.conf"
+  rose-regrid: "/home/Dana.Singh/pp/ppyaml/fre-cli/fre/frepp/configUpdates/regrid-rose-app.conf"


There 3 files are the rewritten configurations, yes?

I think it would be better to have these paths passed to the fre pp configure tool rather than in the pp.yaml proper.

Partly because these paths are not "first-class" pp configuration and more internal settings, and partly as someone might want to apply the same pp.yaml to multiple workflows using the same pp.yaml:

fre pp configure -y ~/my-stock-pp.yaml -e am5_r1i1p1 -p gfdl.ncrc5-intel -t prod-openmp

fre pp configure -y ~/my-stock-pp.yaml -e am5_test -p gfdl.ncrc5-intel -t prod-openmp

ceblanton · 2024-02-27T20:21:47Z

fre/frepp/configUpdates/pp.yaml

+chunk96: &PP_AMIP_CHUNK96 "1yr"
+analysisStart: &ANA_AMIP_START "1980"
+analysisStop: &ANA_AMIP_STOP "1988"
+defxyInterp: &defaultxyInterp "180,288"


Neat... these are like FRE properties, user variables that can be reused later?

I see the value set there, e.g. 10yr, but is the variable analysisLen or ANA_AMIP_LEN?

Yes! They're used later in the components section. It was slightly confusing, the variable is defined as analysisLen in the yaml, but the value actually reused later, that saves the value the user sets, is the one with the &

ceblanton · 2024-02-27T20:24:23Z

fre/frepp/configUpdates/pp.yaml

+rose-suite:
+  refineDiag:
+    datastager_script: "$(includeDir)/refineDiag/refineDiag_data_stager_globalAve.csh"
+    atmoscmip6_script: "$(includeDir)/refineDiag/refineDiag_atmos_cmip6.csh"


Is "atmoscmip6_script" part of the yaml schema or a user setting?

Oh! This was originally following the draft you had started in the issue (#9). There were 2 refineDiag scripts but both were called script so I found that the latter was the only one found. Changing the names from script just let's us have access to both scripts. atmoscmip6_script would be part of the schema here though. Good catch, I should change it to be more general, like taking away the 6.

Quick clarifying question here, in the rose-suite.conf, the variable REFINEDIAG_SCRIPTS is sometimes defined (if do_refineDiag is True). Would that variable need to point to both of these scripts?

Yes. REFINEDIAG_SCRIPTS can be space-separated, and each filepath should be an absolute path to the script.

ceblanton · 2024-02-27T20:27:32Z

fre/frepp/configUpdates/pp.yaml

+    target: "prod-openmp"
+
+#regrid-xy and remap-pp-components information
+regrid-remap-info:


Yes! The components part down is perfect as is, clearly analogous to a Bronx XML and I think users will be able to understand it.

Can the regrid-remap-info header be removed?

Yes, it can. I put that in there for more organization but I'm sure it would work with just components as the header here.

ceblanton · 2024-02-27T20:28:21Z

fre/frepp/configUpdates/schema-pp.json

+      }
+    },
+    "analysisLen": {"type": "string"},
+    "chunk96": {"type": "string"},


The name chunk96 is part of the schema, here?

Yeah, the chunk96 was part of those user variables that could be reused later. This name could change

User variable names shouldn't be in the pp.yaml schema, should they? Can users name their variables whatever they like?

Well in the pp.yaml, in order for it to be validated correctly, I think it should be defined as something. I can change it to something like pp_amip_chunk or something instead of chunk96. The actual variable that can be reused throughout the pp.yaml, i.e. &PP_AMIP_CHUNK96, can be called anything the user wants though, as long as it's consistent through the yaml.

Sorry, I'm still confused. is the variable called pp_amip_chunk or PP_AMIP_CHUNK96?

You're good! That wasn't a great explanation on my part at all. I was looking for this link to help.

https://docs.geoserver.org/2.21.x/en/user/styling/ysld/reference/variables.html

With reusable yaml variables, it has this structure:
define: &variable <value>
Where:
define = pp_amip_chunk
$variable = &PP_AMIP_CHUNK96
(PP_AMIP_CHUNK96was what the variable was called in the XML, and the variable to be reused in the yaml later on in the components section)
<value> = "1yr"

To my understanding, I see the define (whatever we call it) to mainly be used to validate with the schema

aha, thanks for that. I see. I agree that's just what we want.

Can't we have just one variable name, PP_AMIP_CHUNK96 that is user set? What does the lower case pp_amip_chunk in?

This is the stock example:

define: &variable <value>

Here's our example:

define: %PP_AMIP_CHUNK96 '1yr'

And then PP_AMIP_CHUNK96 can be used in the yaml below as:

*PP_AMIP_CHUNK96

ceblanton

Dana, thank you so much for this. I can't follow all of it, and I think some of the pp.yaml could be simplified, but if mostly works now, I'm inclined to think we should merge it and begin to use and test it more. We can always adjust as we go.

singhd789 · 2024-02-27T21:19:10Z

Thanks for taking a look through this Chris!

I attempted to test this before, to see if the configurations were actually good to run a workflow, but there were some error messages in the rose macro --validate step. I know this has it's own issues, but it brought to light that I may have forgot some variables or potentially set some incorrectly, like outputGridType or inputRealm in rose-apps. I'm in the process of looking through these.

…nd `edits-template.yaml`

Dana Singh and others added 12 commits January 3, 2024 09:36

#9 Add current config.py and pp.yaml updates

6058393

#9 Update pp.yaml

4f1f86f

#9 Update config2.py to write more of regrid rose-app config

0bbc589

#9 Update pp.yaml

071d6d0

#9 Update config2.py

f056f2f

- add yaml validation - add click usage - include all source information for components in regrid rose-app (until rose is retired)

#9 Add json schema for yaml validation

60b45ce

Merge branch 'main' of github.com:NOAA-GFDL/fre-cli into 9.configure-…

b038018

…script-update

#9 Pull in main branch structure changes

9d80272

Merge branch 'main' of github.com:NOAA-GFDL/fre-cli into 9.configure-…

9ab4da4

…script-update

#9 Update configureYAML script, edits yaml, and schema to validate yaml

477426a

- configure script validates yaml and writes more information in rose-apps - edits yaml updated to feel more like previous experiment XMLS - schema.json updated to validate new yaml format

#9 Add example pp.yaml

d6dde5d

#9 Remove unnecessary files

a7e2923

- accidentally added these files from a merge conflict that was fixed previously on branch

singhd789 requested a review from bcc2761 February 21, 2024 22:19

Dana Singh added 2 commits February 22, 2024 17:50

Merge branch 'main' of github.com:NOAA-GFDL/fre-cli into 9.configure-…

b1e5fe8

…script-update

#9 Add filepath checks and update pp.yaml example

728f2a6

singhd789 marked this pull request as ready for review February 23, 2024 21:17

singhd789 requested a review from ceblanton February 23, 2024 21:27

singhd789 marked this pull request as draft February 27, 2024 14:56

ceblanton reviewed Feb 27, 2024

View reviewed changes

ceblanton approved these changes Feb 27, 2024

View reviewed changes

#9 Update configureScriptYaml, pp.yaml example, schema.json, a…

01cd7a6

…nd `edits-template.yaml`

singhd789 marked this pull request as ready for review March 5, 2024 20:42

singhd789 merged commit 0cdd5f7 into main Mar 5, 2024
1 check passed

ilaflott deleted the 9.configure-script-update branch June 18, 2024 17:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

9.configure script update #37

9.configure script update #37

singhd789 commented Feb 21, 2024

singhd789 commented Feb 23, 2024

ceblanton Feb 27, 2024

ceblanton Feb 27, 2024

singhd789 Feb 27, 2024

ceblanton Feb 27, 2024

singhd789 Feb 27, 2024 •

edited

Loading

singhd789 Mar 5, 2024

ceblanton Mar 5, 2024

ceblanton Feb 27, 2024

singhd789 Feb 27, 2024

ceblanton Feb 27, 2024

singhd789 Feb 27, 2024

ceblanton Feb 29, 2024

singhd789 Feb 29, 2024 •

edited

Loading

ceblanton Feb 29, 2024

singhd789 Feb 29, 2024 •

edited

Loading

ceblanton Feb 29, 2024

ceblanton left a comment

singhd789 commented Feb 27, 2024

9.configure script update #37

9.configure script update #37

Conversation

singhd789 commented Feb 21, 2024

singhd789 commented Feb 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

singhd789 Feb 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

singhd789 Feb 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

singhd789 Feb 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ceblanton left a comment

Choose a reason for hiding this comment

singhd789 commented Feb 27, 2024

singhd789 Feb 27, 2024 •

edited

Loading

singhd789 Feb 29, 2024 •

edited

Loading

singhd789 Feb 29, 2024 •

edited

Loading