PE_Obs and SR_Obs for nmonth table

[jdduh]

Per George's request to add the two parameter to the PRMS parameter table,

Thanks for sending the files. This message is to summarize the steps for calculating the two new nmonth variables: PE_Obs and SR_Obs. I also have my questions highlighted.

1. PE_Obs (Observed Potential Evapotranspiration) is defined as the AOI mean daily PET (inch/day). The PET source data are from the National Weather Service (please provide a download link). These rasters have a spatial resolution of 16.5 by 16.5 Km. The unit of the NWS PET values is mm/month. The output unit for PE_Obs is inch/day. (If the inch/day unit is correct, then the unit conversion routine in reformat-PE.f contains a bug. It uses a wrong number to convert mm to inch.) The theoretic minimum value for PE_Obs is 0.0001 inch/day. Set PE_Obs to 0.0001 if the observed value is smaller than the theoretic min.

We got the maps from a colleague at the National Weather Service. In a paper on the use of the data it was described as

"The basin mean
monthly PET values were calculated from PET maps
provided by the NWS. The NWS derived the PET values
from the free water evaporation atlas of
Farnsworth et al. (1982). These PET maps were used
for calibration and evaluation purposes in this study
because these values are available for the entire U.S.
and the ability to apply this calibration procedure to
numerous basins across the U.S. depends on having a
data source for PET."

The output is inch/day and you are correct, the conversion should be 25.4

1. SR_Obs (Observed Solar Radiation) is defined as the averaged daily solar radiation, expressed as kWh/(m^2 day), for each month of the year. The SR source data are from (please provide a link to the source). The data file (solarDAT.oct2004) contains monthly solar radiation values for 3059 stations. The station number, x, y, coordinates (in Albers Projection coordinates), and station elevation (in meters) are also recorded in the data file. The monthly solar radiation values of the station that is the closest to the centroid of the AOI are extracted from the data file as the SR_Obs values.

The radiation unit is langleys/day or calories/sq cm/day for each month of the year.

In the same paper with the PE description SR is described as:

"Measured monthly SR values are available for 217
stations in the U.S. (http://rredc.nrel.gov/solar/old_
data/nsrdb/redbook/mon2/). A data set of mean
monthly SR values at each of the climate station sites
(SNOTEL and NWS) in the U.S. was developed. A
multiple linear regression (MLR) was developed
between measured monthly SR values at the 217 stations
and independent variables (climate statistics
calculated using climate stations co-located with the
measured solar radiation). For each month a separate
MLR equation was developed, choosing from the following
independent variables: latitude, longitude, elevation,
mean precipitation (days > 0˚C), mean
precipitation, number of rain days, mean maximum
temperature, and the difference between mean maximum
and mean minimum temperature. Adjusted R2
values ranged from 0.83-0.98. Mean monthly SR
values were calculated at each SNOTEL and NWS climate
station site using the monthly MLR equations.
As noted earlier in this paper, there is a lack of
additional data sources for model calibration and
evaluation in nonresearch oriented basins. Mean
monthly values of SR are used for calibration and
evaluation purposes in this study because these values
are available for the entire U.S. and the ability to
apply this calibration procedure to numerous basins
across the U.S. depends on having a data source for
SR in each of these basins.
SR output from the station closest to the centroid of
the Yampa River basin (23 km) was used as the calibration
data set for the first step in the calibration
process. These SR values are referred to as interpolated
values in the text."

[jdduh]

The calculated values are stored in settings.xml (same as jh_coef). There are 12 values returned for each parameter. Have a separate tool (form) for calculating PE_Obs and SR_Obs. Put it under the Timberline tool, name it "PE_Obs and SR_Obs Tool". On the export form, add the dates of each parameter that was last calculated. The parameters should be present at the settings.xml file and the parameter template doesn't have to have these two fields in the input template. The export tool will determine if it's an insert of update during export.

[jdduh]

SR source data is a point featureclass (in a gdb or a shapefile). The PE source data has 12 rasters in ArcInfo GRID format or as rasters in gdb. The raster files must follow a certain naming convention. Users need to select the folder containing the January raster for the execution. The tool will parse the prefix and suffix of the other rasters' names.

[jdduh]

The export tool still allow export even if these two parameters are not present in the settings.xml file

[jdduh]

The PE rasters and solar radiation station point featureclass are available at D:\NRCS\GIS\Static\nmonth_PE_SR on CH459C03. There are two sets of PE rasters: one in the nmonth_databin,gdb and the other is a set of ArcInfo GRID in the nmonth_PE_SR folder (workspace).

[lbross]

Some questions on the tool:

1. Should we indicate if the parameters were previously calculated for the selected AOI? Date calculated? Show the values on the form?
2. Do we validate the solar radiation points layer? Should user choose column name for January similar to PE data? Or do we assume column names are SR01 .. SR12 and validate these columns exist?
3. The PE layers in the gdb use a different GCS from BAGIS-P: GCS_Clarke_1866. Is it possible to reproject before distributing the data to NWCC? Or does the tool need to do this?
4. Do we need any validation for the PE layers outside of making sure all 12 months are present?
5. See sample form below. Suggestions welcome:
   

[jdduh]

1. Yes and display date calculated.
2. Yes. Users must prepare the data that conforms to the attribute field naming convention (SR01, etc). Please add a button "Tell me more" on the form and describe the PE and SR calculation and data specs.
3. I will create a new dataset for PE and SR that has the same projection/datum as NWCC project.
4. Such as?
5. Move the note to the "tell me more" description. In the tell me more description, please also mention that the calculated values will be saved to the nmonth table of the output PRMS parameter during parameter export.

[lbross]

1. OK
2. We will assume the column names are SR01 .. SR12 and throw an error if any are missing. Yes on "Tell me more" button. We should also validate that this layer is in same projection as BAGIS-P data and throw an error if it isn't.
3. Again, will validate the projection of these raster
4. Sounds like no
5. OK
6. Any suggestions on how to find the closest point to the AOI centroid? I've located the centroid as a Point using ArcObjects. Spatial query?

[jdduh]

See http://forums.esri.com/Thread.asp?c=93&f=992&t=186118. I'm not sure if that's an easy approach. You can also try the near geoprocessor. The input needs to be a point featureclass.

[lbross]

See updated versions of PE and SR Obs form. Also "Tell Me More" output. Any suggestions/edits?

I am recording the SR values with no transformation, is this correct? It appears that the PE values do require a transformation from mm/month to in/day. So we need to:

1. Multiply the value by 0.0393701 to go from mm to in
2. Divide that result by the number of days in the month. Does this have to be exact for each month? Or can we just use 30 days?
   Is this transformation correct?

[lbross]

Seeking guidance on how to handle AOI-level parameters for nmonths table in export file. Current AOI-level parameters are jh_coef, pe_obs, and sr_obs. If there is an error in calculation for any month, I populate the parameter/month value in memory with -9999.

Should all 3 of these parameters ALWAYS be included in the nmonths table? If the column exists in the input template, its contents will be overwritten with the output of the calculation. If the column doesn't exists in the input template, the column will be added to the export file.

Is the answer different if there is an error in the calculation? That is, if any of the values are -9999. I have some error handling in the export form now that warns of the jh_coeff calculation is invalid. That will need to be adjusted to accommodate additional parameters.

[jdduh]

The message in the Tell me more dialog looks good, except the extra "to" in the first sentence. The conversion from mm to inch is correct. Alternatively, you can just divide the mm value by 25.4. We do need to differentiate 28, 30, and 31 days in the different months.

The jh_coef must always be in the output table. Can you make pe_obs and sr_obs optional (but include them as the default)? The export behavior you described is correct.

Ok, write -9999 to the cache and warn the user during export.

[lbross]

How do we make pe_obs and sr_obs optional? Do you mean that we only include them if they have been calculated for the AOI? Or should we add a checkbox to the export form that allows the user to turn this on and off? Checked by default?

[jdduh]

How about add the following checkbox to the export tool (see #26)?

"Export observed PE and SR to the nmonth table."

[lbross]

See issue #26 for screenshot of updated parameter export tool. We originally said we would put the date last calculated for these parameters, but this screen is getting busy. Added a comment referring users to the PE_Obs_SR_Obs tool to manage these parameters, With this design, it is all or nothing. They either export both or none. Is this okay? Also plan some validation on this checkbox to pop an error message if they try to include the parameters and they are missing.

[jdduh]

The current GUI design and checkbox options (see #26 screenshot) are good.

[lbross]

The datasets provided for PE has a different spatial reference than BAGIS aoi_v. The PE data uses Clarke_1866_Albers. aoi_v uses GCS_North_American_1983/NAD_1983_Albers. Should the PE data need to be reprojected to match grid_v?

[lbross]

I am using the number of days from each month in 2015. ie: not a leap year. This is configurable in the code. Could be current year or ... Let me know if this should change.

[jdduh]

A new nmonth_Databin.gdb is copied to D:\NRCS\GIS\Static\nmonth_PE_SR on CH459C03. The layers that are in the "USA_Contiguous_Albers_Equal_Area_Conic_USGS_version" projection have a suffix of USGS_Albers.

[lbross]

VB .NET has built-in functionality to determine the number of days in a month. https://msdn.microsoft.com/en-us/library/system.datetime.daysinmonth(v=vs.100).aspx. The only variation should be for leap years and the original fortran did not use leap years.

This tool is done. I just ran it against the reprojected layers and it worked fine.

[jdduh]

Ran the tool on the Santa Fe AOI and got a zonalstats error (12 errors total) on PE calculation. The AOI is posted at ftp://basins.geog.pdx.edu/BAGIS/BAGIS_aois/Santa_Fe_R_nr_Santa_Fe.zip. Please give it a test. This is one of the AOIs that you need to use SR and PE layers that have different projections.

[lbross]

I am stumped. The only difference I can find between this AOI and Teton (which works) is the projection of aoi_v: NAD_1983_Albers vs Projected coordinate system: USA_Contiguous_Albers_Equal_Area_Conic_USGS_version. However, I have tried re-projecting aoi_v to match the PE layer and vice versa but it still fails. I am testing using the Zonal Statistics as Table tool interactively in ArcMap. This tool does not return an error message, it just fails.

When I try to run the PE_Obs calculation using the add-in, this is the error message returned by the GP when it runs the ZonalStatisticsAsTable tool:
ERROR 999999: Error executing function.
WindowSetLyr : Window box is outside layer boundary
WindowSetLyr : Window box is outside layer boundary
WindowSetLyr : Window box is outside layer boundary
WindowSetLyr : Window box is outside layer boundary
ERROR 010366: All cells in a raster have NODATA value. VAT will not be built. All cells in grid c:\users\lbross\appdata\local\temp\t_t8020 have NODATA value. VAT will not be built.
ERROR 010067: Error in executing grid expression.
ERROR 010152: Object ITable is null.

I've added both layers to the viewer, and aoi_v falls well within the boundaries of the PE01 raster. Actually, the AOI falls completely within a single cell of PE01. Is this a problem? My suggestion would be to provide both layers to ESRI to see if they can get the ZonalStatisticsAsTable tool to run. Or maybe you can?

[jdduh]

I think this is the problem - "AOI falls completely within a single cell of PE01" Maybe the "center" of the cell falls outside the AOI so their intersection returns a null. How about creating a buffer of the AOI if its area is smaller than 2 pixels on the PE layer (around 200 square miles)? Set the buffer distance to 8 km.

[lbross]

It appears that the area of the AOI is the problem. I tried an 8K buffer on this particular AOI and it failed to complete. Dropped it down to 4K and that worked. The area(s) of the AOI's respectively are 47,466 sq km and 242,543 sq km. A cell area is 272,481 sq km. I used the 4K buffered AOI as input to the Zonal Statistics tool and that worked as well.

So, I'm having trouble developing an algorithm that we can add to the code. We can get the linear unit(s) from both layers, the area of the aoi polygon, and the area of a cell on the PE layer. But I'm not sure how to derive a reasonable buffer distance from this information. We don't have any information about the shape of the AOI (width v length). If the distance is too big relative to the AOI area, the buffer son't be created. Thoughts?

[jdduh]

An alternative approach is to use the centroid of the AOI boundary to extract values from the PE layers when the AOI is too small. This approach doesn't require us to determine a buffer distance.

[lbross]

If we did this, would we just return the value of the cell that was located at the centroid of the AOI if the AOI area were < 75% of the cell area? With the PE tool we normally extract the mean value of the cells for each AOI using the zonal statistics tool.

[jdduh]

Yes, use the values directly. This approach is for AOIs with area < 200% of the cell area. I think 75% is not enough to catch most cases that the "clipped" PE layers are empty.

[lbross]

Updated PE calculation to use value of cell at centroid of AOI if AOI area is < 200% of the cell area. This change will be included with the next release of BAGIS-P.

[jdduh]

The output function needs to create a new .csv file (obs_monthly_pe_sr.csv) and package the file with the other output files (i.e., parameter filr, resampled HRU and DEM ASCII files) in the zip file. There is no need to remove the PET and SR from the parameter file's nmonth table. The file format of obs_monthly_pe_sr.csv) is:

@t,obs_monthly
len,12
created_at,03/22/2016
created_by,BAGIS-P
converted_from,BAGIS-P
date_format,MM
aoi,Animas_R_at_Durango_03152016
unit_for_PE,inch/day
unit_for_SR,langleys/day or calories/sq cm/day
@h,date,PET,SR
type,Date,Real,Real
,1,0.02239967,263.75
,2,0.043641029,304.8
,3,0.078041655,398.8
,4,0.12121063,513.72
,5,0.159559943,669.6
,6,0.182939631,752.47
,7,0.184928249,591.73
,8,0.164639956,536.65
,9,0.128133202,490.11
,10,0.08499492,412.42
,11,0.046735565,285.44
,12,0.023749048,255.66

[lbross]

Can you either attach a copy of a valid obs_monthly_pe_sr.csv to this issue or e-mail it to me? This would be easier to work with than just the text. Thanks!

[lbross]

The export parameters form has a checkbox "Export observed Potential Evaporation and Solar Radiation to the nmonths table". Should we update this to say "Include observed Potential Evaporation and Solar Radiation in export file" ?

I assume we include this file if the user checks the "Output parameter file only" checkbox?

Also, I haven't looked at the code yet but because this is a toggle, chances are it would not be difficult to stop adding these parameters to the nmonths table which might avoid confusion in the future.

[lbross]

Note: I am changing the parameter names to match what is in the provided .csv. If you have older AOI's with PE and SR calculated, you will have to recalculate.

[lbross]

@jdduh This is almost done. Due to the form design, I decided to write the obs_monthly_pe_sr.csv directly to the zip folder. Thus, this file is NOT created when the "Output parameter file only" checkbox is checked. The export form has a field for specifying the output parameter .csv file name. Since the pe_sr file name is always the same, I was was worried that if there were > 1 aoi parameter files in a folder and we wrote the file alongside the parameter.csv file the user wouldn't know which aoi parameter file the obs_monthly_pe_sr.csv was associated with.

Waiting for you to answer my previous questions about the form label and if it is okay to stop adding these parameters to the nmonths table before posting a new version of BAGIS-P.

[jdduh]

Please keep the PET and SR in the nmonth table. It's fine to update their parameter names. These two parameters are not directly used in PRMS/eWSF for modeling purpose. They are used to calibrate model outputs in a different eWSF menu/dialog window.

Please update form label as you suggested.

One AOI has one unique set of PET and SR values. The values are not affected by HRU delineation or parameters calculated. Can you still create the csv when "Output parameter file only" checkbox is checked? You can choose to either overwrite or not creating a new file when the file exists.

[jdduh]

Ok, I changed my mind in 2 seconds. Please don't create the PET & SR file when the "Output parameter file only" checkbox is checked. First, the PET & SR can be found in the nmonth table of the parameter file. Second, the parameter files from different AOIs might be written to the same output folder.

[lbross]

Agreed on not creating PET & SR files alongside the parameter export for the second reason you mentioned. I have posted v1.9.7 of BAGIS-P which includes this new file export.