Separation of order and domain of HRU/GRU in attribute file from forcing file(s)

[ShervanGharari]

Separation of attribute file and forcing file orders and domain

• Description of feature: currently the order of HRUs in the attribute file and (remapped) forcing file(s) should be the same (to the level of my understanding from SUMMA). This adds the burden to the model-specific preparation workflow to subset, and reorder, both forcing file(s) and attribute files. It would be desirable to have internal sorting mechanisms that assure the simulation for overlapping set of the HRU IDs from the attribute file and forcing file(s). This will allow a subset of a domain in the form of an attribute file to be used with existing prepared forcing file(s) that may cover larger domains. Also allows a rager attribute domain to be simulated for a smaller forcing domain (or domain where forcing is not available from a product for example). The simulation files include only the overlapping HRUs/GRUs from attribute and forcing files.
• Will the proposed feature be backward compatible? Yes
• Will the proposed feature change the science results of SUMMA? No, but add flexibility in setting up future setups.

[andywood]

The main downside is some added runtime from the internal sorting. For very large domains, this may be non-trivial. May also require checking some index-subsetting assumptions, but I'm no longer sure if that's not handled. We faced similar questions in VIC 20 years ago ... either require some assumptions about input ordering or add in some kind of sort/match functionality with a time cost. Users can always do some minimal work to provide inputs to a model's requirements. My model setup workflows/scripts do this by default. Users who care about speed can easily align their inputs. ie, either sort/match every single time you run the model, or do it once during input file preparation. Needless to say, I'd require benchmarking some very large test cases before including this, but it may be worthwhile as a usability enhancement.

…

On Mon, Feb 12, 2024 at 3:42 PM Shervan Gharari ***@***.***> wrote: Separation of attribute file and forcing file orders and domain - Description of feature: currently the order of HRUs in the attribute file and (remapped) forcing file(s) should be the same (to the level of my understanding from SUMMA). This adds the burden to the model-specific preparation workflow to subset, and reorder, both forcing file(s) and attribute files. It would be desirable to have internal sorting mechanisms that assure the simulation for overlapping set of the HRU IDs from the attribute file and forcing file(s). This will allow a subset of a domain in the form of an attribute file to be used with existing prepared forcing file(s) that may cover larger domains. Also allows a rager attribute domain to be simulated for a smaller forcing domain (or domain where forcing is not available from a product for example). The simulation files include only the overlapping HRUs/GRUs from attribute and forcing files. - Will the proposed feature be backward compatible? Yes - Will the proposed feature change the science results of SUMMA? No, but add flexibility in setting up future setups. — Reply to this email directly, view it on GitHub <#558>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIKAROIOQPDE3PDCRNYVZTYTKLGXAVCNFSM6AAAAABDFQE722VHI2DSMVQWIX3LMV43ASLTON2WKOZSGEZTCMJRHEYDMNY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ShervanGharari]

Hi Andy, Thank you for the reply.
Yes, the capability to reorder, subset is there so we wont have issues.
Maybe a bit of background. We are simulating MESH and SUMMA. Both need some massaging of the forcing data. If the model-specific tasks are reduced to a minimum for each model, we would be able to use the same files for both models. That is the best outcome. This needs each model also to be insensitive to the order of input as well. Currently, we have to duplicate files or find a compromising denominator that services both models (without going into details it is very very suboptimal). This might get more complicated if a third model is explored.
That is the basis of the suggestion here, to make sure forcing and attribute file with different orders can still work.
In additional, a case is desirable in whcih, we can subset the domain without really subsetting the forcing and only by submitting the attribute file. subsetting attribute files are much easier while the setup can point at larger prepared forcing files.
From what I know about mizuRoute, sorting is always done when the forcings are read into mizuRoute based on the index in augmented ntopo that mizuRoute internally creates. For checking the computational need, we can maybe look into mizuRoute first(?)
Thank you Andy again.

[andywood]

Hi Shervan, that makes sense. Thinking back, we may have made it an option in VIC4, eg, if your input files were sorted you could skip that step to avoid the cost. Perhaps an efficient sort will take minimal time, e.g., not a bubble sort type routine. My main comment is not that it's a bad idea, but to do a bit of benchmarking of the sort routine on a large domain to make sure you don't introduce much extra run time.

…

On Mon, Feb 12, 2024 at 10:52 PM Shervan Gharari ***@***.***> wrote: Hi Andy, Thank you for the reply. Yes, the capability to reorder, subset is there so we wont have issues. Maybe a bit of background. We are simulating MESH and SUMMA. Both need some massaging of the forcing data. If the model-specific tasks are reduced to a minimum for each model, we would be able to use the same files for both models. That is the best outcome. This needs each model also to be insensitive to the order of input as well. Currently, we have to duplicate files or find a compromising denominator that services both models (without going into details it is very very suboptimal). This might get more complicated if a third model is explored. That is the basis of the suggestion here, to make sure forcing and attribute file with different orders can still work. In additional, a case is desirable in whcih, we can subset the domain without really subsetting the forcing and only by submitting the attribute file. subsetting attribute files are much easier while the setup can point at larger prepared forcing files. From what I know about mizuRoute, sorting is always done when the forcings are read into mizuRoute based on the index in augmented ntopo that mizuRoute internally creates. For checking the computational need, we can maybe look into mizuRoute first(?) Thank you Andy again. — Reply to this email directly, view it on GitHub <#558 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIKARKHKZF2EQZ3TMGNX7LYTL5RRAVCNFSM6AAAAABDFQE722VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBQGQ3DQNZVGI> . You are receiving this because you commented.Message ID: ***@***.***>