Development Plan: Merges into develop-specact (1-2/2024)

[rburghol]

Branches

• Current Working Branches:
  • develop: the next step before master on respec repo
  • develop-equ-advanced: will introduce equations,
  • develop-broadcast: Built off develop-equ-advanced: Add broadcast
  • develop-datamatrix: Built off develop-equ-advanced: Add datamatrix
  • develop-flowby: Built off develop-equ-advanced: Add flowby
  • TBD develop-impoundment: Do after equations, matrices, flowby and broadcast variables, which are the base components needed to do a non-reservoir water supply operational model test
• Remnant Branches (may contains code that is needed for migration)
  • develop-specact-eq: child of develop-specact - matrices and equations and lookups is current working version (copied from branch called develop-specact-speed-eq to keep things organized)
  • develop-specact-ts: introduce timeseries loading code and integration into runtime
    • Cannot yet test code until develop-specact has merge from develop-specact-depend (see above)
  • develop-specact-3: contains all currently developed OM code, with most recent version (except equation), including impoundment and channel prototypes
• Recently merged branches:
  • develop-specact-speed merged into develop-specact 4/2/2024
  • develop-specact-depend: merged 4/24/2024 - introduce end-point dependencies - PR sent Develop specact depend respec/HSPsquared#150
• See also;
  • Loading Models from json: How to Debug a Model Object Loader #61
  • Overview of data models and function hsp2 planning/review 2024/01/05 #81
  • Developing to python with version specific dependencies: Virtual Environments - Install & Run HSP2 Python on Windows #55
  • VScode dev environment: https://code.visualstudio.com/docs/python/debugging

Migrating Code from Branch to Branch

• git copying via checkout is most efficient
  • Use: git checkout [source branch] [code file]
  • Ex: git checkout origin/develop-specact-eq HSP2/om_model_broadcast.p
• Note: after refactor, hsp2 went from HSP2 to src/hsp2/hsp2
  • need to checkout, then copy into src/hsp2/hsp2 and then delete directory HSP2

Steps

• Merges for OM components
  • Pending/upcoming pull requests:
    • develop-specact-speed: changed to fixed width integer array for tokens and do not iterate th… respec/HSPsquared#134
      • modelObject, modelLinkage, SpecialAction, om lib: just the new time efficient code
      • benchmark.py: performance testing of the speed increase with state_ix as an ndarray instead of Dict (see use: Test Components #85)
      • end-point run hierarchy Single Component Dependency end-point function respec/HSPsquared#133
      • Note: both state_ix and op_tokens have been converted to ndarray, but the performance improvement with op_tokens is only on the order of 10ths of a second, because writes to data structures are very expensive in Dict but reads are cheap, or negligible slowdown compared to ndarray.Enable Continuous Integration Testing respec/HSPsquared#135
      • Provide access to timeseries values in both specl and regular om operations.
      • Test outputting state values to timeseries in json
    • (underway) develop-equ-advanced:
      • equation - basic mathematical operations with parentheses and standard operation hierarchy support.
      • test equation by mimicking the specl test code writes
      • timeseries: allow reads and writes from the hdf5 timeseries
    • develop-ts-read: hsp2 populates ts with only the domain specific inputs, for example hydr only has water balance related inputs for the specific RCHRES included -- see details at Add timeseries/other DSN access to STATE/specl #60
      - [x] Accessor code
      - [ ] Convert to common timestep
      - [ ] Write to timeseries
      - [ ] Demo
      - [ ] Continuous integration testing code
    • develop-datamatrix:
      • datamatrix - lookup tables
    • develop-broadcast:
      • model_broadcast - allows a very flexible way of aggregating and linking variables, mostly a "pre-processor" type object, as it gets tokenized as links and has an auto reset to zero at the beginning of every timestep.
    • develop-flowby : FlowBy Object #82
      • Flowby - maybe done as parser only arrangement which combines primitives equation, datamatrix and datamatrx function (like min, max)
    • develop-specl-uv - introduce timing for repeats on basic ACTION and the 2 UV ops
      • UVNAME or UVQUAN - which ever should logically appear first and can support work in equation or other components
    • (TBD) advanced operational model
      • datamatrix: table data structure to provide lookup lists, as well as statiscal analysis (i.e. get min value of arbitrarily long list of elements)
    • Single Component Dependency end-point function respec/HSPsquared#133
  • Continuous Integration Testing Enable Continuous Integration Testing respec/HSPsquared#135
    • get it set up :) (tricky)
    • automatically runs any time a pull or merge is done
    • Test individual functions
    • test run for given set of must implement UCIs
      • check for catastrophic errors (i.e., the UCI failed to execute)
      • check for numerical accuracy (summarize timeseries outputs)
    • Test runtime/performance (could fail or issue a warning for a new pull if a given benchmark runtime was exceeded)
  • Advanced hydrologic sub-components (testing for small facility impoundments, NHD+ scale modeling, and prototyping improved hydrology methods)
    • flowby
    • simple channel
    • impoundment
• Include performance and bug testing scripts for use before pull requests Test Components #85
• Start tracking a branch created since we forked:
  • View Branches in github UI -> Select "New Branch" -> select upstream branch to fork

• Can we do it inside of git on our local machine? So far, this has not worked, but there may be a kernel of knowledge in here somewhere
  • git remote add --track develop-specact upstream-develop-specact https://github.com/respec/hspsquared

[rburghol]

Testing equation and data matrix:

• Copy directory test 10 SPECL to test 10 EQ, and replicate sediment SPECL with equation
• create a copy of test and make test 10wd With a WTM withdrawal
• Use the WDM withdrawal test as a baseline for testing the data matrix