Merge pull request #163 from Dewberry/release/0.5.x

Release/0.5.0

docs/reference.rst → docs/api_reference.rst

@@ -4,7 +4,14 @@ API Reference / Postman Collection
 
 For reference and documentation of the API, please open the postman collection for the version of ripple1d 
 
-`v0.4.1: <https://github.com/Dewberry/ripple1d/blob/v0.4.1/ripple1d/api/postman_collection.json> `_ This beta version contains the endpoints included in the first production testing release. Note that the following variables should be set in the postman environment
+
+`v0.5.0: <https://github.com/Dewberry/ripple1d/blob/3c90acc3fa212fde9c9b361dc3b907beaca17919/ripple1d/api/postman_collection.json>`_ This beta version contains new endpoints: 
+  - `geom_to_gpkg`: Extract the data from a model source dirctory to a gepoackage.
+  - `conflate`: Conflate all reaches from the NWM network corresponding to the source model.
+  - `conflation_metrics`: Apply conflation metrics for a conflated source model.
+
+
+`v0.4.1-v0.4.2: <https://github.com/Dewberry/ripple1d/blob/666190451620e033e8783241c020d2cde21660c9/ripple1d/api/postman_collection.json>`_ This beta version contains the endpoints included in the first production testing release. Note that the following variables should be set in the postman environment
 
 
 .. code-block:: YAML

docs/index.rst

@@ -15,7 +15,8 @@ on the NOAA National Water Model network.
    :caption: For Users
 
    for_users.rst
-   reference.rst
+   api_reference.rst
+   tech_reference.rst
    users_change_log.rst
 
 

docs/users_change_log.rst

@@ -3,10 +3,72 @@ Change Log for Users
 
 Go to the `Releases <https://github.com/Dewberry/ripple1d/releases.html>`_  page for a list of all releases.
 
-Bugfix Release 0.4.1
+
+Feature Release 0.5.0
 ~~~~~~~~~~~~~~~~~~~~~
 Users Changelog
 ----------------
+
+This release of `ripple1d` incorporates geometry extraction, conflation, and conflation metrics into the API, and fixes several bugs.
+
+
+Features Added
+----------------
+**Conflation improvements**
+
+- The source HEC-RAS river centerline is now clipped to the most upstream and most downstream cross sections prior to starting conflation. This helps prevent identifying network reaches that are far away from the cross sections and improves the accuracy of the conflation.  
+- Overlapped reaches are now tracked and documented in the conflation json file.
+- A bbox column has been added to the network parquet file for faster reading. This was especially needed for the new conflation endpoint since each request needs to load the parquet file. Load times without the bbox column were between 5-20 seconds; this is reduced to 1-2 seconds with the bbox column. 
+- The conflation function now reads locally instead of from s3.
+- The conflation function no longer creates a STAC item.
+- RAS metadata is now added to the conflation json. 
+- The source network's metadata is now added to the conflation json.
+- Length and flow units are now documented in the conflation json file.
+
+**Conflation Metrics**
+
+Three metrics are computed to asses the qualitiy of the conflation:
+
+- `Coverage`: The the start and end location of the reach coverage is computed as a ratio of the length of the network reach.
+- `Lengths`: The lengths between the most upstream cross section and most downstream cross section along the network reach and source HEC-RAS Model's centerline is computed. The ratio of the two lengths is also provided.
+- `XS`: The distance between where the network reach and HEC-RAS Model's centerline intersects the cross sections is computed. A similar comparison is performed using the cross section's thalweg location and the network reaches intersection location with the cross sections. The mean, min, max, std, and quartiles are provided as a summary for both comparisons.  
+
+
+**Geometry Extraction improvements**
+
+- A new function to verify .prj file is a HEC-RAS file has been added.
+- The extracted geopackage now contians a non-spatial metadata table for the souce HEC-RAS model. 
+- Tests have been added for extracting geopackage from HEC-RAS model.
+- Additional attributes are added to the source model gpkg for downstream use. 
+- Units are extracted from the source RAS model and added to metadata.
+
+**API**
+
+- An endpoint was added for extracting geometry and relevant metdata for the soure HEC-RAS models and storing it in in a geopackage. 
+- An endpoint to compute conflation metric was added.
+- An endpoint for conflation (which includes metrics calculations) was added.
+- Tests were added for the conflation, conflation metrics, and geopackage endpoints.
+
+
+Bug Fixes
+----------
+
+- Reaches whose conflation results indicate upstream and downstream cross sections are the same are now considered a failed conflation. 
+- The function to create a concave hull for the cross sections has been improved when junctions are present. 
+- Eclipsed reaches are now better identified and are documented in the conflation json with a boolean. 
+- A check is now performed to ensure cross sections intersect the source HEC-RAS model's river centerline. If cross sections do not intersect the centerline they are dropped. 
+- A conflation json is no longer written for source HEC-RAS models that fail to conflate. 
+- Handling has been added to subset gpkg endpoint for river stationings of interpolated. These river stations contain an "*" to indicate interpolated cross section.
+- Several issues with the automated API tests were identified and fixed. 
+- API tests no longer re-run gpkg_from_ras and conflate_model for every reach; just once per source test model. 
+- When API tests pass the resulting files are now removed automatically. Resulting files for tests that fail are not removed so that the tester can better trouble shoot.
+
+
+
+Bugfix Release 0.4.1-0.4.2
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Users Changelog
+----------------
 This release of `ripple1d` fixes several bugs identified during testing.
 
 Features Added
@@ -21,9 +83,7 @@ Bug Fixes
 - A bug due to a hard coded terrain path name causing an error on  `create_fim_lib` has been resolved.
 - A bug associated with the `ripple_version` parameter has been resolved by removing the parameter from the body of requests (see note in API above).
 - An issue with including lateral structures (not yet implemented) in the ras geometry files causing hang ups  has been resolved. This fix resolved another issue where stationing was mis-applied in the newly created ras geometry files.
-
-
------------
+- A bug which caused a failure when calling subset_gpkg in cases where the model geometries are simple (no structures / no junctions).
 
 
 Feature Release 0.4.0
@@ -59,11 +119,13 @@ Features Added
 
 
 **API**
- - `ripple_version` is no longer a required argument for any endpoint.
+
+- `ripple_version` is no longer a required argument for any endpoint.
 
 
 Bug Fixes
 ----------
 Numerous small bug fixes were made to enable the support of hydraulic structures. Other notable bugs include:
- - HEC-RAS stations with length > 8 characters are now supported.
- - Mangled profile names resulting from negative elevations producing FIM libraries has been fixed.
+
+- HEC-RAS stations with length > 8 characters are now supported.
+- Mangled profile names resulting from negative elevations producing FIM libraries has been fixed.

production/step_1_extract_geometry.py

@@ -9,7 +9,7 @@
 from ripple1d.errors import (
     NotAPrjFile,
 )
-from ripple1d.ras_to_gpkg import geom_to_gpkg_s3
+from ripple1d.ras_to_gpkg import gpkg_from_ras_s3
 from ripple1d.ripple1d_logger import configure_logging
 from ripple1d.utils.s3_utils import init_s3_resources, list_keys
 
@@ -30,7 +30,7 @@ def process_one_geom(
 
     # read the geometry and write the geopackage
     if bucket:
-        geom_to_gpkg_s3(key, crs, gpkg_path, bucket)
+        gpkg_from_ras_s3(key, crs, gpkg_path, bucket)
     return f"s3://{bucket}/{gpkg_path}"
 
 

production/step_3_5_metrics.py

@@ -0,0 +1,10 @@
+from ripple1d.ops.metrics import compute_conflation_metrics
+from ripple1d.ripple1d_logger import configure_logging
+
+if __name__ == "__main__":
+    TEST_DIR = os.path.dirname(__file__).replace("production", "tests")
+
+    src_gpkg_path = os.path.join(TEST_DIR, "ras-data\\Baxter\\Baxter.gpkg")
+    conflation_json = os.path.join(TEST_DIR, "ras-data\\Baxter\\Baxter.conflation.json")
+    nwm_pq_path = os.path.join(TEST_DIR, "nwm-data\\flows.parquet")
+    conflation_parameters = compute_conflation_metrics(src_gpkg_path, nwm_pq_path, conflation_json)

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "ripple1d"
-version = "0.4.2"
+version = "0.5.0"
 description = "HEC-RAS model automation"
 readme = "README.md"
 maintainers = [

ripple1d/api/app.py

@@ -11,20 +11,41 @@
 from ripple1d.api import tasks
 from ripple1d.api.utils import get_unexpected_and_missing_args
 from ripple1d.ops.fim_lib import create_fim_lib, fim_lib_stac, nwm_reach_model_stac
+from ripple1d.ops.metrics import compute_conflation_metrics
+from ripple1d.ops.ras_conflate import conflate_model
 from ripple1d.ops.ras_run import (
     create_model_run_normal_depth,
     run_incremental_normal_depth,
     run_known_wse,
 )
 from ripple1d.ops.ras_terrain import create_ras_terrain
 from ripple1d.ops.subset_gpkg import extract_submodel
+from ripple1d.ras_to_gpkg import gpkg_from_ras
 
 app = Flask(__name__)
 
 
+@app.route("/processes/conflate_model/execution", methods=["POST"])
+def process__conflate_model():
+    """Enqueue a task to conflate a source model."""
+    return enqueue_async_task(conflate_model)
+
+
+@app.route("/processes/compute_conflation_metrics/execution", methods=["POST"])
+def process__compute_conflation_metrics():
+    """Enqueue a task to compute conflation metrics."""
+    return enqueue_async_task(compute_conflation_metrics)
+
+
+@app.route("/processes/gpkg_from_ras/execution", methods=["POST"])
+def process__gpkg_from_ras():
+    """Enqueue a task to create a new GeoPackage from a source model."""
+    return enqueue_async_task(gpkg_from_ras)
+
+
 @app.route("/processes/extract_submodel/execution", methods=["POST"])
 def process__extract_submodel():
-    """Enqueue a task to create a new GeoPackage."""
+    """Enqueue a task to create a new GeoPackage for a NWM reach."""
     return enqueue_async_task(extract_submodel)
 
 

ripple1d/api/postman_collection.json

@@ -1,10 +1,10 @@
 {
 	"info": {
-		"_postman_id": "9c07b074-5f35-4309-bbaf-3ed3f1aec045",
+		"_postman_id": "a4de6034-22c7-4a2b-b91c-12494de19040",
 		"name": "ripple1d",
 		"description": "Collection for processing existing HEC-RAS models for use in the production of Flood Inundation Maps (FIMs) and rating curves for use in near-real time flood forecasting on the NOAA National Water Model",
 		"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json",
-		"_exporter_id": "11115435"
+		"_exporter_id": "29128857"
 	},
 	"item": [
 		{
@@ -68,6 +68,64 @@
 					},
 					"response": []
 				},
+				{
+					"name": "compute_conflation_metrics",
+					"request": {
+						"method": "POST",
+						"header": [],
+						"body": {
+							"mode": "raw",
+							"raw": "{\r\n    \"source_model_directory\": \"{{source_model_directory}}\",\r\n    \"source_network\": {\"file_name\":\"{{nwm_data_directory}}\\\\flows.parquet\",\r\n    \"version\":\"2.1\", // optional\r\n    \"file_hash\": \"a6786ba02073f0233b9a69afc49d92bf-178\", // optional\r\n    \"type\":\"nwm_hydrofabric\"}  \r\n}",
+							"options": {
+								"raw": {
+									"language": "json"
+								}
+							}
+						},
+						"url": {
+							"raw": "{{url}}/processes/compute_conflation_metrics/execution",
+							"host": [
+								"{{url}}"
+							],
+							"path": [
+								"processes",
+								"compute_conflation_metrics",
+								"execution"
+							]
+						},
+						"description": "Compute conflation metrics for a given gpkg/conflation json."
+					},
+					"response": []
+				},
+				{
+					"name": "conflate_model",
+					"request": {
+						"method": "POST",
+						"header": [],
+						"body": {
+							"mode": "raw",
+							"raw": "{\r\n    \"source_model_directory\": \"{{source_model_directory}}\",\r\n    \"source_network\": {\"file_name\":\"{{nwm_data_directory}}\\\\flows.parquet\",\r\n                        \"version\":\"2.1\",  // optional\r\n                        \"type\":\"nwm_hydrofabric\"}\r\n    \r\n}",
+							"options": {
+								"raw": {
+									"language": "json"
+								}
+							}
+						},
+						"url": {
+							"raw": "{{url}}/processes/conflate_model/execution",
+							"host": [
+								"{{url}}"
+							],
+							"path": [
+								"processes",
+								"conflate_model",
+								"execution"
+							]
+						},
+						"description": "Conflate a source HEC-RAS model with the NWM network."
+					},
+					"response": []
+				},
 				{
 					"name": "extract_submodel",
 					"request": {
@@ -97,6 +155,35 @@
 					},
 					"response": []
 				},
+				{
+					"name": "gpkg_from_ras",
+					"request": {
+						"method": "POST",
+						"header": [],
+						"body": {
+							"mode": "raw",
+							"raw": "{\r\n    \"source_model_directory\": \"{{source_model_directory}}\",\r\n    \"crs\": 2227,\r\n    \"metadata\": {\"stac_api\":\"https://stac2.dewberryanalytics.com\",  // optional\r\n                \"stac_collection_id\":\"ebfe-12090301_LowerColoradoCummins\",  // optional\r\n                \"stac_item_id\":\"137a9667-e5cf-4cea-b6ec-2e882a42fdc8\"}  // optional\r\n}\r\n",
+							"options": {
+								"raw": {
+									"language": "json"
+								}
+							}
+						},
+						"url": {
+							"raw": "{{url}}/processes/gpkg_from_ras/execution",
+							"host": [
+								"{{url}}"
+							],
+							"path": [
+								"processes",
+								"gpkg_from_ras",
+								"execution"
+							]
+						},
+						"description": "Create a new model for a given nwm_id (NWM reach) from an existing HEC-RAS model."
+					},
+					"response": []
+				},
 				{
 					"name": "create_ras_terrain",
 					"request": {
@@ -151,7 +238,7 @@
 								"execution"
 							]
 						},
-						"description": "Run a range of flows for Normal Depth on a HEC-RAS model. The results will be used to inform flows required to develop rating curves at a provided increment."
+						"description": "Run a range of flows for dormal depth on a HEC-RAS model. The results will be used to inform flows required to develop rating curves at a provided increment."
 					},
 					"response": []
 				},
@@ -209,7 +296,7 @@
 								"execution"
 							]
 						},
-						"description": "Run a series of HEC-RAS simulations using Known Water Surface Elevations."
+						"description": "Run a HEC-RAS simulation using Known Water Surface Elevations."
 					},
 					"response": []
 				},
@@ -266,7 +353,8 @@
 								"nwm_reach_model_stac",
 								"execution"
 							]
-						}
+						},
+						"description": "Create a STAC item for a HEC-RAS model representing a NWM reach."
 					},
 					"response": []
 				},
@@ -294,7 +382,8 @@
 								"fim_lib_stac",
 								"execution"
 							]
-						}
+						},
+						"description": "Create a STAC item for a FIM library for a NWM reah."
 					},
 					"response": []
 				}
@@ -505,6 +594,11 @@
 			"key": "source_model_directory",
 			"value": "~\\\\repos\\\\ripple1d\\\\tests\\\\ras-data\\\\Baxter",
 			"type": "string"
+		},
+		{
+			"key": "nwm_data_directory",
+			"value": "~\\\\repos\\\\ripple1d\\\\tests\\\\nwm-data",
+			"type": "string"
 		}
 	]
 }

ripple1d/conflate/plotter.py

@@ -99,7 +99,7 @@ def plot_conflation_results(
         buf.seek(0)
         s3_client.put_object(Bucket=bucket, Key=key, Body=buf, ContentType="image/png")
     else:
-        with open(Path(key).name, "w") as f:
-            plt.savefig(f, format="png")
+        output_png = str(Path(rfc.source_model_directory) / Path(key).name)
+        plt.savefig(output_png, format="png")
 
     plt.close()