From 3d4d664b69af4a9e1b80198c8d66e40155528271 Mon Sep 17 00:00:00 2001
From: Eric Peterson <etpeters@sri.com>
Date: Wed, 27 Nov 2024 07:31:30 -0800
Subject: [PATCH] Updates from meeting

---
 tests/README.txt | 46 ++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 40 insertions(+), 6 deletions(-)

diff --git a/tests/README.txt b/tests/README.txt
index 8104d8d..db49243 100644
--- a/tests/README.txt
+++ b/tests/README.txt
@@ -11,21 +11,55 @@ This document describes the testing done in the OSIPI TF2.4 IVIM repository
 -- Testing philosophy --
 Testing is integral to the repository.
 There are many different contributions from many different people and only through diligent testing can they all be ensured to work correctly together.
+Automated testing happens on different platforms and versions.
 There are 3 major types of tests we conduct.
 1. Requirements
-    - Which are required to pass
+    - Runs on each commit
+    - Must pass to merge
     - All algorithms have the same requirements
-    - E.g. bounds honored, code runs reasonably
+    - E.g. bounds honored, code runs reasonably, properly integrated
+    - Would prevent a merge if not passing
+    - Flexibile input/output
+    - Categories for testing
+        -- Contributions - Some aspects currently tested as unit_test
+            --- Initial bounds are respected
+                ---- Needs implemening
+            --- Initial guess is respected
+                ---- Needs implemening - may no be possible
+            --- Runs reasonably
+                ---- Needs implemening: reduced data size, broadened limits
+            --- Contains information about algorithm
+        -- Wrapper
+            --- Initial guess is in bounds
+            --- Reasonable default bounds - f: [0 1], D >= 0 & D < D*, D* >= 0
+            --- Input size is respected - result is same size as input
+            --- Dictionary is returned - worth explicit testing?
+        -- Phantom - lower priority
+            --- Data can be pulled
 2. Expectations
+    - Run on each merge
     - Considered warnings
     - Should not necessarily prevent a merge
-    - Should cover performance changes
-    - E.g. performance changes
-3. Performance
-    - The accuracy of the results
+    - Categories for testing
+        -- Determine performance changes from reference run
+            --- Currently implemented but could be made easier to interact with
+            --- Could be made easier and faster
+3. Characterization
+    - Run on demand
+    - Performance of the algorithms
+    - The accuracy and precision of the results
     - The speed of the generated results
+    - Human readable report of the wrapped algorithms
+    - Categories for testing
+        -- Simulations
+            --- Voxels from tissue and characterize algorithms
+            --- Visualize parameter maps
+        -- True data
+            --- Visualize parameter maps
+            --- Correlations between algorithms - plot the results and differences
 
 -- Testing structure --
+
 * The testing is controlled in several places.
 * The testing itself is done with pytest which parses files for "test_" and runs the appropriate tests.
 * The pytest testing can be done on your own machine by running "python -m pytest".