Edits to documentation marksdown files. replaced the placeholder for …

…the numbered process diagram.
mlte-team · Sep 27, 2024 · 21d483a · 21d483a
1 parent 51c8f7f
commit 21d483a
Show file tree

Hide file tree

Showing 7 changed files with 93 additions and 77 deletions.
diff --git a/docs/docs/development.md b/docs/docs/development.md
@@ -24,21 +24,21 @@ There are several demos available in the `demo\` folder, as Jupyter notebooks. T
 $ poetry install --with demo
 ```
 
-If using macOS, please also install this additional group:
+If using macOS, also install this additional group:
 
 ```bash
 $ poetry install --with demo-mac
 ```
 
 ## Project Development Commands
 
-You can run most project commands (to format sources, lint, etc.), in two ways: using the commands in the included Makefile, or running things manually. Using the Makefile works on UNIX-like systems (or anywhere `make` is available), and is shorter to type. Alternatively, you can run each command manually. The sections below describe how to run commands in both ways.
+You can run most project commands (e.g., format sources, lint), in two ways: using the commands in the included Makefile, or running things manually. Using the Makefile works on UNIX-like systems (or anywhere `make` is available), and is shorter to type. Alternatively, you can run each command manually. The sections below describe how to run commands in both ways.
 
 Also, the commands below do not assume that you have your virtual environment enabled. Calling `poetry run` ensures things run in the current virtual environment even if it is not activated. If you manually activate your virtual environment with `source .venv/bin/activate` (see above), you can run all the commands below without the `poetry run` prefix.
 
 ### Import Sorting
 
-We sort all Python imports code in this project with <a href="https://github.com/PyCQA/isort" target="_blank">`isort`</a>. Assuming you have followed the instructions in the [Quickstart](#quickstart), you can run this locally with:
+We sort all Python import code in this project with <a href="https://github.com/PyCQA/isort" target="_blank">`isort`</a>. Assuming you have followed the instructions in the [Quickstart](#quickstart), you can run this locally with:
 
 ```bash
 $ poetry run make isort
@@ -126,7 +126,7 @@ Alternatively, you can run the tests manually from the project root:
 $ poetry run pytest test
 ```
 
-Unit tests failures result in build failures in CI.
+Unit test failures result in build failures in CI.
 
 To test the Juypter notebooks present in the demo folders, run:
 
@@ -154,8 +154,8 @@ Unit test failures result in build failures in CI.
 
 There are a couple of shorthand commands in the Makefile to run several of the above commands at the same time. The most useful ones include:
 
-* `poetry run make qa`: execues the source sorting, formatting, source linting, and static type checking commands.
-* `poetry run make ci`: execues the same commands as `qa`, but also runs `gen` to generate updated schemas if needed, and runs `test` to execute the unit tests.
+* `poetry run make qa`: executes the source sorting, formatting, source linting, and static type checking commands.
+* `poetry run make ci`: executes the same commands as `qa`, but also runs `gen` to generate updated schemas if needed, and runs `test` to execute the unit tests.
 
 
 ## Front End
@@ -214,7 +214,7 @@ We utilize <a href="https://docs.github.com/en/actions" target="_blank">GitHub A
 
 We build documentation with <a href="https://www.mkdocs.org" target="_blank">`mkdocs`</a> and host documentation on <a href="https://readthedocs.org/" target="_blank">ReadTheDocs</a>. A webhook is set up in the MLTE repository to trigger an integration effect on ReadTheDocs when certain changes to the repo are made.
 
-We maintain a group of requirements for building the documentation under asthe `docs` optional group. They are installed if you follow the general instructions to set up the environment. But if you only want to build the documentation locally, install the requirements from this group, either in the same dev environment or a separate one:
+We maintain a group of requirements for building the documentation under the `docs` optional group. They are installed if you follow the general instructions to set up the environment. But if you only want to build the documentation locally, install the requirements from this group, either in the same dev environment or a separate one:
 
 ```bash
 $ poetry install --with docs
@@ -258,15 +258,15 @@ $ bash build.sh
 You can also do this manually:
 1. Build the static distribution for the front end; the command below assumes that you have the dependencies for frontend builds installed:
 
-```bash
-$ cd mlte/frontend/nuxt-app && npm run build
-```
+    ```bash
+    $ cd mlte/frontend/nuxt-app && npm run build
+    ```
 
 2. Create the source distribution and wheel from the main repo folder:
 
-```bash
-$ poetry build
-```
+    ```bash
+    $ poetry build
+    ```
 
 Once the package is built, publish the package to `PyPi` using a PyPi API token:
 
@@ -290,9 +290,9 @@ Run the containers with:
 bash start.sh
 ```
 
-This exposes the backend on the host at `localhost:8080`, and the frontend at `localhost:8080`. By default, PostgreSQL database is used in a container, and the data is mapped to the local `./pgdata` folder.
+This exposes the backend on the host at `localhost:8080`, and the frontend at `localhost:8000`. By default, PostgreSQL database is used in a container, and the data is mapped to the local `./pgdata` folder.
 
-You can CTRL+C to stop seeing the output in the console, but the containers will continue running. You can check back the current logs at any time with:
+You can CTRL+C to stop seeing the output in the console, but the containers will continue running. You can check the current logs at any time with:
 
 ```bash
 # From inside the docker/deployment folder
@@ -314,9 +314,9 @@ Currently, `MLTE` supports the following Python versions:
 - `3.10`
 - `3.11`
 
-<a href="https://github.com/pyenv/pyenv" target="_blank">`pyenv`</a> can be used to manage multiple Python versions locally. The following procedure can be used to ensure you are running the Python version you need. This procedure only needs to be performed once, during initial version establishment, meaning you _probably_ don't need to be repeating this step in order to contribute to `MLTE`.
+<a href="https://github.com/pyenv/pyenv" target="_blank">`pyenv`</a> can be used to manage multiple Python versions locally. The following procedure can be used to ensure you are running the Python version you need. This procedure only needs to be performed once, during initial version establishment, meaning you _probably_ don't need to repeat this step to contribute to `MLTE`.
 
-### Establishing Depdencies for a Particular Python Version
+### Establishing Dependencies for a Particular Python Version
 
 Install the desired version with:
 
@@ -345,4 +345,4 @@ Once all QA checks and unit tests pass, we can be assured that the environment d
 
 ## Contributing
 
-To contribute to `MLTE`, check out our <a href="https://github.com/mlte-team/mlte" target="_blank">GitHub</a>!
+To contribute to `MLTE`, check out our <a href="https://github.com/mlte-team/mlte" target="_blank">GitHub</a> repository!
diff --git a/docs/docs/img/schema_example.png b/docs/docs/img/schema_example.png
diff --git a/docs/docs/index.md b/docs/docs/index.md
@@ -10,27 +10,27 @@
 *Diagram to be updated October 2024*
 
 ### Continuous Negotiation
-To begin, model developers and project stakeholders meet to determine mission and system requirements that will influence model development such as the deployment environment, available data, model requirements, and system requirements. Throughout the process, teams continue to have meetings to update their assumptions and requirements.
+To begin, model developers and project stakeholders meet to determine mission, business, and system requirements that will influence model development such as the deployment environment, available data, model requirements, and system requirements. Throughout the process, teams continue to have meetings to update their assumptions and requirements.
 
 #### MLTE Negotiation Card
-As part of the negotiation, teams fill out a `MLTE` [negotiation card](negotiation_card.md) which allows them to record agreements and drives model development and testing.
+As part of the negotiation, teams fill out a `MLTE` [Negotiation Card](negotiation_card.md) which allows them to record agreements and drives model development and testing.
 
 #### Quality Attribute Scenarios
 Quality attributes are a way to specify a system’s structural and behavioral requirements; MLTE leverages this approach during negotiations to help teams move from vague statements to concrete requirements.
 
 ### Initial Model Testing (IMT)
-Teams use information from the [negotiation card](negotiation_card.md) during initial model development to inform model requirements and thresholds. Once initial development is complete, model teams do initial testing during this step to determine when the model exceeds their baselines.
+Teams use information from the [Negotiation Card](negotiation_card.md) during initial model development to inform model requirements and thresholds. Once initial development is complete, model teams do initial testing during this step to determine when the model exceeds their baselines.
 
 ### System Dependent Model Testing (SDMT)
-Once a model passes its baseline requirements in IMT, teams can then focus on ensuring that it passes the larger set of system and model requirements. To do so, teams use system requirement and quality attribute information from the [negotiation card](negotiation_card.md) to develop a test specification, which contains code that will evaluate each model or system requirement.
+Once a model passes its baseline requirements in IMT, teams can then focus on ensuring that it passes the larger set of system and model requirements. To do so, teams use system requirement and quality attribute information from the [Negotiation Card](negotiation_card.md) to develop a test specification, which contains code that will evaluate each model or system requirement.
 
 #### Test Catalog
 The `MLTE` Test Catalog contains reusable — local or organizational — examples of test cases organized by quality attribute. Model developers can use it to find examples of tests (like looking for code examples on StackOverflow).
 
 #### Communicating Results
 Once SDMT has provided evidence of how a model performs against required model and system qualities, a MLTE Report can be generated to communicate test results and provide the context for requirements and results.
 
-If results are satisfactory, the output is a production-ready model (meaning that is meets defined system and model requirements), along with all testing evidence (code, data, and results). 
+If results are satisfactory, the output is a production-ready model (meaning that it meets defined system and model requirements), along with all testing evidence (code, data, and results). 
 
 If results are not satisfactory, more negotiation is required to determine if requirements are realistic, whether more experimentation is required, or whether results triggered additional requirements or tests.
 
@@ -39,23 +39,38 @@ If results are not satisfactory, more negotiation is required to determine if re
 - [MLTE Process](mlte_process.md) (A more detailed guide than above)
 - [Setting Up MLTE](setting_up_mlte.md)
 - [Development](development.md)
-- MLTE <a href="https://arxiv.org/abs/2303.01998" target="_blank">Paper</a> (ICSE 2023)
+- MLTE <a href="https://arxiv.org/abs/2303.01998" target="_blank">Paper</a> (ICSE 2023 - 45th International Conference on Software Engineering)
+- <a href="https://doi.org/10.48550/arXiv.2406.08575" target="_blank">Using Quality Attribute Scenarios for ML Model Test Case Generation</a> (SAML 2024 - 3rd International Workshop on Software Architecture and Machine Learning)
 
 ## MLTE Metadata
 
-- Version: 0.3.0
+- Version: 1.0
 - Contact Email: mlte dot team dot info at gmail dot com
-- Citation: While not required, it is highly encouraged and greatly appreciated if you cite our paper when you use `MLTE` for academic research.
-
-```
-@article{maffey2023mlteing,
-  title={MLTEing Models: Negotiating, Evaluating, and Documenting 
-  Model and System Qualities},
-  author={Maffey, Katherine R and Dotterrer, Kyle and Niemann, Jennifer 
-  and Cruickshank, Iain and Lewis, Grace A and K{\"a}stner, Christian},
-  journal={arXiv preprint arXiv:2303.01998},
-  year={2023}
-}
-```
+- Citations: While not required, it is highly encouraged and greatly appreciated if you cite our paper when you use `MLTE` for academic research.
+
+    ```
+    @inproceedings{maffey2023,
+        title={{MLTEing models: Negotiating, Evaluating, and Documenting Model and System Qualities}},
+        author={Maffey, Katherine R and Dotterrer, Kyle and Niemann, Jennifer and Cruickshank, Iain 
+        and Lewis, Grace A and K{\"a}stner, Christian},
+        booktitle={2023 IEEE/ACM 45th International Conference on Software Engineering: New Ideas and
+        Emerging Results (ICSE-NIER)},
+        pages={31--36},
+        year={2023},
+        organization={IEEE}
+    }
+    ```
+    ... or if you use, or are inspired by, quality attributes for ML model test case generation.
+    ```
+    @inproceedings{brower2024,
+        author={Brower-Sinning, Rachel and Lewis, Grace A. and Echeverría, Sebastián and Ozkaya, Ipek},
+        booktitle={2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C)}, 
+        title={Using Quality Attribute Scenarios for ML Model Test Case Generation}, 
+        year={2024},
+        pages={307-310},
+        organization={IEEE}
+    }  
+    ```
+
 
 ## Check out `MLTE` on <a href="https://github.com/mlte-team/mlte" target="_blank">GitHub</a>!
diff --git a/docs/docs/mlte_process.md b/docs/docs/mlte_process.md
@@ -1,34 +1,33 @@
 # MLTE Process
-![MLTE Diagram](img/placeholder_numbered_diagram.png)
-*Placeholder diagram*
+![MLTE Diagram](img/mlte_numbered_diagram.png)
 
 ## 1. Continuous Negotiation
 The process starts with a negotiation step between model developers and project stakeholders where the goal is to share information about mission and system requirements that will influence model development, such as the deployment environment, available data, model requirements, and system requirements.
 
-This negotiation continues throughout model development, in response (for example) to missing information, unrealistic expectations, and/or test results that do not meet system requirements,
+This negotiation continues throughout model development, in response to, for example, to missing information, unrealistic expectations, and/or test results that do not meet system requirements.
 
-### 2. Quality Attribute Scenarios
+## 2. Quality Attribute Scenarios
 Quality attributes are a way to specify a system’s structural and behavioral requirements; MLTE leverages this approach during negotiations to help teams move from vague statements to concrete requirements. More information on using quality attributes can be found by reading this <a href="https://arxiv.org/abs/2406.08575" target="_blank">paper</a>.
 
 
-### 3. Negotiation Card
-As part of the negotiation, teams fill out a `MLTE` [negotiation card](negotiation_card.md) which allows them to record agreements and drives model development and testing.
+## 3. Negotiation Card
+As part of the negotiation, teams fill out a `MLTE` [Negotiation Card](negotiation_card.md) which allows them to record agreements and drives model development and testing.
 
 ## 4. Initial Model Testing (IMT)
-Teams use information from the [negotiation card](negotiation_card.md) during initial model development to inform model requirements and thresholds. Once initial development is complete, model teams do initial testing during this step to determine when the model exceeds their baselines.
+IMT recognizes the iterative and experimental nature of model development. Teams use information from the [Negotiation Card](negotiation_card.md) during initial model development to inform model requirements, performance thresholds, and design decisions. Once initial development is complete, model teams perform initial testing during this step to determine when the model exceeds baselines. Once model performance exceeds baselines, or if additional testing is needed to validate assumptions, the procees moves to SDMT.
 
 ## 5. System Dependent Model Testing (SDMT)
-Once a model passes its baseline requirements in IMT, teams can then focus on ensuring that it passes the larger set of system and model requirements. To do so, teams use system requirement and quality attribute information from the [negotiation card](negotiation_card.md) to develop a test specification, which contains code that will evaluate each model or system requirement.
+In SDMT, teams focus on ensuring that the model passes the larger set of system and model requirements. To do so, teams use system requirements and quality attribute information from the [Negotiation Card](negotiation_card.md) to develop a test specification, which contains code that will evaluate each model or system requirement.
 
-### 6. Test Catalog
-The `MLTE` Test Catalog contains reusable — local or organizational — examples of test cases organized by quality attribute. Model developers can use it to find examples of tests (like looking for code examples on StackOverflow). Model developers can also contribute test code back to the Test Catalog so that it can be used by others.
+## 6. Test Catalog
+The `MLTE` Test Catalog contains reusable — local or organizational — examples of test cases organized by quality attribute. Model developers can use the catalog to find examples of tests, similar to looking for code examples on StackOverflow. Model developers can also contribute test code back to the Test Catalog so that it can be used by others.
 
-### 7. Test Cases
+## 7. Test Cases
 Test cases are derived from the test specification that defines metrics, measurement methods, and passing conditions.
 
-### 8. `MLTE` Report
+## 8. `MLTE` Report
 Once test cases are executed, a `MLTE` Report can be generated to communicate test results and provide the context for requirements and results.
 
-If stakeholders consider the results to be satisfactory, the result is a production-ready model (meaning that is meets defined system and model requirements), along with all testing evidence (code, data, and results). This evidence can be used for stakeholders to repeat tests, expand tests, or make decisions about additional testing effort required. An additional benefit is support for regression testing in response to model maintenance and evolution.
+If stakeholders consider the results to be satisfactory, the outcome is a production-ready model (meaning that is meets defined system and model requirements), along with all testing evidence (code, data, and results). This evidence can be used for stakeholders to repeat tests, expand tests, or make decisions about additional testing effort required. An additional benefit is support for regression testing in response to model maintenance and evolution.
 
 If stakeholders do not consider the results to be satisfactory, more negotiation is required to determine if requirements are realistic, whether more experimentation is required, or whether results triggered additional requirements or tests.
diff --git a/docs/docs/negotiation_card.md b/docs/docs/negotiation_card.md
@@ -224,7 +224,7 @@ Measures used to determine that the responses enumerated for the scenario have b
 
 ### System Quality Statement
 
-Scenario for {System Quality}: {Stimulus} from {Source} during {Environment}. {Response} {Response Measure}.
+As the information above is added, the text for the full scenario is constructed. Adjust the infornation until there is a clear statement of the quaity attribute scenario.
 
 #### Example: Response Time