From 21dcd7211f6261c6b8658de3e915d40cc8e6a362 Mon Sep 17 00:00:00 2001 From: "Roscoe A. Bartlett" Date: Fri, 23 Aug 2024 12:04:30 -0400 Subject: [PATCH 01/10] Fix multi-level list formatting, add newlines for every sentence I noticed the messed up formatting when I saw this article on bssw.io. (I did not realize that this howto article got converted over to bssw.io. Not sure what I did not at least review that article at the time.) Each sentence is now on its own line. --- ...oImproveTestinginyourCSESoftwareProject.md | 83 ++++++++++++------- 1 file changed, 53 insertions(+), 30 deletions(-) diff --git a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md index 30ba9ab276..8b814325ed 100644 --- a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md +++ b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md @@ -3,62 +3,85 @@ #### Publication Date: August 6, 2019 -Software requires regular extensive testing to ensure correctly functioning code. This article provides basic information on CSE software testing. +Software requires regular extensive testing to ensure correctly functioning code. +This article provides basic information on CSE software testing. ### Overview -Adding tests of sufficient coverage and quality improves confidence in software and makes it easier to change and extend. Tests should be added to existing code before the code is changed. Tests should be added to new code before (or while) it is being written. These tests then become the foundation of a regression test suite that helps effectively drive future -development and improves long-term sustainability. +Adding tests of sufficient coverage and quality improves confidence in software and makes it easier to change and extend. +Tests should be added to existing code before the code is changed. +Tests should be added to new code before (or while) it is being written. +These tests then become the foundation of a regression test suite that helps effectively drive future development and improves long-term sustainability. ### Target Audience -CSE software project leaders and developers who are facing significant refactoring efforts because of hardware architecture changes or increased demands for multiphysics and multiscale coupling, and who want to increase the quality and speed of -development and reduce development and maintenance costs. +CSE software project leaders and developers who are facing significant refactoring efforts because of hardware architecture changes or increased demands for multiphysics and multiscale coupling, and who want to increase the quality and speed of development and reduce development and maintenance costs. ### Purpose -Show how to add quality testing to a project in order to support efficient modification of existing code or addition of new code. Show how to add tests to support (1) **adding a new feature**, (2) **fixing a bug**,(3) **improving the design and implementation**,or (4) **optimizing resource usage** +Show how to add quality testing to a project in order to support efficient modification of existing code or addition of new code. +Show how to add tests to support (1) **adding a new feature**, (2) **fixing a bug**,(3) **improving the design and implementation**,or (4) **optimizing resource usage** ### Prerequisites First read the document *[What Are Software Testing Practices?](./UnderstandingSoftwareTestingPractices.md)* and browse through *[Definition and Categorization of Tests for CSE Software](./DefinitionsCategorizationsOfTests.md)*. ### Steps + 1. Set up **automated builds of the code** with high warning levels and eliminate all warnings. 2. **Select test harness frameworks** - a. **Select a system-level test harness** for system-executable tests that report results appropriately (e.g., CTest/CDash, Jenkins). - b. **Select a unit test harness** to effectively define and run finer-grained integration and unit tests (e.g., Google Test, pFUnit). - c. **Customize or streamline** system-level and/or unit test frameworks for use in your particular project. + 1. **Select a system-level test harness** for system-executable tests that report results appropriately (e.g., CTest/CDash, Jenkins). + 1. **Select a unit test harness** to effectively define and run finer-grained integration and unit tests (e.g., Google Test, pFUnit). + 1. **Customize or streamline** system-level and/or unit test frameworks for use in your particular project. 3. **Add system-level tests** to protect major user functionality. - a. Select inputs for several important problem classes and run code to produce outputs. - b. Set up no-change or verification tests with a system-level test harness in order to pin down important behavior. + 1. Select inputs for several important problem classes and run code to produce outputs. + 1. Set up no-change or verification tests with a system-level test harness in order to pin down important behavior. 4. **Add integration and unit tests** (as needed for adding/changing code) - a. **Incorporate test** [1, 2] **for code to be changed** - * **Identify change points** for target change or new code - * **Find test points** where code behavior can be sensed. - * **Break dependencies** in order to get the targeted code into the unit test harness. - * **Cover targeted code** to be changed with sufficient (characterization) tests. - b. **Add new features or fix bugs with tests** [1, 2, 3, 4] - * **Add new tests** that define desired behavior (feature or bug). - * Run new tests and **verify they fail**. - * Add the minimal code to **get new tests to pass**. - * **Refactor** the covered code to clean up and remove duplication. - * **Review** all changes to existing code, new code and new tests. + 1. **Incorporate test** [1, 2] **for code to be changed** + * **Identify change points** for target change or new code + * **Find test points** where code behavior can be sensed. + * **Break dependencies** in order to get the targeted code into the unit test harness. + * **Cover targeted code** to be changed with sufficient (characterization) tests. + 2. **Add new features or fix bugs with tests** [1, 2, 3, 4] + * **Add new tests** that define desired behavior (feature or bug). + * Run new tests and **verify they fail**. + * Add the minimal code to **get new tests to pass**. + * **Refactor** the covered code to clean up and remove duplication. + * **Review** all changes to existing code, new code and new tests. 5. Select **code coverage** (e.g., gcov/lcov) and **memory usage error detection** (e.g., valgrind) analysis tools. 6. Define a set of **regression test suites**. - a. Define a faster-running **pre-push regression test suite** (e.g., single build with faster running tests) and **run it before every push**. - b. Define a more comprehensive nightly **regression test suite** (e.g., builds and all tests on several platforms and compilers, code coverage, and memory usage error detection) and **run every night**. + 1. Define a faster-running **pre-push regression test suite** (e.g., single build with faster running tests) and **run it before every push**. + 1. Define a more comprehensive nightly **regression test suite** (e.g., builds and all tests on several platforms and compilers, code coverage, and memory usage error detection) and **run every night**. 7. Have a policy of **100% passing pre-push regression tests** and work hard to maintain that. 8. Work to **fix all failing nighty regression tests** on a reasonable schedule. ### FAQs: -**Q: Why do you need both a system-level and a unit test harness?** -A: A unit test harness aggregates hundreds of unit and integration tests into single executables. A system-level test harness runs these aggregate integration and unit test executables along with the other system-level acceptance and verification tests and alerts developers of any failures. -**Q: Why not just add all of the tests for an existing code and get it over with?** -A: Taking weeks or months (or years) to add sufficient tests for an entire existing code (that lacks sufficient testing) is not usually economical or necessary. Tests need to be added to code only when it is changed (or when adding new code). In that way tests can be added while regular development work is being done. +**Q: Why do you need both a system-level and a unit test harness?**
+**A**: A unit test harness aggregates hundreds of unit and integration tests into single executables. +A system-level test harness runs these aggregate integration and unit test executables along with the other system-level acceptance and verification tests and alerts developers of any failures. + +**Q: Why not just add all of the tests for an existing code and get it over with?**
+**A**: Taking weeks or months (or years) to add sufficient tests for an entire existing code (that lacks sufficient testing) is not usually economical or necessary. +Tests need to be added to code only when it is changed (or when adding new code). +In that way tests can be added while regular development work is being done. + +**Q: Why demand 100% passing pre-push regression tests?**
+**A**: This avoids expensive debugging and other investigations needed to determine whether your changes are breaking failing tests or not (hard). +If all tests pass, then your changes could be breaking them (easy). + + ### References: From 21283341b8f08f318a3296f8f4f5901e5d2196f5 Mon Sep 17 00:00:00 2001 From: "Roscoe A. Bartlett" Date: Fri, 23 Aug 2024 12:12:29 -0400 Subject: [PATCH 02/10] Fix list of authors to be the full set This is consistent with the other copied over article "Definition and Categorization of Tests for CSE Software" that listed the full set of original authors. --- .../HowtoImproveTestinginyourCSESoftwareProject.md | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md index 8b814325ed..fc96195e47 100644 --- a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md +++ b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md @@ -1,5 +1,7 @@ # How to Improve Testing for CSE Software -#### Contributed by [Ulrike Yang](https://github.com/ulrikeyang) + +#### Contributed by [Roscoe A. Bartlett](https://github.com/bartlettroscoe), [Barry Smith](https://github.com/BarrySmith), [Ulrike Meier Yang](https://github.com/ulrikeyang), [Glenn Hammond](https://github.com/ghammond86), [Xiaoye Li](https://github.com/xiaoyeli), and [James Willenbring](https://github.com/jwillenbring) + #### Publication Date: August 6, 2019 @@ -73,14 +75,10 @@ If all tests pass, then your changes could be breaking them (easy). ToDos: -* Fix list formatting ... IN PROGRESS ... - -* Fix list of authors ... +* Switch to wikize-refs.py format ... * Improve the deck text ... -* Switch to wikize-refs.py format ... - ---> ### References: @@ -90,9 +88,6 @@ ToDos: 3. [Test Driven Development](https://www.oreilly.com/library/view/test-driven-development/0321146530/), Kent Beck, 2003. 4. [Code Complete (Second Edition)](https://www.microsoftpressstore.com/store/code-complete-9780735619678), Steve McConnell, 2004. -This document was prepared with input from Roscoe A. Bartlett, Glenn Hammond, Xiaoye Li, Barry Smith, -and James M. Willenbring. - -### References: - -1. [Working Effectively with Legacy Code. Prentice Hall](https://www.oreilly.com/library/view/working-effectively-with/0131177052/), Michael Feathers, 2005. -2. [Legacy Software Change Algorithm](https://trilinos.org/trac/trilinos/wiki/TribitsLegacySoftwareChangeAlgorithm). -3. [Test Driven Development](https://www.oreilly.com/library/view/test-driven-development/0321146530/), Kent Beck, 2003. -4. [Code Complete (Second Edition)](https://www.microsoftpressstore.com/store/code-complete-9780735619678), Steve McConnell, 2004. + +[1]: https://bssw.io/items/working-effectively-with-legacy-code "Working Effectively with Legacy Code {Michael Feathers. Prentice Hall, 2005}" +[2]: https://www.oreilly.com/library/view/test-driven-development/0321146530/ "Test Driven Development {Kent Beck, Addison-Wesley Professional, 2003, ISBN: 0321146530}" +[3]: https://bssw.io/items/code-complete-a-practical-handbook-of-software-construction "Code Complete: Second Edition}" + From d81f9413b93c3e230a415504c363b7422a222554 Mon Sep 17 00:00:00 2001 From: "Roscoe A. Bartlett" Date: Fri, 23 Aug 2024 12:43:42 -0400 Subject: [PATCH 04/10] Fix some more space typos --- Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md index ca9694d966..b6d51d8682 100644 --- a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md +++ b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md @@ -20,7 +20,7 @@ CSE software project leaders and developers who are facing significant refactori ### Purpose Show how to add quality testing to a project in order to support efficient modification of existing code or addition of new code. -Show how to add tests to support (1) **adding a new feature**, (2) **fixing a bug**,(3) **improving the design and implementation**,or (4) **optimizing resource usage** +Show how to add tests to support (1) **adding a new feature**, (2) **fixing a bug**, (3) **improving the design and implementation**, or (4) **optimizing resource usage** ### Prerequisites From 855050b8d543d35050b7f91bee548d657af1a1e3 Mon Sep 17 00:00:00 2001 From: "Roscoe A. Bartlett" Date: Fri, 23 Aug 2024 12:45:01 -0400 Subject: [PATCH 05/10] WIP: Run wikize-refs.py (broken links) This runs wikize-refs.py but it creates an off-by one error. I will fix that in the next commit manually. --- ...toImproveTestinginyourCSESoftwareProject.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md index b6d51d8682..3b943197dc 100644 --- a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md +++ b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md @@ -88,7 +88,17 @@ Topics: Testing Track: how to ---> -[1]: https://bssw.io/items/working-effectively-with-legacy-code "Working Effectively with Legacy Code {Michael Feathers. Prentice Hall, 2005}" -[2]: https://www.oreilly.com/library/view/test-driven-development/0321146530/ "Test Driven Development {Kent Beck, Addison-Wesley Professional, 2003, ISBN: 0321146530}" -[3]: https://bssw.io/items/code-complete-a-practical-handbook-of-software-construction "Code Complete: Second Edition}" - +[1-sfer-ezikiw]: https://bssw.io/items/working-effectively-with-legacy-code "Working Effectively with Legacy Code {Michael Feathers. Prentice Hall, 2005}" +[2-sfer-ezikiw]: https://www.oreilly.com/library/view/test-driven-development/0321146530/ "Test Driven Development {Kent Beck, Addison-Wesley Professional, 2003, ISBN: 0321146530}" +[3-sfer-ezikiw]: https://bssw.io/items/code-complete-a-practical-handbook-of-software-construction "Code Complete: Second Edition}" + + +[1]: #sfer-ezikiw-1 "Working Effectively with Legacy Code" +[2]: #sfer-ezikiw-2 "Test Driven Development" +[3]: #sfer-ezikiw-3 "Code Complete: Second Edition}" + +### References + +* 2[Working Effectively with Legacy Code
Michael Feathers. Prentice Hall, 2005](https://bssw.io/items/working-effectively-with-legacy-code) +* 3[Test Driven Development
Kent Beck, Addison-Wesley Professional, 2003, ISBN: 0321146530](https://www.oreilly.com/library/view/test-driven-development/0321146530/) +* 4[Code Complete: Second Edition}](https://bssw.io/items/code-complete-a-practical-handbook-of-software-construction) From 283acd380b0e355b65478e2c2f28682e4017664e Mon Sep 17 00:00:00 2001 From: "Roscoe A. Bartlett" Date: Fri, 23 Aug 2024 12:47:16 -0400 Subject: [PATCH 06/10] Fix references off-by-one error from wikize-refs.py Now the links are correct. It was easy to fix. --- .../Blog/HowtoImproveTestinginyourCSESoftwareProject.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md index 3b943197dc..73ea23251d 100644 --- a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md +++ b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md @@ -99,6 +99,6 @@ Track: how to ### References -* 2[Working Effectively with Legacy Code
Michael Feathers. Prentice Hall, 2005](https://bssw.io/items/working-effectively-with-legacy-code) -* 3[Test Driven Development
Kent Beck, Addison-Wesley Professional, 2003, ISBN: 0321146530](https://www.oreilly.com/library/view/test-driven-development/0321146530/) -* 4[Code Complete: Second Edition}](https://bssw.io/items/code-complete-a-practical-handbook-of-software-construction) +* 1[Working Effectively with Legacy Code
Michael Feathers. Prentice Hall, 2005](https://bssw.io/items/working-effectively-with-legacy-code) +* 2[Test Driven Development
Kent Beck, Addison-Wesley Professional, 2003, ISBN: 0321146530](https://www.oreilly.com/library/view/test-driven-development/0321146530/) +* 3[Code Complete: Second Edition}](https://bssw.io/items/code-complete-a-practical-handbook-of-software-construction) From db414fec45a32c0f3d87f1174965944099ec0cee Mon Sep 17 00:00:00 2001 From: "Roscoe A. Bartlett" Date: Fri, 23 Aug 2024 13:06:02 -0400 Subject: [PATCH 07/10] Update text * More informative deck text * Define CSE * Add "uncovered" for LSCA stuff * Change from "push" to "merge" to recognised the modern way to do testing with modern CI testing systems like GitHub Actions and GitLab CI --- ...oImproveTestinginyourCSESoftwareProject.md | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md index 73ea23251d..4feb44c151 100644 --- a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md +++ b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md @@ -6,21 +6,21 @@ Software requires regular extensive testing to ensure correctly functioning code. -This article provides basic information on CSE software testing. +This article provides a straightforward process to add testing to an existing software project that has no testing (or insufficient testing). ### Overview Adding tests of sufficient coverage and quality improves confidence in software and makes it easier to change and extend. -Tests should be added to existing code before the code is changed. +Tests should be added to existing uncovered code before the code is changed. Tests should be added to new code before (or while) it is being written. -These tests then become the foundation of a regression test suite that helps effectively drive future development and improves long-term sustainability. +These tests then become the foundation of a regression test suite that helps effectively drive future development while maintaining behavior and improves long-term sustainability. ### Target Audience -CSE software project leaders and developers who are facing significant refactoring efforts because of hardware architecture changes or increased demands for multiphysics and multiscale coupling, and who want to increase the quality and speed of development and reduce development and maintenance costs. +Computational Science and Engineering (CSE) software project leaders and developers who are facing significant refactoring efforts because of hardware architecture changes or increased demands for multi-physics and multi-scale coupling, and who want to increase the quality and speed of development and reduce development and maintenance costs. ### Purpose Show how to add quality testing to a project in order to support efficient modification of existing code or addition of new code. -Show how to add tests to support (1) **adding a new feature**, (2) **fixing a bug**, (3) **improving the design and implementation**, or (4) **optimizing resource usage** +Show how to add tests to support (1) **adding a new feature**, (2) **fixing a bug**, (3) **improving the design and implementation**, or (4) **optimizing resource usage**[1] ### Prerequisites @@ -37,23 +37,23 @@ First read the document *[What Are Software Testing Practices?](./UnderstandingS 1. Select inputs for several important problem classes and run code to produce outputs. 1. Set up no-change or verification tests with a system-level test harness in order to pin down important behavior. 4. **Add integration and unit tests** (as needed for adding/changing code) - 1. **Incorporate tests for code to be changed**[1] - * **Identify change points** for target change or new code + 1. **Incorporate tests for uncovered code to be changed** using the ***Legacy Software Change Algorithm***[1] + * **Identify change points** for target change or new code. * **Find test points** where code behavior can be sensed. * **Break dependencies** in order to get the targeted code into the unit test harness. * **Cover targeted code** to be changed with sufficient (characterization) tests. - 2. **Add new features or fix bugs with tests**[1],[2],[3] + 2. **Add new features or fix bugs**[1],[2],[3] * **Add new tests** that define desired behavior (feature or bug). * Run new tests and **verify they fail**. * Add the minimal code to **get new tests to pass**. * **Refactor** the covered code to clean up and remove duplication. * **Review** all changes to existing code, new code and new tests. -5. Select **code coverage** (e.g., gcov/lcov) and **memory usage error detection** (e.g., valgrind) analysis tools. +5. Select **code coverage** (e.g., gcov/lcov) and **memory usage error detection** (e.g., valgrind, clang memory/address/leak sanitizers) analysis tools. 6. Define a set of **regression test suites**. - 1. Define a faster-running **pre-push regression test suite** (e.g., single build with faster running tests) and **run it before every push**. + 1. Define a faster-running **pre-merge regression test suite** (e.g., single build with faster running tests) and **run it before every merge to the mainline branch**. 1. Define a more comprehensive nightly **regression test suite** (e.g., builds and all tests on several platforms and compilers, code coverage, and memory usage error detection) and **run every night**. -7. Have a policy of **100% passing pre-push regression tests** and work hard to maintain that. -8. Work to **fix all failing nighty regression tests** on a reasonable schedule. +7. Have a policy of **100% passing pre-merge regression tests** and work hard to maintain that. +8. Work to **fix all failing nightly regression tests** on a reasonable schedule. ### FAQs: @@ -67,7 +67,7 @@ A system-level test harness runs these aggregate integration and unit test execu Tests need to be added to code only when it is changed (or when adding new code). In that way tests can be added while regular development work is being done. -**Q: Why demand 100% passing pre-push regression tests?**
+**Q: Why demand 100% passing pre-merge regression tests?**
**A**: This avoids expensive debugging and other investigations needed to determine whether your changes are breaking failing tests or not (hard). If all tests pass, then your changes could be breaking them (easy). From 0c4b2a9d60ee4a100aedfb9cdabcbb444760cabf Mon Sep 17 00:00:00 2001 From: "Roscoe A. Bartlett" Date: Fri, 23 Aug 2024 13:12:46 -0400 Subject: [PATCH 08/10] Specifically mention GitHub Actions, GitLab CI and move Jenkins down with them --- Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md index 4feb44c151..7fc83d0a88 100644 --- a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md +++ b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md @@ -30,7 +30,7 @@ First read the document *[What Are Software Testing Practices?](./UnderstandingS 1. Set up **automated builds of the code** with high warning levels and eliminate all warnings. 2. **Select test harness frameworks** - 1. **Select a system-level test harness** for system-executable tests that report results appropriately (e.g., CTest/CDash, Jenkins). + 1. **Select a system-level test harness** for system-executable tests that report results appropriately (e.g., CTest/CDash). 1. **Select a unit test harness** to effectively define and run finer-grained integration and unit tests (e.g., Google Test, pFUnit). 1. **Customize or streamline** system-level and/or unit test frameworks for use in your particular project. 3. **Add system-level tests** to protect major user functionality. @@ -52,7 +52,7 @@ First read the document *[What Are Software Testing Practices?](./UnderstandingS 6. Define a set of **regression test suites**. 1. Define a faster-running **pre-merge regression test suite** (e.g., single build with faster running tests) and **run it before every merge to the mainline branch**. 1. Define a more comprehensive nightly **regression test suite** (e.g., builds and all tests on several platforms and compilers, code coverage, and memory usage error detection) and **run every night**. -7. Have a policy of **100% passing pre-merge regression tests** and work hard to maintain that. +7. Have a policy of **100% passing pre-merge regression tests** (run using a CI testing system like GitHub Actions, GitLab CI, or Jenkins) and work hard to maintain that. 8. Work to **fix all failing nightly regression tests** on a reasonable schedule. From 79c2a2125b6e2edb80b719f0e6cb00b78837be70 Mon Sep 17 00:00:00 2001 From: "Roscoe A. Bartlett" Date: Fri, 23 Aug 2024 17:19:48 -0400 Subject: [PATCH 09/10] Switch to explicit HTML multi-level enumerated list This is a lot of manual HTML formatting which included having to: * Switch to HTML formatting for bold and italics * Manually put in links to the references (which does not support the tool-tip). Not so much markup anymore, but let's see what the bssw.io site generator does with this. --- ...oImproveTestinginyourCSESoftwareProject.md | 73 ++++++++++++------- 1 file changed, 46 insertions(+), 27 deletions(-) diff --git a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md index 7fc83d0a88..d491d9e724 100644 --- a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md +++ b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md @@ -28,33 +28,52 @@ First read the document *[What Are Software Testing Practices?](./UnderstandingS ### Steps -1. Set up **automated builds of the code** with high warning levels and eliminate all warnings. -2. **Select test harness frameworks** - 1. **Select a system-level test harness** for system-executable tests that report results appropriately (e.g., CTest/CDash). - 1. **Select a unit test harness** to effectively define and run finer-grained integration and unit tests (e.g., Google Test, pFUnit). - 1. **Customize or streamline** system-level and/or unit test frameworks for use in your particular project. -3. **Add system-level tests** to protect major user functionality. - 1. Select inputs for several important problem classes and run code to produce outputs. - 1. Set up no-change or verification tests with a system-level test harness in order to pin down important behavior. -4. **Add integration and unit tests** (as needed for adding/changing code) - 1. **Incorporate tests for uncovered code to be changed** using the ***Legacy Software Change Algorithm***[1] - * **Identify change points** for target change or new code. - * **Find test points** where code behavior can be sensed. - * **Break dependencies** in order to get the targeted code into the unit test harness. - * **Cover targeted code** to be changed with sufficient (characterization) tests. - 2. **Add new features or fix bugs**[1],[2],[3] - * **Add new tests** that define desired behavior (feature or bug). - * Run new tests and **verify they fail**. - * Add the minimal code to **get new tests to pass**. - * **Refactor** the covered code to clean up and remove duplication. - * **Review** all changes to existing code, new code and new tests. -5. Select **code coverage** (e.g., gcov/lcov) and **memory usage error detection** (e.g., valgrind, clang memory/address/leak sanitizers) analysis tools. -6. Define a set of **regression test suites**. - 1. Define a faster-running **pre-merge regression test suite** (e.g., single build with faster running tests) and **run it before every merge to the mainline branch**. - 1. Define a more comprehensive nightly **regression test suite** (e.g., builds and all tests on several platforms and compilers, code coverage, and memory usage error detection) and **run every night**. -7. Have a policy of **100% passing pre-merge regression tests** (run using a CI testing system like GitHub Actions, GitLab CI, or Jenkins) and work hard to maintain that. -8. Work to **fix all failing nightly regression tests** on a reasonable schedule. - +
    +
  1. Set up automated builds of the code with high warning levels and eliminate all warnings.
    2. +
  2. Select test harness frameworks +
      +
    1. Select a system-level test harness for system-executable tests that report results appropriately (e.g., CTest/CDash).
      2. +
    2. Select a unit test harness to effectively define and run finer-grained integration and unit tests (e.g., Google Test, pFUnit).
      3. +
    3. Customize or streamline system-level and/or unit test frameworks for use in your particular project.
      4. +
    +
    3. +
  3. Add system-level tests to protect major user functionality. +
      +
    1. Select inputs for several important problem classes and run code to produce outputs.
      2. +
    2. Set up no-change or verification tests with a system-level test harness in order to pin down important behavior.
      3. +
    +
    4. +
  4. Add integration and unit tests (as needed for adding/changing code) +
      +
    1. Incorporate tests for uncovered code to be changed using the Legacy Software Change Algorithm1 +
        +
      • Identify change points for target change or new code.
        • +
      • Find test points where code behavior can be sensed.
        • +
      • Break dependencies in order to get the targeted code into the unit test harness.
        • +
      • Cover targeted code to be changed with sufficient (characterization) tests.
        • +
      +
      2. +
    2. Add new features or fix bugs1,2,3 +
        +
      • Add new tests that define desired behavior (feature or bug).
        • +
      • Run new tests and verify they fail.
        • +
      • Add the minimal code to get new tests to pass.
        • +
      • Refactor the covered code to clean up and remove duplication.
        • +
      • Review all changes to existing code, new code and new tests.
        • +
      +
      3. +
    +
    5. +
  5. Select code coverage (e.g., gcov/lcov) and memory usage error detection (e.g., valgrind, clang memory/address/leak sanitizers) analysis tools.
    6. +
  6. Define a set of regression test suites. +
      +
    1. Define a faster-running pre-merge regression test suite (e.g., single build with faster running tests) and run it before every merge to the mainline branch.
      2. +
    2. Define a more comprehensive nightly regression test suite (e.g., builds and all tests on several platforms and compilers, code coverage, and memory usage error detection) and run every night.
      3. +
    +
    7. +
  7. Have a policy of 100% passing pre-merge regression tests (run using a CI testing system like GitHub Actions, GitLab CI, or Jenkins) and work hard to maintain that.
    8. +
  8. 8. Work to fix all failing nightly regression tests on a reasonable schedule.
    9. +
### FAQs: From d45b91134af72a40a7034b131f792d351a7741b8 Mon Sep 17 00:00:00 2001 From: "Roscoe A. Bartlett" Date: Fri, 23 Aug 2024 17:52:15 -0400 Subject: [PATCH 10/10] Revert "Switch to explicit HTML multi-level enumerated list" This reverts commit 79c2a2125b6e2edb80b719f0e6cb00b78837be70. This did not improve the display of the generated page on preview.bssw.io :-( --- ...oImproveTestinginyourCSESoftwareProject.md | 73 +++++++------------ 1 file changed, 27 insertions(+), 46 deletions(-) diff --git a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md index d491d9e724..7fc83d0a88 100644 --- a/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md +++ b/Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md @@ -28,52 +28,33 @@ First read the document *[What Are Software Testing Practices?](./UnderstandingS ### Steps -
    -
  1. Set up automated builds of the code with high warning levels and eliminate all warnings.
    2. -
  2. Select test harness frameworks -
      -
    1. Select a system-level test harness for system-executable tests that report results appropriately (e.g., CTest/CDash).
      2. -
    2. Select a unit test harness to effectively define and run finer-grained integration and unit tests (e.g., Google Test, pFUnit).
      3. -
    3. Customize or streamline system-level and/or unit test frameworks for use in your particular project.
      4. -
    -
    3. -
  3. Add system-level tests to protect major user functionality. -
      -
    1. Select inputs for several important problem classes and run code to produce outputs.
      2. -
    2. Set up no-change or verification tests with a system-level test harness in order to pin down important behavior.
      3. -
    -
    4. -
  4. Add integration and unit tests (as needed for adding/changing code) -
      -
    1. Incorporate tests for uncovered code to be changed using the Legacy Software Change Algorithm1 -
        -
      • Identify change points for target change or new code.
        • -
      • Find test points where code behavior can be sensed.
        • -
      • Break dependencies in order to get the targeted code into the unit test harness.
        • -
      • Cover targeted code to be changed with sufficient (characterization) tests.
        • -
      -
      2. -
    2. Add new features or fix bugs1,2,3 -
        -
      • Add new tests that define desired behavior (feature or bug).
        • -
      • Run new tests and verify they fail.
        • -
      • Add the minimal code to get new tests to pass.
        • -
      • Refactor the covered code to clean up and remove duplication.
        • -
      • Review all changes to existing code, new code and new tests.
        • -
      -
      3. -
    -
    5. -
  5. Select code coverage (e.g., gcov/lcov) and memory usage error detection (e.g., valgrind, clang memory/address/leak sanitizers) analysis tools.
    6. -
  6. Define a set of regression test suites. -
      -
    1. Define a faster-running pre-merge regression test suite (e.g., single build with faster running tests) and run it before every merge to the mainline branch.
      2. -
    2. Define a more comprehensive nightly regression test suite (e.g., builds and all tests on several platforms and compilers, code coverage, and memory usage error detection) and run every night.
      3. -
    -
    7. -
  7. Have a policy of 100% passing pre-merge regression tests (run using a CI testing system like GitHub Actions, GitLab CI, or Jenkins) and work hard to maintain that.
    8. -
  8. 8. Work to fix all failing nightly regression tests on a reasonable schedule.
    9. -
+1. Set up **automated builds of the code** with high warning levels and eliminate all warnings. +2. **Select test harness frameworks** + 1. **Select a system-level test harness** for system-executable tests that report results appropriately (e.g., CTest/CDash). + 1. **Select a unit test harness** to effectively define and run finer-grained integration and unit tests (e.g., Google Test, pFUnit). + 1. **Customize or streamline** system-level and/or unit test frameworks for use in your particular project. +3. **Add system-level tests** to protect major user functionality. + 1. Select inputs for several important problem classes and run code to produce outputs. + 1. Set up no-change or verification tests with a system-level test harness in order to pin down important behavior. +4. **Add integration and unit tests** (as needed for adding/changing code) + 1. **Incorporate tests for uncovered code to be changed** using the ***Legacy Software Change Algorithm***[1] + * **Identify change points** for target change or new code. + * **Find test points** where code behavior can be sensed. + * **Break dependencies** in order to get the targeted code into the unit test harness. + * **Cover targeted code** to be changed with sufficient (characterization) tests. + 2. **Add new features or fix bugs**[1],[2],[3] + * **Add new tests** that define desired behavior (feature or bug). + * Run new tests and **verify they fail**. + * Add the minimal code to **get new tests to pass**. + * **Refactor** the covered code to clean up and remove duplication. + * **Review** all changes to existing code, new code and new tests. +5. Select **code coverage** (e.g., gcov/lcov) and **memory usage error detection** (e.g., valgrind, clang memory/address/leak sanitizers) analysis tools. +6. Define a set of **regression test suites**. + 1. Define a faster-running **pre-merge regression test suite** (e.g., single build with faster running tests) and **run it before every merge to the mainline branch**. + 1. Define a more comprehensive nightly **regression test suite** (e.g., builds and all tests on several platforms and compilers, code coverage, and memory usage error detection) and **run every night**. +7. Have a policy of **100% passing pre-merge regression tests** (run using a CI testing system like GitHub Actions, GitLab CI, or Jenkins) and work hard to maintain that. +8. Work to **fix all failing nightly regression tests** on a reasonable schedule. + ### FAQs: