diff --git a/README.md b/README.md index 887ff00..04901ee 100644 --- a/README.md +++ b/README.md @@ -71,6 +71,8 @@ If Playwright fails in CI, we can still see what went wrong: - Inside the zipped artifact will be _another_ zip: `trace.zip`. - Don't unzip it! Instead, open it with [trace.playwright.dev](https://trace.playwright.dev/). +For more details, see the [TEST-PLAN](TEST-PLAN.md). + ### Conventions Branch names should be of the form `NNNN-short-description`, where `NNNN` is the issue number being addressed. diff --git a/TEST-PLAN.md b/TEST-PLAN.md new file mode 100644 index 0000000..854ab44 --- /dev/null +++ b/TEST-PLAN.md @@ -0,0 +1,109 @@ +# DP Creator II Test Plan + +## Introduction + +### Purpose + +Automate the end-to-end testing of the application to ensure it meets user requirements and maintains reliability, especially in the context of privacy-preserving data processing. + +### Project Overview + +The single user application will be installed with pip, and offer a web application (running on localhost) that guides the user through the application of differential privacy on a data file they provide. + +## Scope + +### In-Scope + +- Starting the application with and without `--demo` and other CLI flags. +- Form filling and navigation between tabs. +- Conditionally disabled controls (for example, controls late in the flow when inputs have not been provided, and controls early in the flow after the release has been made). +- Report export. +- Runnability of any generated code. + +### Out-of-Scope + +- Performance testing: Test timeouts may incidentally identify components which are slow ([issue](https://github.com/opendp/dp-creator-ii/issues/116)), but it's not the focus of automated testing. +- Rendering of preview charts: Might [add results table](https://github.com/opendp/dp-creator-ii/issues/122) to complement graphs, but we're not going to try to make any assertions against images. +- Testing exact values: The outputs is randomized, so we can not test for any particular value in the output. +- Design and usability. +- Correctness of DP calculations. + +## Testing Strategy + +### Test Objectives + +- Ensure reliable execution of user flows across all primary functionalities. +- Validate critical data inputs and outputs are consistent with user expectations. +- Verify UI responsiveness and behavior on supported devices and browsers. +- Conduct regression testing to maintain code stability with new updates. + +### Test Assumptions + +- OpenDP will correctly perform the required calculations. +- Users are able to successfully install the software. + +### Data Approach + +- Any CSVs used for testing will be checked in to the fixtures directory. +- Outputs will be checked for structure, but we will not expect any particular values. + +### Automation Strategy + +- Doc tests: For simple functions, use doctests so the developer can just work with one file. +- For more complex functions and operations, use pytest unit tests. +- Run linting and type checking tools as part of tests. +- Use Playwright for end-to-end tests. +- Use test matrix on Github CI if we think there will be any senstivity to versions or browsers. + +### Test Case Prioritization + +- There's just one test suite, which is run on every PR, so there's nothing that needs to be ordered. + +## Execution Strategy + +### Entry Criteria + +N/A: Tests are run automatically on every PR. + +### Exit criteria + +N/A: Tests are run automatically on every PR. + +### Validation and Defect Management + +N/A: PRs should not be merged with failing tests. + +## Github Workflow + +### Issue Submission and Prioritization + +[See README](https://github.com/opendp/dp-creator-ii/#conventions). + +### Pull Request Review and Testing + +- Reviewers should read and understand code. +- Reviewers are not expected to checkout or manually test code. + +### Continuous Integration and Deployment + +Github CI will run tests on each PR. We require tests to pass before merge. + +We do not require branches to be up to date with main: There is a small chance that one PR may break because of changes in a separate before, but that can be addressed after the fact. + +## Environment Requirements + +### Dataverse-Internal Environment + +N/A + +### AWS-Based Environments + +N/A + +## Significantly Impacted Division/College/Department + +N/A + +## Dependencies + +N/A