Merge pull request #473 from OHDSI/develop

Develop

.Rbuildignore

@@ -1,11 +1,12 @@
 ^.*\.Rproj$
 ^\.Rproj\.user$
-^\.github$
-^LICENSE$
-_pkgdown.yml/*
-compare_versions
+extras
+^\.git
+man-roxygen
+_pkgdown.yml
 deploy.sh
-docs/*
-extras/*
-^doc$
-^Meta$
+compare_versions
+.github
+docs
+^LICENSE$
+^inst/doc/.*\.pdf$

DESCRIPTION

@@ -1,8 +1,8 @@
 Package: DataQualityDashboard
 Type: Package
 Title: Execute and View Data Quality Checks on OMOP CDM Database
-Version: 2.3.0
-Date: 2023-05-21
+Version: 2.4.0
+Date: 2023-07-26
 Authors@R: c(
   person("Katy", "Sadowski", email = "[email protected]", role = c("aut", "cre")),
   person("Clair", "Blacketer", role = c("aut")),
@@ -15,7 +15,9 @@ Authors@R: c(
 Maintainer: Katy Sadowski <[email protected]>
 Description: An R package for assessing data quality in standardized OMOP Common Data Model data sources.
 License: Apache License (>= 2)
+Config/build/clean-inst-doc: FALSE
 VignetteBuilder: knitr
+Config/testthat/edition: 3
 URL: https://github.com/OHDSI/DataQualityDashboard
 BugReports: https://github.com/OHDSI/DataQualityDashboard/issues
 Depends:

DataQualityDashboard.Rproj

@@ -13,6 +13,5 @@ RnwWeave: Sweave
 LaTeX: pdfLaTeX
 
 BuildType: Package
-PackageUseDevtools: Yes
 PackageInstallArgs: --no-multiarch --with-keep.source
-PackageRoxygenize: rd,collate,namespace,vignette
+PackageRoxygenize: rd,collate,namespace

NEWS.md

@@ -1,3 +1,23 @@
+DataQualityDashboard 2.4.0
+==========================
+This release includes:
+
+### Threshold file updates
+**The following changes involve updates to the default data quality check threshold files. If you are currently using an older version of DQD and update to v2.4.0, you may see changes in your DQD results. The failure threshold changes are fixes to incorrect thresholds in the v5.4 files and thus should result in more accurate, easier to interpret results. The unit concept ID changes ensure that long-invalid concepts will no longer be accepted as plausible measurement units.**
+
+- The incorrect failure thresholds for `measurePersonCompleteness` and `measureValueCompleteness` were fixed in the v5.4 table & field level threshold files.  This issue has existed since v5.4 support was initially added in March 2022
+  - Many `measurePersonCompleteness` checks had a threshold of 0 when it should have been 95 or 100
+  - Many `measureValueCompleteness` checks had a threshold of 100 when it should have been 0, and many had no threshold (defaulting to 0) when it should have been 100
+  - The thresholds have now been updated to match expectations for required/non-required tables/fields
+- In the v5.2, v5.3, and v5.4 table level threshold files, `measurePersonCompleteness` for the DEATH table has been toggled to `Yes`, with a threshold of 100
+- In the v5.2, v5.3, and v5.4 concept level threshold files, all references to unit concept 9117 in `plausibleUnitConceptIds` have been updated to 720870.  Concept 9117 became non-standard and was replaced with concept 720870, on 28-Mar-2022
+- In the v5.2, v5.3, and v5.4 concept level threshold files, all references to unit concepts 9258 and 9259 in `plausibleUnitConceptIds` have been removed. These concepts were deprecated on 05-May-2022
+
+### Bugfix
+- Call to new function `convertJsonResultsFileCase` in Shiny app was appended with `DataQualityDashboard::`. This prevents potential issues related to package loading and function naming conflicts
+
+Some minor refactoring of testthat files and package build configuration and some minor documentation updates were also added in this release.
+
 DataQualityDashboard 2.3.0
 ==========================
 This release includes:
@@ -78,6 +98,10 @@ This release includes:
 
   - The `outputFolder` parameter for the `executeDqChecks` function is now REQUIRED and no longer has a default value.  **This may be a breaking change for users who have not specified this parameter in their script to run DQD.**
 
+### Removal of measurement plausibility checks
+
+  - Most plausibleValueLow and plausibleValueHigh measurement values were removed from the concept check threshold files, due to feedback from the community that many of these ranges included plausible values and as such were causing unexpected check failures. An initiative is planned to reinterrogate these ranges and add them back once the team has higher confidence that they will only flag legitimately implausible values 
+
 ### Integrated testing was also added and the package was refactored on the backend
 
 DataQualityDashboard 1.4.1

R/convertResultsCase.R

@@ -15,7 +15,7 @@
 # limitations under the License.
 
 #' @title Convert JSON results file case
-#' 
+#'
 #' @description Convert a DQD JSON results file between camelcase and (all-caps) snakecase. Enables viewing of pre-v.2.1.0 results files in later DQD versions, and vice versa
 #'
 #' @param jsonFilePath  Path to the JSON results file to be converted

R/evaluateThresholds.R

@@ -195,7 +195,7 @@
       dplyr::filter(checkResults, .data$checkName == "measureValueCompleteness" &
         .data$numDenominatorRows == .data$numViolatedRows &
         is.na(.data$tableIsMissing) & is.na(.data$fieldIsMissing) & is.na(.data$tableIsEmpty)),
-      .data$cdmTableName, .data$cdmFieldName
+      "cdmTableName", "cdmFieldName"
     )
   if (nrow(emptyFields) > 0) {
     emptyFields$fieldIsEmpty <- 1

README.md

@@ -88,6 +88,16 @@ User Documentation
 ==================
 Documentation can be found on the [package website](https://ohdsi.github.io/DataQualityDashboard/index.html).
 
+PDF versions of the documentation are also available:
+
+* Vignette: [Add a New Data Quality Check](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/AddNewCheck.pdf)
+* Vignette: [Check Status Descriptions](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/CheckStatusDefinitions.pdf)
+* Vignette: [Data Quality Check Type Descriptions](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/CheckTypeDescriptions.pdf)
+* Vignette: [Running the DQD on a Cohort](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/DqdForCohorts.pdf)
+* Vignette: [Failure Thresholds and How to Change Them](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/Thresholds.pdf)
+* Vignette: [SqlOnly Mode](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/SqlOnly.pdf)
+* Package manual: [DataQualityDashboard manual](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/DataQualityDashboard.pdf) 
+
 Support
 =======