[IBCDPE-688] Great Expectations Implementation for Metabolomics Data

.github/workflows/dev.yml

@@ -21,8 +21,9 @@ jobs:
         python-version:
           - "3.8"
           - "3.9"
-          - "3.10"
-          - "3.11"
+          # Support for Python 3.10 and 3.11 is temproarily disabled
+          # - "3.10"
+          # - "3.11"
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}

.gitignore

@@ -75,10 +75,6 @@ docs/_build/
 # PyBuilder
 target/
 
-# Jupyter Notebook
-.ipynb_checkpoints
-*.ipynb
-
 # IPython
 profile_default/
 ipython_config.py

CONTRIBUTING.md

@@ -139,6 +139,25 @@ This package has a `src/agoradatatools/etl/transform` submodule.  This folder ho
     - Use `pytest.mark.parameterize` to loop through multiple datasets in a single test.
     - The class `TestTransformGenesBiodomains` can be used as an example for future tests contibuted.
 
+### Great Expectations
+
+This package uses [Great Expectations](https://greatexpectations.io/) to validate output data.  The `great_expectations` folder houses our file system data context and Great Expectations-specific configuration files. Eventually, our goal is for each `agora-data-tools` dataset to be convered by an expectation suite. To add data validation for more datasets, follow these steps:
+
+1. Create a new expectation suite by defining the expectations for the new dataset in a Jupyter Notebook inside the `gx_suite_definitions` folder. Use `metabolomics.ipynb` as an example.
+1. Run the notebook to generate the new expectation suite. It should populate as a JSON file in the `great_expectations/expectations` folder.
+1. Add support for running Great Expectations on a dataset by adding the `gx_folder` key to the configuration for your datatset in both `test_config.yaml` and `config.yaml`. The `gx_folder` should be a Synapse ID pointing to the folder where we will upload generated HTML reports from Great Expectations. If a folder specific to your dataset does not yet exist in the proper locations ([Prod](https://www.synapse.org/#!Synapse:syn52948668), [Testing](https://www.synapse.org/#!Synapse:syn52948670)), create folders named after the dataset and copy the new folders' Synapse IDs to the config files.
+1. Test data processing by running `adt test_config.yaml` and ensure that HTML reports with all expectations are generated and uploaded to the proper folder in Synapse.
+
+#### Custom Expectations
+
+This repository is currently home to three custom expectations that were created for use on `agora-data-tools` datasets:
+
+1. `ExpectColumnValuesToHaveListLength`: checks to see if the lists in a particular column are the length that we expect.
+1. `ExpectColumnValuesToHaveListMembers`: checks to see if the lists in a particular column contain only values that we expect.
+1. `ExpectColumnValuesToHaveListMembersOfType`: checks to see if the lists in a particular column contain members of the type we expect.
+
+These expectations are defined in the `great_expectations/gx/plugins/expectations` folder. To add more custom expectations, follow the instructions provided in the Great Expectations [documentation](https://docs.greatexpectations.io/docs/guides/expectations/custom_expectations_lp).
+
 ### DockerHub
 
 Rather than using GitHub actions to build and push Docker images to DockerHub, the Docker images are automatically built in DockerHub. This requires the `sagebiodockerhub` GitHub user to be an Admin of this repo. You can view the docker build [here](https://hub.docker.com/r/sagebionetworks/agora-data-tools).

config.yaml

@@ -13,7 +13,7 @@
         column_rename:
           biodomain: name
         destination: *dest
-        
+
     - genes_biodomains:
         files:
           - name: genes_biodomains
@@ -102,6 +102,7 @@
         provenance:
           - syn26064497.1
         destination: *dest
+        gx_folder: syn52948669
 
     - gene_info:
         files:

great_expectations/gx/.gitignore

@@ -0,0 +1,3 @@
+
+uncommitted/
+.ge_store_backend_id

great_expectations/gx/checkpoints/metabolomics.yml

@@ -0,0 +1,30 @@
+name: metabolomics
+config_version: 1.0
+template_name:
+module_name: great_expectations.checkpoint
+class_name: Checkpoint
+run_name_template:
+expectation_suite_name:
+batch_request: {}
+action_list:
+  - name: store_validation_result
+    action:
+      class_name: StoreValidationResultAction
+  - name: store_evaluation_params
+    action:
+      class_name: StoreEvaluationParametersAction
+  - name: update_data_docs
+    action:
+      class_name: UpdateDataDocsAction
+evaluation_parameters: {}
+runtime_configuration: {}
+validations:
+  - batch_request:
+      datasource_name: default_pandas_datasource
+      data_asset_name: '#ephemeral_pandas_asset'
+      options: {}
+      batch_slice:
+    expectation_suite_name: metabolomics
+profilers: []
+ge_cloud_id:
+expectation_suite_ge_cloud_id: