Processing data tutorial (#104)

* adding perturbation response relationship tutorial

* addressed the changes to the notebook: added analysis to all graphs, typehinting and docstrings to methods, and other misc changes

* added new methods and analysis to enable comparison between different dataset combinations. Added everything under the last subtitle, with each section having smaller subheadings underneath to group everything

* Calculate variance explained (#88)

* fixed adjustment function so its based on enrichment strength

* added new file util.py and new test suite and updated notebook

* Update pyproject.toml

This is already in the dev dependencies. I forgot to go over that.

To add 'production' depdencies with python, you add to the default dependencies section with just:

```
poetry add <package>
```

You can also add dependencies to a group, eg:

```
poetry add --group dev <package>
```

See https://python-poetry.org/docs/cli/#options-4

That way, you can control what dependencies get installed.

For a typical user, I don't think we'll want to install jupyter in the environment. They should have jupyter in their environment, and then install yeastdnnexplorer into it.

* parameterizing the max_adjustment value and adding the calculate_variance_explained function and test suite

* removing the function and test suite for calculating the variance explained and adding the function to the visualizing_and_testing_data_generation_methods notebook

* Added docstrings and typehinting, removed unnecessary work and added exposition to graphs and methods

* updated notebook to use sphinx docstrings, added headings and subheadings and improved exposition

---------

Co-authored-by: Eric Jia <[email protected]>
Co-authored-by: Chase Mateusiak <[email protected]>

* fixed adjustment function so its based on enrichment strength (#86)

Co-authored-by: Eric Jia <[email protected]>

* Database Interface (#90)

* adding new file for explanation

* adding ParamsDict

* init implementation of the API classes. Documentation and some testing included. RankResponse is not, and the testing is minimal due to the difficulty of testing futures

* adding some words to the project ignore settings

* rank response api working

* addressing unused imports in RankResponseAPI

* updating the database_interface notebook for the new database backend; addressing logging warning on instantiation

* updating the tutorial to show how to use the aggregated data (#91)

* table data retrieved as gzip; addtiional columns now present from DB

* Update README.md

closes #81

* Adding update to manualqc (#96)

* removing new file, part of a demo

* adding update() method to bindingmanualqc; added _delimiter_detect method to AbstractRecords

* addressing pre-commit issues

* This is getting the dev branch rebased onto the main branch (#100)

* Calculate variance explained (#88)

* fixed adjustment function so its based on enrichment strength

* added new file util.py and new test suite and updated notebook

* Update pyproject.toml

This is already in the dev dependencies. I forgot to go over that.

To add 'production' depdencies with python, you add to the default dependencies section with just:

```
poetry add <package>
```

You can also add dependencies to a group, eg:

```
poetry add --group dev <package>
```

See https://python-poetry.org/docs/cli/#options-4

That way, you can control what dependencies get installed.

For a typical user, I don't think we'll want to install jupyter in the environment. They should have jupyter in their environment, and then install yeastdnnexplorer into it.

* parameterizing the max_adjustment value and adding the calculate_variance_explained function and test suite

* removing the function and test suite for calculating the variance explained and adding the function to the visualizing_and_testing_data_generation_methods notebook

* Added docstrings and typehinting, removed unnecessary work and added exposition to graphs and methods

* updated notebook to use sphinx docstrings, added headings and subheadings and improved exposition

---------

Co-authored-by: Eric Jia <[email protected]>
Co-authored-by: Chase Mateusiak <[email protected]>

* fixed adjustment function so its based on enrichment strength (#86)

Co-authored-by: Eric Jia <[email protected]>

* Database Interface (#90)

* adding new file for explanation

* adding ParamsDict

* init implementation of the API classes. Documentation and some testing included. RankResponse is not, and the testing is minimal due to the difficulty of testing futures

* adding some words to the project ignore settings

* rank response api working

* addressing unused imports in RankResponseAPI

* updating the database_interface notebook for the new database backend; addressing logging warning on instantiation

* updating the tutorial to show how to use the aggregated data (#91)

* table data retrieved as gzip; addtiional columns now present from DB

* Update README.md

closes #81

* Adding update to manualqc (#96)

* removing new file, part of a demo

* adding update() method to bindingmanualqc; added _delimiter_detect method to AbstractRecords

* addressing pre-commit issues

---------

Co-authored-by: ejiawustl <[email protected]>
Co-authored-by: Eric Jia <[email protected]>

* Add branch protection CI to prevent pulls directly to main (#101)

This should only allow pulls from a branch called `dev` or `patch` directly to main. otherwise, pull requests will be required to be against `dev`

* fixed adjustment function so its based on enrichment strength (#86)

Co-authored-by: Eric Jia <[email protected]>

* adding perturbation response relationship tutorial

* addressed the changes to the notebook: added analysis to all graphs, typehinting and docstrings to methods, and other misc changes

* added new methods and analysis to enable comparison between different dataset combinations. Added everything under the last subtitle, with each section having smaller subheadings underneath to group everything

* added docstring and typehinting to all methods, and added exposition to better explain the different conditions we use the model in. TODO: need to hide some of the output when training models or create an issue if I am unable to do so.

* Update exploring_perturbation_response_relationship notebook, still WIP

* adding notebook, new pyproject

* updated notebook: including a lot of new things based on the research work we have been doing for the last month. This notebook currently ends with a guide on creating the linear models on the cluster, but I can include more recent work involving the correlations and models we have been experimenting with

* went through all notebooks in vim and resolved merges by keeping the current changes

* adding statsmodels to pyproject

---------

Co-authored-by: Eric Jia <[email protected]>
Co-authored-by: Chase Mateusiak <[email protected]>
Co-authored-by: Chase Mateusiak <[email protected]>

.gitignore

@@ -1,8 +1,11 @@
+#mac files
+**/.DS_Store
+
 # Dataset directory
 data/
 
 # logs
-logs/
+**/logs/
 
 # local tmp files
 tmp/*

docs/tutorials/database_interface.ipynb

@@ -2142,7 +2142,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": ".venv",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -2156,9 +2156,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.9"
+   "version": "3.11.1"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }