Merge pull request #682 from matthiaskoenig/develop

Changes for release 0.9.4

README.md

@@ -31,17 +31,6 @@ Important features are
 
 PK-DB is available at https://pk-db.com
 
-## License
-[[^]](https://github.com/matthiaskoenig/pkdb#pk-db---a-pharmacokinetics-database)
-PK-DB code and documentation is licensed as
-* Source Code: [LGPLv3](http://opensource.org/licenses/LGPL-3.0)
-* Documentation: [CC BY-SA 4.0](http://creativecommons.org/licenses/by-sa/4.0/)
-
-## Funding
-[[^]](https://github.com/matthiaskoenig/pkdb#pk-db---a-pharmacokinetics-database)
-Jan Grzegorzewski and Matthias König are supported by the Federal Ministry of Education and Research (BMBF, Germany)
-within the research network Systems Medicine of the Liver ([LiSyM](http://www.lisym.org/), grant number 031L0054).
-
 ## How to cite
 [[^]](https://github.com/matthiaskoenig/pkdb#pk-db---a-pharmacokinetics-database)
 If you use PK-DB data or the web interface cite
@@ -50,8 +39,24 @@ If you use PK-DB data or the web interface cite
 > Jan Grzegorzewski, Janosch Brandhorst, Dimitra Eleftheriadou, Kathleen Green, Matthias König   
 > bioRxiv 760884; doi: https://doi.org/10.1101/760884 
 
+> Grzegorzewski J, Brandhorst J, Green K, Eleftheriadou D, Duport Y, Barthorscht F, Köller A, Ke DYJ, De Angelis S, König M. 
+> *PK-DB: pharmacokinetics database for individualized and stratified computational modeling*. 
+> Nucleic Acids Res. 2020 Nov 5:gkaa990. doi: [10.1093/nar/gkaa990](https://doi.org/10.1093/nar/gkaa990). Epub ahead of print. PMID: [33151297](https://pubmed.ncbi.nlm.nih.gov/33151297/).
+
 If you use PK-DB code cite in addition 
 
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1406979.svg)](https://doi.org/10.5281/zenodo.1406979)
 
+## License
+[[^]](https://github.com/matthiaskoenig/pkdb#pk-db---a-pharmacokinetics-database)
+PK-DB code and documentation is licensed as
+* Source Code: [LGPLv3](http://opensource.org/licenses/LGPL-3.0)
+* Documentation: [CC BY-SA 4.0](http://creativecommons.org/licenses/by-sa/4.0/)
+
+## Funding
+[[^]](https://github.com/matthiaskoenig/pkdb#pk-db---a-pharmacokinetics-database)
+Jan Grzegorzewski and Matthias König are supported by the Federal Ministry of Education and Research (BMBF, Germany)
+within the research network Systems Medicine of the Liver ([LiSyM](http://www.lisym.org/), grant number 031L0054).
+
+
 © 2017-2020 Jan Grzegorzewski & Matthias König; https://livermetabolism.com.

backend/README.md

@@ -1,43 +1 @@
-# PKDB Backend (`django`)
-
-- [ ] documentation page with queries and searches
-
-## PKData Query
-The event cycle of PKData is:
-1. Query studies, interventions, groups, individuals, and outputs by adding
- the respective word as a prefix following two underscores to the url filter 
- (e.g. ...api/v1/pkdata/?studies__sid=PKDB00008 is equivalent to ...api/v1/studies/?sid=PKDB00008).
- The search/filter is performed on the indexed database. For more details on how to construct the query by patterns in the 
- url check "https://django-elasticsearch-dsl-drf.readthedocs.io/en/latest/".
-
-2. All tables are updated to get rid of redundant entries. This results in a concise set of entries
-in all tables (e.g. a filter on the study table for a specific sid reduces the entries of the other tables 
-only to interventions, groups, individuals, and outputs which are part of the study).
-
-3. paginated studies, interventions, groups, individuals, and outputs are returned. Getting the next page for one of the tables 
-works equivalently to the filters (e.g. getting the second studies page while searching for the interventions containing caffeine.  ...api/v1/pkdata/?interventions__substance=caffeine&studies__page=2).
-
-
-## PKDData
-documentation
-
-### Queries
-
-Query for single study:
-``` 
-http://localhost:8000/api/v1/pkdata/?studies__sid=PKDB00008 
-```
-Query for multiple studies based on sids:
-```
-http://localhost:8000/api/v1/pkdata/?studies__sid__in=PKDB00008__PKDB00001
-```
-Query for interventions substance:
-```
-http://localhost:8000/api/v1/pkdata/?interventions__substance=codeine
-```
-Query for interventions and outputs simultaneously:
-```
-http://localhost:8000/api/v1/pkdata/?interventions__substance=codeine&outputs__measurement_type=clearance
-```
-
-© 2017-2020 Jan Grzegorzewski & Matthias König.
+# PKDB

backend/download_extra/README.md

@@ -11,9 +11,6 @@ and
 * [How to cite](https://github.com/matthiaskoenig/pkdb#how-to-cite)
 * [License](https://github.com/matthiaskoenig/pkdb#license)
 * [Funding](https://github.com/matthiaskoenig/pkdb#funding)
-* [Installation](https://github.com/matthiaskoenig/pkdb#installation)
-* [REST API](https://github.com/matthiaskoenig/pkdb#rest-api)
-* [Docker interaction](https://github.com/matthiaskoenig/pkdb#docker-interaction)
 
 ## Overview
 [[^]](https://github.com/matthiaskoenig/pkdb#pk-db---a-pharmacokinetics-database)
@@ -34,155 +31,28 @@ Important features are
 
 PK-DB is available at https://pk-db.com
 
-## License
-[[^]](https://github.com/matthiaskoenig/pkdb#pk-db---a-pharmacokinetics-database)
-PK-DB code and documentation is licensed as
-* Source Code: [LGPLv3](http://opensource.org/licenses/LGPL-3.0)
-* Documentation: [CC BY-SA 4.0](http://creativecommons.org/licenses/by-sa/4.0/)
-
-## Funding
-[[^]](https://github.com/matthiaskoenig/pkdb#pk-db---a-pharmacokinetics-database)
-Jan Grzegorzewski and Matthias König are supported by the Federal Ministry of Education and Research (BMBF, Germany)
-within the research network Systems Medicine of the Liver ([LiSyM](http://www.lisym.org/), grant number 031L0054).
-
 ## How to cite
 [[^]](https://github.com/matthiaskoenig/pkdb#pk-db---a-pharmacokinetics-database)
 If you use PK-DB data or the web interface cite
 
-> *PK-DB: PharmacoKinetics DataBase for Individualized and Stratified Computational Modeling*  
-> Jan Grzegorzewski, Janosch Brandhorst, Dimitra Eleftheriadou, Kathleen Green, Matthias König   
-> bioRxiv 760884; doi: https://doi.org/10.1101/760884 
+> Grzegorzewski J, Brandhorst J, Green K, Eleftheriadou D, Duport Y, Barthorscht F, Köller A, Ke DYJ, De Angelis S, König M. 
+> *PK-DB: pharmacokinetics database for individualized and stratified computational modeling*. 
+> Nucleic Acids Res. 2020 Nov 5:gkaa990. doi: [10.1093/nar/gkaa990](https://doi.org/10.1093/nar/gkaa990). Epub ahead of print. PMID: [33151297](https://pubmed.ncbi.nlm.nih.gov/33151297/).
 
 If you use PK-DB code cite in addition 
 
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1406979.svg)](https://doi.org/10.5281/zenodo.1406979)
 
-## Installation
-[[^]](https://github.com/matthiaskoenig/pkdb#pk-db---a-pharmacokinetics-database)
-PK-DB is deployed via `docker` and `docker-compose`. 
-
-### Requirements
-To setup the development server 
-the following minimal requirements must be fulfilled
-- `docker`
-- `docker-compose`
-- `Python3.6`
-
-For elasticsearch the following system settings are required
-```
-sudo sysctl -w vm.max_map_count=262144
-```
-To set `vm.max_map_count` persistently change the value in 
-```
-/etc/sysctl.conf
-```
-### Start development server
-To start the local development server
-```bash
-# clone or pull the latest code
-git clone https://github.com/matthiaskoenig/pkdb.git
-cd pkdb
-git pull
-
-# set environment variables
-set -a && source .env.local
-
-# create/rebuild all docker containers
-./docker-purge.sh
-```
-This setups a clean database and clean volumes and starts the containers for `pkdb_backend`, `pkdb_frontend`, `elasticsearch` and `postgres`.
-You can check that all the containers are running via
-```bash
-docker container ls
-```
-which lists the current containers
-```
-CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                              NAMES
-bc7f9204468f        pkdb_backend          "bash -c '/usr/local…"   27 hours ago        Up 18 hours         0.0.0.0:8000->8000/tcp             pkdb_backend_1
-17b8d243e956        pkdb_frontend         "/bin/sh -c 'npm run…"   27 hours ago        Up 18 hours         0.0.0.0:8080->8080/tcp             pkdb_frontend_1
-7730c6fe2210        elasticsearch:6.8.1   "/usr/local/bin/dock…"   27 hours ago        Up 18 hours         9300/tcp, 0.0.0.0:9123->9200/tcp   pkdb_elasticsearch_1
-e880fbb0f349        postgres:11.4         "docker-entrypoint.s…"   27 hours ago        Up 18 hours         0.0.0.0:5433->5432/tcp             pkdb_postgres_1
-```
-The locally running develop version of PK-DB can now be accessed via the web browser from
-- frontend: http://localhost:8080
-- backend: http://localhost:8000
-
-### Fill database
-Due to copyright, licensing and privacy issues this repository does not contain any data.
-All data is managed via a separate private repository at https://github.com/matthiaskoenig/pkdb_data.
-This also includes the curation scripts and curation workflows.
-
-If you are interested in curating data or contributing data please contact us at https://livermetabolism.com.
-
-
-## REST API
+## License
 [[^]](https://github.com/matthiaskoenig/pkdb#pk-db---a-pharmacokinetics-database)
-PKDB provides a REST API which allows simple interaction with the database and easy access of data.
-An overview over the REST endpoints is provided at [`http://localhost:8000/api/v1/`](http://localhost:8000/api/v1/).
-
-### Query examples
-The REST API supports elastisearch queries, with syntax examples  
-available [here](https://django-elasticsearch-dsl-drf.readthedocs.io/en/latest/basic_usage_examples.html)
-* http://localhost:8000/api/v1/comments_elastic/?user_lastname=K%C3%B6nig
-* http://localhost:8000/api/v1/characteristica_elastic/?group_pk=5&final=true
-* http://localhost:8000/api/v1/characteristica_elastic/?search=group_name:female&final=true
-* http://localhost:8000/api/v1/substances_elastic/?search:name=cod
-* http://localhost:8000/api/v1/substances_elastic/?search=cod 
-* http://localhost:8000/api/v1/substances_elastic/?ids=1__2__3 
-* http://localhost:8000/api/v1/substances_elastic/?ids=1__2__3&ordering=-name
-* http://localhost:8000/api/v1/substances_elastic/?name=caffeine&name=acetaminophen
-
-### Suggestion example
-In addition suggestion queries are possible
-* http://localhost:8000/api/v1/substances_elastic/suggest/?search:name=cod
+PK-DB code and documentation is licensed as
+* Source Code: [LGPLv3](http://opensource.org/licenses/LGPL-3.0)
+* Documentation: [CC BY-SA 4.0](http://creativecommons.org/licenses/by-sa/4.0/)
 
-## Docker interaction
+## Funding
 [[^]](https://github.com/matthiaskoenig/pkdb#pk-db---a-pharmacokinetics-database)
-In the following typical examples to interact with the PK-DB docker containers are provided.
-
-### Check running containers
-To check the running containers use
-```bash
-watch docker container ls
-```
-
-### Interactive container mode
-```bash
-./docker-interactive.sh
-```
-
-### Container logs
-To get access to individual container logs use `docker container logs <container>`. For instance to check the
-django backend logs use
-```bash
-docker container logs pkdb_backend_1 
-```
-
-### Run command in container
-To run commands inside the docker container use
-```bash
-docker-compose run --rm backend [command]
-```
-or to run migrations
-```bash
-docker-compose run --rm backend python manage.py makemigrations
-```
-
-### Authentication data
-The following examples show how to dump and restore the authentication data.
-
-Dump authentication data
-```bash
-docker-compose -f $PKDB_DOCKER_COMPOSE_YAML run --rm backend ./manage.py  dumpdata auth  --indent 2 > ./backend/pkdb_app/fixtures/auth.json
-docker-compose -f $PKDB_DOCKER_COMPOSE_YAML run --rm backend ./manage.py  dumpdata users  --indent 2 > ./backend/pkdb_app/fixtures/users.json
-docker-compose -f $PKDB_DOCKER_COMPOSE_YAML run --rm backend ./manage.py  dumpdata rest_email_auth  --indent 2 > ./backend/pkdb_app/fixtures/rest_email_auth.json
-```
+Jan Grzegorzewski and Matthias König are supported by the Federal Ministry of Education and Research (BMBF, Germany)
+within the research network Systems Medicine of the Liver ([LiSyM](http://www.lisym.org/), grant number 031L0054).
 
-Restore authentication data
-```bash
-docker-compose -f $PKDB_DOCKER_COMPOSE_YAML run --rm backend ./manage.py  loaddata auth pkdb_app/fixtures/auth.json
-docker-compose -f $PKDB_DOCKER_COMPOSE_YAML run --rm backend ./manage.py  loaddata users pkdb_app/fixtures/users.json
-docker-compose -f $PKDB_DOCKER_COMPOSE_YAML run --rm backend ./manage.py  loaddata rest_email_auth pkdb_app/fixtures/rest_email_auth.json
-```
 
 &copy; 2017-2020 Jan Grzegorzewski & Matthias König; https://livermetabolism.com.

backend/pkdb_app/_version.py

@@ -1,4 +1,4 @@
 """
 Definition of version string.
 """
-__version__ = "0.9.3"
+__version__ = "0.9.4"

backend/pkdb_app/behaviours.py

@@ -54,7 +54,6 @@ def study_sid(self):
 def map_field(fields):
     return [f"{field}_map" for field in fields]
 
-VALUE_FIELDS_SAME_SCALE =  ["value", "mean", "median", "min", "max"]
 VALUE_FIELDS_SAME_SCALE =  ["value", "mean", "median", "min", "max"]
 VALUE_FIELDS_NO_UNIT = VALUE_FIELDS_SAME_SCALE + ["sd", "se", "cv"]
 VALUE_FIELDS = VALUE_FIELDS_NO_UNIT + ["unit"]

backend/pkdb_app/data/documents.py

@@ -1,6 +1,6 @@
 from django_elasticsearch_dsl import Document, fields, ObjectField
 from django_elasticsearch_dsl.registries import registry
-from pkdb_app.data.models import Dimension, SubSet
+from pkdb_app.data.models import Dimension, SubSet, Data
 
 from ..documents import string_field, elastic_settings, info_node, study_field
 
@@ -98,6 +98,40 @@ class SubSetDocument(Document):
     study = study_field
     study_sid = string_field('study_sid')
     study_name = string_field('study_name')
+    access = string_field('access')
+    allowed_users = fields.ObjectField(
+        attr="allowed_users",
+
+        properties={
+            'username': string_field("username")
+        },
+        multi=True
+    )
+    def get_queryset(self):
+        """Not mandatory but to improve performance we can select related in one sql request"""
+        return super(SubSetDocument, self).get_queryset().prefetch_related("data_points__outputs")
+    class Django:
+        model = SubSet
+        # Ignore auto updating of Elasticsearch when a model is saved/deleted
+        ignore_signals = True
+        # Don't perform an index refresh after every update
+        auto_refresh = False
+
+    class Index:
+        name = 'subset'
+        settings = elastic_settings
+        settings['number_of_shards'] = 5
+        settings['number_of_replicas'] = 1
+        settings['max_result_window'] = 100000
+
+
+'''
+@registry.register_document
+class TimeCourseDocument(Document):
+    study_sid = string_field('study_sid')
+    study_name = string_field('study_name')
+    outputs_pk = fields.ListField('timecourse')
+    
     # for permissions
     access = string_field('access')
     allowed_users = fields.ObjectField(
@@ -108,6 +142,11 @@ class SubSetDocument(Document):
         multi=True
     )
 
+    def get_queryset(self):
+        """Not mandatory but to improve performance we can select related in one sql request"""
+        return super(TimeCourseDocument, self).get_queryset().filter(data__data_type=Data.DataTypes.Timecourse)  # .prefetch_related("interventions").
+
+
     class Django:
         model = SubSet
         # Ignore auto updating of Elasticsearch when a model is saved/deleted
@@ -120,4 +159,6 @@ class Index:
         settings = elastic_settings
         settings['number_of_shards'] = 5
         settings['number_of_replicas'] = 1
-        settings['max_result_window'] = 100000
+        settings['max_result_window'] = 100000
+
+'''