From b4959682a0d4ed1cbc80bf39cbced4193c449a84 Mon Sep 17 00:00:00 2001
From: Gianna Jordan <61707471+GiaJordan@users.noreply.github.com>
Date: Wed, 5 Oct 2022 11:28:11 -0700
Subject: [PATCH 01/35] add text from meeting notes
---
README.md | 34 ++++++++++++++++++++++++++++++++++
1 file changed, 34 insertions(+)
diff --git a/README.md b/README.md
index 1b4c36b34..5a6ecf3f0 100644
--- a/README.md
+++ b/README.md
@@ -179,6 +179,40 @@ For new features, bugs, enhancements
*Note*: Make sure you have the latest version of the `develop` branch on your local machine.
+## Installation guide - Docker
+
+1. Install docker from https://www.docker.com/ .
+ Or, run `docker compose up` after cloning the schematic repo
+2. Identify docker container of interest from [Schematic DockerHub](https://hub.docker.com/r/sagebionetworks/schematic/tags)
+ Ex `docker pull sagebionetworks/schematic:latest` from the CLI
+3. Run Command
+ For REST API
+`docker run --rm -p 3001:3001 \
+ -v $(pwd):/schematic -w /schematic --name schematic \
+ -e SCHEMATIC_CONFIG=/schematic/schematic_config.yml \
+ -e GE_HOME=/usr/src/app/great_expectations/ \
+ sagebionetworks/schematic:commit-d833dd3 \
+ python /usr/src/app/run_api.py`
+
+For Schematic on mac/linux
+`docker run -v $(pwd):/schematic \
+ -v ~/Downloads/biospecimen-example.csv:/schematic/test.csv \
+ -w /schematic \
+ -e SCHEMATIC_CONFIG=/schematic/schematic_config.yml \
+ -e GE_HOME=/usr/src/app/great_expectations/ \
+ sagebionetworks/schematic model \
+ -c schematic_config.yml validate \
+ -mp test.csv -dt Biospecimen \
+ -js /schematic/data/example.model.jsonld`
+
+For schematic on Windows
+`docker run -v %cd%:/schematic \
+ -w /schematic \
+ -e GE_HOME=/usr/src/app/great_expectations/ \
+ sagebionetworks/schematic \
+ schematic model \
+ -c config.yml validate -mp tests/data/mock_manifests/inValid_Test_Manifest.csv -dt MockComponent -js /schematic/data/example.model.jsonld`
+
# Other Contribution Guidelines
## Updating readthedocs documentation
1. `cd docs`
From dbde3917a4860b186849a5bd5dc757f92486f716 Mon Sep 17 00:00:00 2001
From: afwillia
Date: Wed, 5 Oct 2022 14:33:47 -0700
Subject: [PATCH 02/35] Update the linux commands with a minimal working
example
---
README.md | 35 ++++++++++++++++++++++-------------
1 file changed, 22 insertions(+), 13 deletions(-)
diff --git a/README.md b/README.md
index 5a6ecf3f0..50441e836 100644
--- a/README.md
+++ b/README.md
@@ -182,36 +182,45 @@ For new features, bugs, enhancements
## Installation guide - Docker
1. Install docker from https://www.docker.com/ .
- Or, run `docker compose up` after cloning the schematic repo
2. Identify docker container of interest from [Schematic DockerHub](https://hub.docker.com/r/sagebionetworks/schematic/tags)
Ex `docker pull sagebionetworks/schematic:latest` from the CLI
-3. Run Command
+ Or, run `docker compose up` after cloning the schematic github repo
+3. Run Schematic Command with `docker run`.
For REST API
-`docker run --rm -p 3001:3001 \
+```
+docker run --rm -p 3001:3001 \
-v $(pwd):/schematic -w /schematic --name schematic \
-e SCHEMATIC_CONFIG=/schematic/schematic_config.yml \
-e GE_HOME=/usr/src/app/great_expectations/ \
sagebionetworks/schematic:commit-d833dd3 \
- python /usr/src/app/run_api.py`
+ python /usr/src/app/run_api.py
+```
For Schematic on mac/linux
-`docker run -v $(pwd):/schematic \
- -v ~/Downloads/biospecimen-example.csv:/schematic/test.csv \
+To run example below, first clone schematic into your home directory `git clone https://github.com/sage-bionetworks/schematic ~/schematic`
+Then update .synapseConfig with their credentials
+```
+docker run \
+ -v ~/schematic:/schematic \
-w /schematic \
- -e SCHEMATIC_CONFIG=/schematic/schematic_config.yml \
+ -e SCHEMATIC_CONFIG=/schematic/config.yml \
-e GE_HOME=/usr/src/app/great_expectations/ \
- sagebionetworks/schematic model \
- -c schematic_config.yml validate \
- -mp test.csv -dt Biospecimen \
- -js /schematic/data/example.model.jsonld`
+ aws_dca_schematic_deploy-schematic schematic model \
+ -c /schematic/config.yml validate \
+ -mp /schematic/tests/data/mock_manifests/Valid_Test_Manifest.csv \
+ -dt MockComponent \
+ -js /schematic/tests/data/example.model.jsonld
+```
For schematic on Windows
-`docker run -v %cd%:/schematic \
+```
+docker run -v %cd%:/schematic \
-w /schematic \
-e GE_HOME=/usr/src/app/great_expectations/ \
sagebionetworks/schematic \
schematic model \
- -c config.yml validate -mp tests/data/mock_manifests/inValid_Test_Manifest.csv -dt MockComponent -js /schematic/data/example.model.jsonld`
+ -c config.yml validate -mp tests/data/mock_manifests/inValid_Test_Manifest.csv -dt MockComponent -js /schematic/data/example.model.jsonld
+```
# Other Contribution Guidelines
## Updating readthedocs documentation
From 6dcba5476eaf16e9a6c6ec1556ac50e001b64122 Mon Sep 17 00:00:00 2001
From: afwillia
Date: Wed, 5 Oct 2022 14:35:32 -0700
Subject: [PATCH 03/35] Formatting tweaks to the docker section of README
---
README.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/README.md b/README.md
index 50441e836..c3518c4b7 100644
--- a/README.md
+++ b/README.md
@@ -185,7 +185,7 @@ For new features, bugs, enhancements
2. Identify docker container of interest from [Schematic DockerHub](https://hub.docker.com/r/sagebionetworks/schematic/tags)
Ex `docker pull sagebionetworks/schematic:latest` from the CLI
Or, run `docker compose up` after cloning the schematic github repo
-3. Run Schematic Command with `docker run`.
+3. Run Schematic Command with `docker run`.
For REST API
```
docker run --rm -p 3001:3001 \
@@ -197,8 +197,8 @@ docker run --rm -p 3001:3001 \
```
For Schematic on mac/linux
-To run example below, first clone schematic into your home directory `git clone https://github.com/sage-bionetworks/schematic ~/schematic`
-Then update .synapseConfig with their credentials
+To run example below, first clone schematic into your home directory `git clone https://github.com/sage-bionetworks/schematic ~/schematic`
+Then update .synapseConfig with your credentials
```
docker run \
-v ~/schematic:/schematic \
From 3edddca347917419c3f610e778a835382e98bd40 Mon Sep 17 00:00:00 2001
From: Gianna Jordan <61707471+GiaJordan@users.noreply.github.com>
Date: Thu, 6 Oct 2022 10:44:54 -0700
Subject: [PATCH 04/35] formatting changes
---
README.md | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/README.md b/README.md
index c3518c4b7..5622ce051 100644
--- a/README.md
+++ b/README.md
@@ -182,11 +182,11 @@ For new features, bugs, enhancements
## Installation guide - Docker
1. Install docker from https://www.docker.com/ .
-2. Identify docker container of interest from [Schematic DockerHub](https://hub.docker.com/r/sagebionetworks/schematic/tags)
- Ex `docker pull sagebionetworks/schematic:latest` from the CLI
+2. Identify docker container of interest from [Schematic DockerHub](https://hub.docker.com/r/sagebionetworks/schematic/tags)
+ Ex `docker pull sagebionetworks/schematic:latest` from the CLI
Or, run `docker compose up` after cloning the schematic github repo
3. Run Schematic Command with `docker run`.
- For REST API
+### Example For REST API
```
docker run --rm -p 3001:3001 \
-v $(pwd):/schematic -w /schematic --name schematic \
@@ -196,7 +196,7 @@ docker run --rm -p 3001:3001 \
python /usr/src/app/run_api.py
```
-For Schematic on mac/linux
+### Example For Schematic on mac/linux
To run example below, first clone schematic into your home directory `git clone https://github.com/sage-bionetworks/schematic ~/schematic`
Then update .synapseConfig with your credentials
```
@@ -212,7 +212,7 @@ docker run \
-js /schematic/tests/data/example.model.jsonld
```
-For schematic on Windows
+### Example For schematic on Windows
```
docker run -v %cd%:/schematic \
-w /schematic \
From 820c7589a03a2d7d0d8dc83708e02367c7e95302 Mon Sep 17 00:00:00 2001
From: Gianna Jordan <61707471+GiaJordan@users.noreply.github.com>
Date: Thu, 6 Oct 2022 11:26:55 -0700
Subject: [PATCH 05/35] add info
---
README.md | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/README.md b/README.md
index 5622ce051..c56d1f5a7 100644
--- a/README.md
+++ b/README.md
@@ -179,20 +179,22 @@ For new features, bugs, enhancements
*Note*: Make sure you have the latest version of the `develop` branch on your local machine.
-## Installation guide - Docker
+## Installation Guide - Docker
1. Install docker from https://www.docker.com/ .
2. Identify docker container of interest from [Schematic DockerHub](https://hub.docker.com/r/sagebionetworks/schematic/tags)
- Ex `docker pull sagebionetworks/schematic:latest` from the CLI
- Or, run `docker compose up` after cloning the schematic github repo
-3. Run Schematic Command with `docker run`.
+ Ex `docker pull sagebionetworks/schematic:latest` from the CLI or, run `docker compose up` after cloning the schematic github repo
+3. Run Schematic Command with `docker run `.
+ - For more information on flags for `docker run` and what they do, visit the [Docker Documentation](https://docs.docker.com/engine/reference/commandline/run/)
+ - These example commands assume that you have navigated to the directory you want to run schematic from. To specify your working directory, use `$(pwd)` on MacOS/Linux or `%cd%` on Windows.
+ - If not using the latest image, then the full name should be specified: ie `sagebionetworks/schematic:commit-e611e4a`
### Example For REST API
```
docker run --rm -p 3001:3001 \
-v $(pwd):/schematic -w /schematic --name schematic \
-e SCHEMATIC_CONFIG=/schematic/schematic_config.yml \
-e GE_HOME=/usr/src/app/great_expectations/ \
- sagebionetworks/schematic:commit-d833dd3 \
+ sagebionetworks/schematic \
python /usr/src/app/run_api.py
```
@@ -212,7 +214,7 @@ docker run \
-js /schematic/tests/data/example.model.jsonld
```
-### Example For schematic on Windows
+### Example For Schematic on Windows
```
docker run -v %cd%:/schematic \
-w /schematic \
From 358a62bf2f927baeddd18e3ecc82a79e3cb97695 Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 10 Oct 2022 12:35:22 -0400
Subject: [PATCH 06/35] add pdoc
---
.github/workflows/pdoc.yml | 69 ++++++++++++++++++++++++++++++++++++++
poetry.lock | 54 ++++++++++++++++++++++++++++-
pyproject.toml | 1 +
3 files changed, 123 insertions(+), 1 deletion(-)
create mode 100644 .github/workflows/pdoc.yml
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
new file mode 100644
index 000000000..0ef5cfbe6
--- /dev/null
+++ b/.github/workflows/pdoc.yml
@@ -0,0 +1,69 @@
+-name: Test schematic
+
+name: pdoc
+
+# build the documentation whenever there are new commits on main
+on:
+ push:
+ branches:
+ - develop
+ workflow_dispatch: # Allow manually triggering the workflow
+
+# security: restrict permissions for CI jobs.
+permissions:
+ contents: read
+
+concurrency:
+ # cancel the current running workflow from the same branch, PR when a new workflow is triggered
+ # when the trigger is not a PR but a push, it will use the commit sha to generate the concurrency group
+ # {{ github.workflow }}: the workflow name is used to generate the concurrency group. This allows you to have more than one workflows
+ # {{ github.ref_type }}: the type of Git ref object created in the repository. Can be either branch or tag
+ # {{ github.event.pull_request.number}}: get PR number
+ # {{ github.sha }}: full commit sha
+ # credit: https://github.com/Sage-Bionetworks-Workflows/sagetasks/blob/main/.github/workflows/ci.yml
+ group: >-
+ ${{ github.workflow }}-${{ github.ref_type }}-
+ ${{ github.event.pull_request.number || github.sha }}
+ cancel-in-progress: true
+jobs:
+ test:
+
+ runson: ubuntu-latest
+ env:
+ POETRY_VERSION: 1.2.0
+ strategy:
+ matrix:
+ python-version: ["3.7", "3.8", "3.9", "3.10"]
+
+ steps:
+ # install poetry
+ - name: Install Poetry Action
+ uses: snok/install-poetry@v1.3.2
+ virtualenvs-create: true
+ virtualenvs-in-project: true
+
+ # install dependencies
+ - run: poetry install --no-interaction --no-root
+ if: steps.cache-deps.outputs.cache-hit != 'true'
+
+ # create documentation
+ - run: pdoc --docformat google -o docs/api api/routes.py
+
+ - uses: actions/upload-pages-artifact@v1
+ with:
+ path: docs/api
+
+ # Deploy the artifact to GitHub pages.
+ # This is a separate job so that only actions/deploy-pages has the necessary permissions.
+ deploy:
+ needs: build
+ runs-on: ubuntu-latest
+ permissions:
+ pages: write
+ id-token: write
+ environment:
+ name: github-pages
+ url: ${{ steps.deployment.outputs.page_url }}
+ steps:
+ - id: deployment
+ uses: actions/deploy-pages@v1
\ No newline at end of file
diff --git a/poetry.lock b/poetry.lock
index 6b1104156..569fae7ed 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -73,6 +73,18 @@ cffi = ">=1.0.1"
dev = ["cogapp", "pre-commit", "pytest", "wheel"]
tests = ["pytest"]
+[[package]]
+name = "astunparse"
+version = "1.6.3"
+description = "An AST unparser for Python"
+category = "main"
+optional = false
+python-versions = "*"
+
+[package.dependencies]
+six = ">=1.6.1,<2.0"
+wheel = ">=0.23.0,<1.0"
+
[[package]]
name = "atomicwrites"
version = "1.4.1"
@@ -1160,6 +1172,23 @@ category = "dev"
optional = false
python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7"
+[[package]]
+name = "pdoc"
+version = "12.2.0"
+description = "API Documentation for Python Projects"
+category = "main"
+optional = false
+python-versions = ">=3.7"
+
+[package.dependencies]
+astunparse = {version = "*", markers = "python_version < \"3.9\""}
+Jinja2 = ">=2.11.0"
+MarkupSafe = "*"
+pygments = ">=2.12.0"
+
+[package.extras]
+dev = ["flake8", "hypothesis", "mypy", "pytest", "pytest-cov", "pytest-timeout", "tox"]
+
[[package]]
name = "pexpect"
version = "4.8.0"
@@ -1970,6 +1999,17 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
dev = ["coverage", "pallets-sphinx-themes", "pytest", "pytest-timeout", "sphinx", "sphinx-issues", "tox"]
watchdog = ["watchdog"]
+[[package]]
+name = "wheel"
+version = "0.37.1"
+description = "A built-package format for Python"
+category = "main"
+optional = false
+python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7"
+
+[package.extras]
+test = ["pytest (>=3.0.0)", "pytest-cov"]
+
[[package]]
name = "widgetsnbextension"
version = "4.0.2"
@@ -2001,7 +2041,7 @@ testing = ["func-timeout", "jaraco-itertools", "pytest (>=6)", "pytest-black (>=
[metadata]
lock-version = "1.1"
python-versions = ">=3.7.1,<3.11"
-content-hash = "77f2068fe6df8f020fe207e2f6ce49d8850e3b507d11288f7f7e0bdc69dc0c0d"
+content-hash = "4546028400d18ed2c6b6de3057a5537db284b8d2e9263b320f1f823280a900b1"
[metadata.files]
alabaster = [
@@ -2047,6 +2087,10 @@ argon2-cffi-bindings = [
{file = "argon2_cffi_bindings-21.2.0-pp38-pypy38_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ed2937d286e2ad0cc79a7087d3c272832865f779430e0cc2b4f3718d3159b0cb"},
{file = "argon2_cffi_bindings-21.2.0-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:5e00316dabdaea0b2dd82d141cc66889ced0cdcbfa599e8b471cf22c620c329a"},
]
+astunparse = [
+ {file = "astunparse-1.6.3-py2.py3-none-any.whl", hash = "sha256:c2652417f2c8b5bb325c885ae329bdf3f86424075c4fd1a128674bc6fba4b8e8"},
+ {file = "astunparse-1.6.3.tar.gz", hash = "sha256:5ad93a8456f0d084c3456d059fd9a92cce667963232cbf763eac3bc5b7940872"},
+]
atomicwrites = [
{file = "atomicwrites-1.4.1.tar.gz", hash = "sha256:81b2c9071a49367a7f770170e5eec8cb66567cfbbc8c73d20ce5ca4a8d71cf11"},
]
@@ -2643,6 +2687,10 @@ pathspec = [
{file = "pathspec-0.9.0-py2.py3-none-any.whl", hash = "sha256:7d15c4ddb0b5c802d161efc417ec1a2558ea2653c2e8ad9c19098201dc1c993a"},
{file = "pathspec-0.9.0.tar.gz", hash = "sha256:e564499435a2673d586f6b2130bb5b95f04a3ba06f81b8f895b651a3c76aabb1"},
]
+pdoc = [
+ {file = "pdoc-12.2.0-py3-none-any.whl", hash = "sha256:043ab59983ea166ba15c9950eca3683194e5fbc7bbcdd61595f95d18fdc5c05c"},
+ {file = "pdoc-12.2.0.tar.gz", hash = "sha256:1a4f2ca3f02772941b7b7fe85cb50b9b0c86ed4c2417bcf7d0bd7ad189ae1ba8"},
+]
pexpect = [
{file = "pexpect-4.8.0-py2.py3-none-any.whl", hash = "sha256:0b48a55dcb3c05f3329815901ea4fc1537514d6ba867a152b581d69ae3710937"},
{file = "pexpect-4.8.0.tar.gz", hash = "sha256:fc65a43959d153d0114afe13997d439c22823a27cefceb5ff35c2178c6784c0c"},
@@ -3280,6 +3328,10 @@ werkzeug = [
{file = "Werkzeug-1.0.1-py2.py3-none-any.whl", hash = "sha256:2de2a5db0baeae7b2d2664949077c2ac63fbd16d98da0ff71837f7d1dea3fd43"},
{file = "Werkzeug-1.0.1.tar.gz", hash = "sha256:6c80b1e5ad3665290ea39320b91e1be1e0d5f60652b964a3070216de83d2e47c"},
]
+wheel = [
+ {file = "wheel-0.37.1-py2.py3-none-any.whl", hash = "sha256:4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a"},
+ {file = "wheel-0.37.1.tar.gz", hash = "sha256:e9a504e793efbca1b8e0e9cb979a249cf4a0a7b5b8c9e8b65a5e39d49529c1c4"},
+]
widgetsnbextension = [
{file = "widgetsnbextension-4.0.2-py3-none-any.whl", hash = "sha256:966bd61443926b6adcc0abef9f499c48bdeda181c333b0f49842d7385d440579"},
{file = "widgetsnbextension-4.0.2.tar.gz", hash = "sha256:07f0e8582f920b24316cef16490f1aeb498f2c875d48980540e5c5dbf0ff5e2d"},
diff --git a/pyproject.toml b/pyproject.toml
index 4cf0a526d..d44157406 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -66,6 +66,7 @@ Jinja2 = "2.11.3"
openpyxl = "^3.0.9"
"backports.zoneinfo" = {markers = "python_version < \"3.9\"", version = "^0.2.1"}
Flask-Cors = "^3.0.10"
+pdoc = "^12.2.0"
[tool.poetry.dev-dependencies]
From 9822f45879120a48e00f0a2458d5b38e66332129 Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 10 Oct 2022 12:38:08 -0400
Subject: [PATCH 07/35] add changes for test
---
.github/workflows/pdoc.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index 0ef5cfbe6..3c368d112 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -6,7 +6,7 @@ name: pdoc
on:
push:
branches:
- - develop
+ - develop-pydoc
workflow_dispatch: # Allow manually triggering the workflow
# security: restrict permissions for CI jobs.
From 776a17fece31d14448b1ee4d6f2742d4f35c81f9 Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 10 Oct 2022 12:38:46 -0400
Subject: [PATCH 08/35] clean up old code
---
.github/workflows/pdoc.yml | 2 --
1 file changed, 2 deletions(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index 3c368d112..573d266c4 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -1,5 +1,3 @@
--name: Test schematic
-
name: pdoc
# build the documentation whenever there are new commits on main
From 498caa65d55cbd8b944f53602f3e37365a64b1d6 Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 10 Oct 2022 12:40:25 -0400
Subject: [PATCH 09/35] fix syntax
---
.github/workflows/pdoc.yml | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index 573d266c4..538085147 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -37,8 +37,9 @@ jobs:
# install poetry
- name: Install Poetry Action
uses: snok/install-poetry@v1.3.2
- virtualenvs-create: true
- virtualenvs-in-project: true
+ with:
+ virtualenvs-create: true
+ virtualenvs-in-project: true
# install dependencies
- run: poetry install --no-interaction --no-root
From f2fa293a4ec7b1dc67c288db469a3116f0618d10 Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 10 Oct 2022 12:42:21 -0400
Subject: [PATCH 10/35] fix poetry install
---
.github/workflows/pdoc.yml | 45 ++++++++++++++++++++++++++++++--------
1 file changed, 36 insertions(+), 9 deletions(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index 538085147..1d941117d 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -34,16 +34,43 @@ jobs:
python-version: ["3.7", "3.8", "3.9", "3.10"]
steps:
- # install poetry
- - name: Install Poetry Action
- uses: snok/install-poetry@v1.3.2
- with:
- virtualenvs-create: true
- virtualenvs-in-project: true
+ #----------------------------------------------
+ # check-out repo and set-up python
+ #----------------------------------------------
+ - name: Check out repository
+ uses: actions/checkout@v2
+
+ - name: Set up Python ${{ matrix.python-version }}
+ uses: actions/setup-python@v2
+ with:
+ python-version: ${{ matrix.python-version }}
- # install dependencies
- - run: poetry install --no-interaction --no-root
- if: steps.cache-deps.outputs.cache-hit != 'true'
+ #----------------------------------------------
+ # install & configure poetry
+ #----------------------------------------------
+ - name: Install Poetry
+ run: |
+ curl -sSL https://install.python-poetry.org \
+ | python3 - --version ${{ env.POETRY_VERSION }};
+ poetry config virtualenvs.create true;
+ poetry config virtualenvs.in-project true;
+
+ #----------------------------------------------
+ # load cached venv if cache exists
+ #----------------------------------------------
+ - name: Load cached venv
+ id: cached-poetry-dependencies
+ uses: actions/cache@v2
+ with:
+ path: .venv
+ key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
+
+ #----------------------------------------------
+ # install dependencies if cache does not exist
+ #----------------------------------------------
+ - name: Install dependencies
+ if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+ run: poetry install --no-interaction --no-root
# create documentation
- run: pdoc --docformat google -o docs/api api/routes.py
From 99dfcf4696b9738f7645159c5550413b19fe6a7b Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 10 Oct 2022 12:43:30 -0400
Subject: [PATCH 11/35] fix syntax
---
.github/workflows/pdoc.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index 1d941117d..8468b0a1c 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -26,7 +26,7 @@ concurrency:
jobs:
test:
- runson: ubuntu-latest
+ runs-on: ubuntu-latest
env:
POETRY_VERSION: 1.2.0
strategy:
From a2a4347e24c5e292d640c2e9489c7cf20efdc716 Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 10 Oct 2022 12:46:06 -0400
Subject: [PATCH 12/35] change job to build
---
.github/workflows/pdoc.yml | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index 8468b0a1c..cf6b412a9 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -24,8 +24,7 @@ concurrency:
${{ github.event.pull_request.number || github.sha }}
cancel-in-progress: true
jobs:
- test:
-
+ build:
runs-on: ubuntu-latest
env:
POETRY_VERSION: 1.2.0
From 7c682b90d6c6b351edf3264e5c8319342d70dd0d Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 10 Oct 2022 12:57:28 -0400
Subject: [PATCH 13/35] update version number from v2 to v3; show pydoc version
being using
---
.github/workflows/pdoc.yml | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index cf6b412a9..a87109e56 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -37,10 +37,10 @@ jobs:
# check-out repo and set-up python
#----------------------------------------------
- name: Check out repository
- uses: actions/checkout@v2
+ uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
- uses: actions/setup-python@v2
+ uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
@@ -71,8 +71,9 @@ jobs:
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
run: poetry install --no-interaction --no-root
- # create documentation
- - run: pdoc --docformat google -o docs/api api/routes.py
+ # create documentation
+ - run: poetry show pdoc
+ - run: poetry run pdoc --docformat google -o docs/api api/routes.py
- uses: actions/upload-pages-artifact@v1
with:
From 9602faa3a7a23c98d0eeaa7f4edbfce0b2496ea6 Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 10 Oct 2022 13:09:32 -0400
Subject: [PATCH 14/35] only use higher python version to avoid typing issue
---
.github/workflows/pdoc.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index a87109e56..8561dcf64 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -30,7 +30,7 @@ jobs:
POETRY_VERSION: 1.2.0
strategy:
matrix:
- python-version: ["3.7", "3.8", "3.9", "3.10"]
+ python-version: ["3.9", "3.10"]
steps:
#----------------------------------------------
From 69305ec0916b53667c995d937403f98e916ef8e3 Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 10 Oct 2022 13:22:58 -0400
Subject: [PATCH 15/35] change deployment branch back to develop
---
.github/workflows/pdoc.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index 8561dcf64..d5e322656 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -4,7 +4,7 @@ name: pdoc
on:
push:
branches:
- - develop-pydoc
+ - develop
workflow_dispatch: # Allow manually triggering the workflow
# security: restrict permissions for CI jobs.
From 8a167b208648b53926d56dfe1c4beacf4f41a184 Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 17 Oct 2022 14:36:10 -0700
Subject: [PATCH 16/35] save temporary
---
api/__init__.py | 4 ++++
docker-compose.yml | 1 +
schematic/configuration.py | 17 ++++++++++++++++-
3 files changed, 21 insertions(+), 1 deletion(-)
diff --git a/api/__init__.py b/api/__init__.py
index 07c1c5c3b..441b2bbf4 100644
--- a/api/__init__.py
+++ b/api/__init__.py
@@ -15,7 +15,11 @@ def create_app():
# path to config.yml file saved as a Flask config variable
default_config = os.path.abspath(os.path.join(__file__, "../../config.yml"))
schematic_config = os.environ.get("SCHEMATIC_CONFIG", default_config)
+
app.config["SCHEMATIC_CONFIG"] = schematic_config
+ # schematic_config_content = os.environ.get("SCHEMATIC_CONFIG_CONTENT")
+ # print('schematic_config_content', schematic_config_content)
+ # app.config["SCHEMATIC_CONFIG"] = schematic_config_content
# Configure flask app
# app.config[] = schematic[]
diff --git a/docker-compose.yml b/docker-compose.yml
index 51aefdeb6..accefcea6 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -13,4 +13,5 @@ services:
APP_HOST: "0.0.0.0"
APP_PORT: "3001"
SCHEMATIC_CONFIG: /schematic/config.yml
+ #SCHEMATIC_CONFIG_CONTENT: "${SCHEMATIC_CONFIG_CONTENT}"
GE_HOME: /usr/src/app/great_expectations/
diff --git a/schematic/configuration.py b/schematic/configuration.py
index 4ba69b32e..534aafbfa 100644
--- a/schematic/configuration.py
+++ b/schematic/configuration.py
@@ -34,6 +34,14 @@ def get(self, key, default):
value = default
return value
+ def load_config_content(str_yaml: str) -> dict:
+ try:
+ config_data = yaml.safe_load(str_yaml)
+ except yaml.YAMLError as exc:
+ print(exc)
+ return None
+ return config_data
+
@staticmethod
def load_yaml(file_path: str) -> dict:
with open(file_path, "r") as stream:
@@ -61,7 +69,7 @@ def load_config_from_env(self):
)
return self.load_config(schematic_config)
- def load_config(self, config_path=None, asset_view=None):
+ def load_config(self, config_path=None, asset_view=None):
# If config_path is None, try loading from environment
if config_path is None and "SCHEMATIC_CONFIG" in os.environ:
return self.load_config_from_env()
@@ -76,6 +84,13 @@ def load_config(self, config_path=None, asset_view=None):
config_path = os.path.expanduser(config_path)
config_path = os.path.abspath(config_path)
self.DATA = self.load_yaml(config_path)
+
+ # test
+ if "SCHEMATIC_CONFIG_CONTENT" in os.environ:
+ config_content = self.load_config_content(os.environ["SCHEMATIC_CONFIG_CONTENT"])
+ self.DATA = config_content
+
+
self.CONFIG_PATH = config_path
# handle user input (for API endpoints)
if asset_view:
From 95e3a4e7b41a17ba949e51367ae218d05b7fc7b5 Mon Sep 17 00:00:00 2001
From: linglp
Date: Wed, 19 Oct 2022 20:14:24 -0700
Subject: [PATCH 17/35] add way to pass config content as an env variable
---
api/__init__.py | 5 ++---
api/routes.py | 7 ++++++-
run_api.py | 2 +-
schematic/configuration.py | 39 +++++++++++++++++++++++++++++---------
4 files changed, 39 insertions(+), 14 deletions(-)
diff --git a/api/__init__.py b/api/__init__.py
index 441b2bbf4..c823b0517 100644
--- a/api/__init__.py
+++ b/api/__init__.py
@@ -17,9 +17,8 @@ def create_app():
schematic_config = os.environ.get("SCHEMATIC_CONFIG", default_config)
app.config["SCHEMATIC_CONFIG"] = schematic_config
- # schematic_config_content = os.environ.get("SCHEMATIC_CONFIG_CONTENT")
- # print('schematic_config_content', schematic_config_content)
- # app.config["SCHEMATIC_CONFIG"] = schematic_config_content
+ schematic_config_content = os.environ.get("SCHEMATIC_CONFIG_CONTENT")
+ app.config["SCHEMATIC_CONFIG_CONTENT"] = schematic_config_content
# Configure flask app
# app.config[] = schematic[]
diff --git a/api/routes.py b/api/routes.py
index 11f0639e6..6066664a7 100644
--- a/api/routes.py
+++ b/api/routes.py
@@ -37,7 +37,12 @@
def config_handler(asset_view=None):
path_to_config = app.config["SCHEMATIC_CONFIG"]
- # check if file exists at the path created, i.e., app.config['SCHEMATIC_CONFIG']
+ # if content of the config file is provided:
+ content_of_config = app.config["SCHEMATIC_CONFIG_CONTENT"]
+ if len(content_of_config) > 0:
+ CONFIG.load_config_content_from_env()
+
+ # check if path to config is provided
if os.path.isfile(path_to_config):
CONFIG.load_config(path_to_config, asset_view = asset_view)
diff --git a/run_api.py b/run_api.py
index cf278da9c..f2a24a192 100755
--- a/run_api.py
+++ b/run_api.py
@@ -15,4 +15,4 @@
# Launch app
app = create_app()
CORS(app, resources={r"*": {"origins": "*"}})
- app.run(port=3001, debug=True)
+ app.run(host=host, port=port, debug=True)
diff --git a/schematic/configuration.py b/schematic/configuration.py
index 534aafbfa..3a8e9849e 100644
--- a/schematic/configuration.py
+++ b/schematic/configuration.py
@@ -9,12 +9,18 @@ def __init__(self):
# entire configuration data
self.DATA = None
+
def __getattribute__(self, name):
value = super().__getattribute__(name)
if value is None and "SCHEMATIC_CONFIG" in os.environ:
self.load_config_from_env()
value = super().__getattribute__(name)
- elif value is None and "SCHEMATIC_CONFIG" not in os.environ:
+
+ elif value is None and "SCHEMATIC_CONFIG_CONTENT" in os.environ:
+ self.load_config_content_from_env()
+ value = super().__getattribute__(name)
+
+ elif value is None and "SCHEMATIC_CONFIG" not in os.environ and "SCHEMATIC_CONFIG_CONTENT" not in os.environ:
raise AttributeError(
"The '%s' configuration field was accessed, but it hasn't been "
"set yet, presumably because the schematic.CONFIG.load_config() "
@@ -34,7 +40,7 @@ def get(self, key, default):
value = default
return value
- def load_config_content(str_yaml: str) -> dict:
+ def load_config_content(self, str_yaml: str) -> dict:
try:
config_data = yaml.safe_load(str_yaml)
except yaml.YAMLError as exc:
@@ -53,9 +59,16 @@ def load_yaml(file_path: str) -> dict:
return config_data
def normalize_path(self, path):
- # Retrieve parent directory of the config to decode relative paths
- parent_dir = os.path.dirname(self.CONFIG_PATH)
- # Ensure absolute file paths
+
+ if self.CONFIG_PATH:
+ # Retrieve parent directory of the config to decode relative paths
+ parent_dir = os.path.dirname(self.CONFIG_PATH)
+ else:
+ # not sure if this is going to work
+ # but assume the parent dir would be the current work dir
+ parent_dir = os.getcwd()
+
+ # Ensure absolute file paths
if not os.path.isabs(path):
path = os.path.join(parent_dir, path)
# And lastly, normalize file paths
@@ -69,6 +82,18 @@ def load_config_from_env(self):
)
return self.load_config(schematic_config)
+ def load_config_content_from_env(self):
+ schematic_config_content = os.environ["SCHEMATIC_CONFIG_CONTENT"]
+
+ print(
+ 'Loading content of config file. The environment variable is: %s' % schematic_config_content
+ )
+
+ config_content_yaml = self.load_config_content(schematic_config_content)
+ self.DATA = config_content_yaml
+
+ return self.DATA
+
def load_config(self, config_path=None, asset_view=None):
# If config_path is None, try loading from environment
if config_path is None and "SCHEMATIC_CONFIG" in os.environ:
@@ -85,10 +110,6 @@ def load_config(self, config_path=None, asset_view=None):
config_path = os.path.abspath(config_path)
self.DATA = self.load_yaml(config_path)
- # test
- if "SCHEMATIC_CONFIG_CONTENT" in os.environ:
- config_content = self.load_config_content(os.environ["SCHEMATIC_CONFIG_CONTENT"])
- self.DATA = config_content
self.CONFIG_PATH = config_path
From 40a32ffef68ce3e681a51220753a9e28e8a8c2b2 Mon Sep 17 00:00:00 2001
From: linglp
Date: Thu, 20 Oct 2022 11:26:01 -0700
Subject: [PATCH 18/35] use schematic_config_content when it is available
---
schematic/configuration.py | 15 ++++++---------
1 file changed, 6 insertions(+), 9 deletions(-)
diff --git a/schematic/configuration.py b/schematic/configuration.py
index 3a8e9849e..af779ce24 100644
--- a/schematic/configuration.py
+++ b/schematic/configuration.py
@@ -12,14 +12,12 @@ def __init__(self):
def __getattribute__(self, name):
value = super().__getattribute__(name)
- if value is None and "SCHEMATIC_CONFIG" in os.environ:
- self.load_config_from_env()
- value = super().__getattribute__(name)
-
- elif value is None and "SCHEMATIC_CONFIG_CONTENT" in os.environ:
+ if value is None and "SCHEMATIC_CONFIG_CONTENT" in os.environ:
self.load_config_content_from_env()
value = super().__getattribute__(name)
-
+ elif value is None and "SCHEMATIC_CONFIG" in os.environ:
+ self.load_config_from_env()
+ value = super().__getattribute__(name)
elif value is None and "SCHEMATIC_CONFIG" not in os.environ and "SCHEMATIC_CONFIG_CONTENT" not in os.environ:
raise AttributeError(
"The '%s' configuration field was accessed, but it hasn't been "
@@ -64,8 +62,7 @@ def normalize_path(self, path):
# Retrieve parent directory of the config to decode relative paths
parent_dir = os.path.dirname(self.CONFIG_PATH)
else:
- # not sure if this is going to work
- # but assume the parent dir would be the current work dir
+ # assume the parent dir would be the current work dir
parent_dir = os.getcwd()
# Ensure absolute file paths
@@ -86,7 +83,7 @@ def load_config_content_from_env(self):
schematic_config_content = os.environ["SCHEMATIC_CONFIG_CONTENT"]
print(
- 'Loading content of config file. The environment variable is: %s' % schematic_config_content
+ 'Loading content of config file: %s' % schematic_config_content
)
config_content_yaml = self.load_config_content(schematic_config_content)
From 4e153aa1fc715c00080fef22ecfdff7019b5a997 Mon Sep 17 00:00:00 2001
From: linglp
Date: Thu, 20 Oct 2022 11:33:31 -0700
Subject: [PATCH 19/35] change location of code
---
api/__init__.py | 2 +-
schematic/configuration.py | 3 ---
2 files changed, 1 insertion(+), 4 deletions(-)
diff --git a/api/__init__.py b/api/__init__.py
index c823b0517..fe8fd1777 100644
--- a/api/__init__.py
+++ b/api/__init__.py
@@ -15,9 +15,9 @@ def create_app():
# path to config.yml file saved as a Flask config variable
default_config = os.path.abspath(os.path.join(__file__, "../../config.yml"))
schematic_config = os.environ.get("SCHEMATIC_CONFIG", default_config)
+ schematic_config_content = os.environ.get("SCHEMATIC_CONFIG_CONTENT")
app.config["SCHEMATIC_CONFIG"] = schematic_config
- schematic_config_content = os.environ.get("SCHEMATIC_CONFIG_CONTENT")
app.config["SCHEMATIC_CONFIG_CONTENT"] = schematic_config_content
# Configure flask app
diff --git a/schematic/configuration.py b/schematic/configuration.py
index af779ce24..df1d200c2 100644
--- a/schematic/configuration.py
+++ b/schematic/configuration.py
@@ -106,9 +106,6 @@ def load_config(self, config_path=None, asset_view=None):
config_path = os.path.expanduser(config_path)
config_path = os.path.abspath(config_path)
self.DATA = self.load_yaml(config_path)
-
-
-
self.CONFIG_PATH = config_path
# handle user input (for API endpoints)
if asset_view:
From 3ce502ac21308cbe87ba92642e934f669be07140 Mon Sep 17 00:00:00 2001
From: linglp
Date: Thu, 20 Oct 2022 11:37:13 -0700
Subject: [PATCH 20/35] if the env variable exists
---
api/routes.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/api/routes.py b/api/routes.py
index 6066664a7..7fbf6c95c 100644
--- a/api/routes.py
+++ b/api/routes.py
@@ -39,7 +39,9 @@ def config_handler(asset_view=None):
# if content of the config file is provided:
content_of_config = app.config["SCHEMATIC_CONFIG_CONTENT"]
- if len(content_of_config) > 0:
+
+ # if the environment variable exists
+ if content_of_config:
CONFIG.load_config_content_from_env()
# check if path to config is provided
From 7f1f104b3e509187583eb97900bc769a808acfa7 Mon Sep 17 00:00:00 2001
From: linglp
Date: Thu, 20 Oct 2022 15:01:57 -0700
Subject: [PATCH 21/35] add schematic config content on docker compose
---
docker-compose.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docker-compose.yml b/docker-compose.yml
index accefcea6..f6e15a901 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -13,5 +13,5 @@ services:
APP_HOST: "0.0.0.0"
APP_PORT: "3001"
SCHEMATIC_CONFIG: /schematic/config.yml
- #SCHEMATIC_CONFIG_CONTENT: "${SCHEMATIC_CONFIG_CONTENT}"
+ SCHEMATIC_CONFIG_CONTENT: "${SCHEMATIC_CONFIG_CONTENT}"
GE_HOME: /usr/src/app/great_expectations/
From 60536d3fa038b5030a6fb74b2f639e7b863e3a89 Mon Sep 17 00:00:00 2001
From: linglp
Date: Tue, 25 Oct 2022 15:59:15 -0400
Subject: [PATCH 22/35] add insturctions related to using
SCHEMATIC_CONFIG_CONTENT
---
README.md | 22 +++++++++++++++++++++-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index c56d1f5a7..2b64034d5 100644
--- a/README.md
+++ b/README.md
@@ -188,16 +188,36 @@ For new features, bugs, enhancements
- For more information on flags for `docker run` and what they do, visit the [Docker Documentation](https://docs.docker.com/engine/reference/commandline/run/)
- These example commands assume that you have navigated to the directory you want to run schematic from. To specify your working directory, use `$(pwd)` on MacOS/Linux or `%cd%` on Windows.
- If not using the latest image, then the full name should be specified: ie `sagebionetworks/schematic:commit-e611e4a`
+ - If using local image created by `docker compose up`, then the docker image name should be changed: i.e. `schematic_schematic`
+
### Example For REST API
+
+#### Use file path of `config.yml` to run API endpoints:
```
docker run --rm -p 3001:3001 \
-v $(pwd):/schematic -w /schematic --name schematic \
- -e SCHEMATIC_CONFIG=/schematic/schematic_config.yml \
+ -e SCHEMATIC_CONFIG=/schematic/config.yml \
-e GE_HOME=/usr/src/app/great_expectations/ \
sagebionetworks/schematic \
python /usr/src/app/run_api.py
```
+#### Use content of `config.yml` as an environment variable to run API endpoints:
+1. save content of `config.yml` as to environment variable `SCHEMATIC_CONFIG_CONTENT` by doing: `export SCHEMATIC_CONFIG_CONTENT=$(cat config.yml)`
+
+2. Pass `SCHEMATIC_CONFIG_CONTENT` as an environment variable by using `docker run`
+
+```
+docker run --rm -p 3001:3001 \
+ -v $(pwd):/schematic -w /schematic --name schematic \
+ -e SCHEMATIC_CONFIG=/schematic/config.yml \
+ -e GE_HOME=/usr/src/app/great_expectations/ \
+ -e SCHEMATIC_CONFIG_CONTENT=$SCHEMATIC_CONFIG_CONTENT \
+ sagebionetworks/schematic \
+ python /usr/src/app/run_api.py
+```
+
+
### Example For Schematic on mac/linux
To run example below, first clone schematic into your home directory `git clone https://github.com/sage-bionetworks/schematic ~/schematic`
Then update .synapseConfig with your credentials
From 2ab1de9b9aa6f38acfe73d57c8dbd244e9b7cc65 Mon Sep 17 00:00:00 2001
From: linglp
Date: Thu, 27 Oct 2022 11:20:55 -0400
Subject: [PATCH 23/35] fix spacing; change documentation
---
.github/workflows/pdoc.yml | 4 ++--
schematic/schemas/df_parser.py | 14 +++++++-------
2 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index d5e322656..2bf759750 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -73,11 +73,11 @@ jobs:
# create documentation
- run: poetry show pdoc
- - run: poetry run pdoc --docformat google -o docs/api api/routes.py
+ - run: poetry run pdoc --docformat google -o docs schematic/manifest schematic/models schematic/schemas schematic/store schematic/utils schematic/visualization
- uses: actions/upload-pages-artifact@v1
with:
- path: docs/api
+ path: docs/schematic
# Deploy the artifact to GitHub pages.
# This is a separate job so that only actions/deploy-pages has the necessary permissions.
diff --git a/schematic/schemas/df_parser.py b/schematic/schemas/df_parser.py
index 5f46b1626..fd0a52fab 100644
--- a/schematic/schemas/df_parser.py
+++ b/schematic/schemas/df_parser.py
@@ -162,13 +162,13 @@ def get_property(
"""Constructs a new schema.org compliant property of an existing schema.org object/class; note that the property itself is a schema.org object class.
Args:
- se: a schema explorer object allowing the traversal and modification of a schema graph
- property_display_name: human readable label for the schema object/attribute: key characteristic X of the assay, related protocol, or downstream data that we want to record as metadata feature
- property_class_name: *schema* label of the class/object that this is a property of
- description: definition or a reference containing the definition of attribute X. Preferably provide a source ontology link or code in addition to the definition.
- requires_range: what is the set/domain of values that this attribute can be assigned to; currently only used to specify primitive types. TODO: extend to reg exp patterns
- requires_dependencies: important characteristics, if any, of property X that need to be recorded as metadata features given property X is specified. These characteristics are attributes themselves and need to pre-exist in the schema as such
- validation_rules: a list of validation rules defined for this class (e.g. defining what is a valid object of this property)
+ se: a schema explorer object allowing the traversal and modification of a schema graph
+ property_display_name: human readable label for the schema object/attribute: key characteristic X of the assay, related protocol, or downstream data that we want to record as metadata feature
+ property_class_name: *schema* label of the class/object that this is a property of
+ description: definition or a reference containing the definition of attribute X. Preferably provide a source ontology link or code in addition to the definition.
+ requires_range: what is the set/domain of values that this attribute can be assigned to; currently only used to specify primitive types. TODO: extend to reg exp patterns
+ requires_dependencies: important characteristics, if any, of property X that need to be recorded as metadata features given property X is specified. These characteristics are attributes themselves and need to pre-exist in the schema as such
+ validation_rules: a list of validation rules defined for this class (e.g. defining what is a valid object of this property)
Returns: a json schema.org property object
From 643cdb43359ddfbc9c00f931314e1bec74a5a00a Mon Sep 17 00:00:00 2001
From: linglp
Date: Thu, 27 Oct 2022 11:22:26 -0400
Subject: [PATCH 24/35] trigger workflow
---
.github/workflows/pdoc.yml | 1 +
1 file changed, 1 insertion(+)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index 2bf759750..32f2f5f36 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -5,6 +5,7 @@ on:
push:
branches:
- develop
+ - develop-pydoc # for experiment
workflow_dispatch: # Allow manually triggering the workflow
# security: restrict permissions for CI jobs.
From 09d9a9d234018dd937601889878c3b659c11eaad Mon Sep 17 00:00:00 2001
From: linglp
Date: Thu, 27 Oct 2022 11:39:33 -0400
Subject: [PATCH 25/35] update actions/cache from v2 to v3
---
.github/workflows/pdoc.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index 32f2f5f36..c71d27343 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -60,7 +60,7 @@ jobs:
#----------------------------------------------
- name: Load cached venv
id: cached-poetry-dependencies
- uses: actions/cache@v2
+ uses: actions/cache@v3
with:
path: .venv
key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
From a3043e08a2e8e9984c2fcaaf80b8785af21eb54d Mon Sep 17 00:00:00 2001
From: linglp
Date: Thu, 27 Oct 2022 12:22:57 -0400
Subject: [PATCH 26/35] change dir
---
.github/workflows/pdoc.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index c71d27343..0038ccbf8 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -74,7 +74,7 @@ jobs:
# create documentation
- run: poetry show pdoc
- - run: poetry run pdoc --docformat google -o docs schematic/manifest schematic/models schematic/schemas schematic/store schematic/utils schematic/visualization
+ - run: poetry run pdoc --docformat google -o docs/schematic schematic/manifest schematic/models schematic/schemas schematic/store schematic/utils schematic/visualization
- uses: actions/upload-pages-artifact@v1
with:
From e42e100e46e1b2c3570637ab9cd0841f85b6104b Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 31 Oct 2022 17:08:29 -0400
Subject: [PATCH 27/35] update error msg
---
schematic/store/synapse.py | 22 +++++++++++++++-------
1 file changed, 15 insertions(+), 7 deletions(-)
diff --git a/schematic/store/synapse.py b/schematic/store/synapse.py
index 206a9957d..1cfc28ace 100644
--- a/schematic/store/synapse.py
+++ b/schematic/store/synapse.py
@@ -79,8 +79,23 @@ def __init__(
self.syn = self.login(token, access_token, input_token)
self.project_scope = project_scope
+
+
+ # check if "master_fileview" has been set
+ try:
+ self.storageFileview = CONFIG["synapse"]["master_fileview"]
+ except:
+ raise MissingConfigValueError(("synapse", "master_fileview"))
+
+ # check if "master_basename" has been set
+ try:
+ self.manifest = CONFIG["synapse"]["manifest_basename"]
+ except:
+ raise MissingConfigValueError(("synapse", "master_basename"))
+
try:
self.storageFileview = CONFIG["synapse"]["master_fileview"]
+ self.manifest = CONFIG["synapse"]["manifest_basename"]
if self.project_scope:
self.storageFileviewTable = self.syn.tableQuery(
f"SELECT * FROM {self.storageFileview} WHERE projectId IN {tuple(self.project_scope + [''])}"
@@ -90,17 +105,10 @@ def __init__(
self.storageFileviewTable = self.syn.tableQuery(
"SELECT * FROM " + self.storageFileview
).asDataFrame()
-
- self.manifest = CONFIG["synapse"]["manifest_basename"]
-
- except KeyError:
- raise MissingConfigValueError(("synapse", "master_fileview"))
except AttributeError:
raise AttributeError("storageFileview attribute has not been set.")
except SynapseHTTPError:
raise AccessCredentialsError(self.storageFileview)
- except ValueError:
- raise MissingConfigValueError(("synapse", "master_fileview"))
@staticmethod
def login(token=None, access_token=None, input_token=None):
From d872a52397fdd3d67c78a08f7f1fa05a8660c4f3 Mon Sep 17 00:00:00 2001
From: linglp
Date: Mon, 31 Oct 2022 18:02:27 -0400
Subject: [PATCH 28/35] specify key error
---
schematic/store/synapse.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/schematic/store/synapse.py b/schematic/store/synapse.py
index 1cfc28ace..69e204397 100644
--- a/schematic/store/synapse.py
+++ b/schematic/store/synapse.py
@@ -84,13 +84,13 @@ def __init__(
# check if "master_fileview" has been set
try:
self.storageFileview = CONFIG["synapse"]["master_fileview"]
- except:
+ except KeyError:
raise MissingConfigValueError(("synapse", "master_fileview"))
# check if "master_basename" has been set
try:
self.manifest = CONFIG["synapse"]["manifest_basename"]
- except:
+ except KeyError:
raise MissingConfigValueError(("synapse", "master_basename"))
try:
From 714c03b8e5b4e1658cea8fcefccef1721db71946 Mon Sep 17 00:00:00 2001
From: Gianna Jordan <61707471+GiaJordan@users.noreply.github.com>
Date: Fri, 4 Nov 2022 14:07:12 -0400
Subject: [PATCH 29/35] extend sleep duration to time functional for HTAN
---
schematic/store/synapse.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/schematic/store/synapse.py b/schematic/store/synapse.py
index 206a9957d..71c49fb39 100644
--- a/schematic/store/synapse.py
+++ b/schematic/store/synapse.py
@@ -1413,7 +1413,7 @@ def make_synapse_table(self,
# remove rows
self.syn.delete(existing_results)
# wait for row deletion to finish on synapse before getting empty table
- sleep(1)
+ sleep(10)
# removes all current columns
current_table = self.syn.get(existingTableId)
From 50b0ef543a2b5bd141cdce04890552e4d4eb43ee Mon Sep 17 00:00:00 2001
From: Gianna Jordan <61707471+GiaJordan@users.noreply.github.com>
Date: Tue, 8 Nov 2022 14:32:44 -0700
Subject: [PATCH 30/35] Update README.md
add info related to image and container names
---
README.md | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/README.md b/README.md
index 2b64034d5..f667ddeec 100644
--- a/README.md
+++ b/README.md
@@ -182,13 +182,15 @@ For new features, bugs, enhancements
## Installation Guide - Docker
1. Install docker from https://www.docker.com/ .
-2. Identify docker container of interest from [Schematic DockerHub](https://hub.docker.com/r/sagebionetworks/schematic/tags)
- Ex `docker pull sagebionetworks/schematic:latest` from the CLI or, run `docker compose up` after cloning the schematic github repo
+2. Identify docker image of interest from [Schematic DockerHub](https://hub.docker.com/r/sagebionetworks/schematic/tags)
+ Ex `docker pull sagebionetworks/schematic:latest` from the CLI or, run `docker compose up` after cloning the schematic github repo
+ in this case, `sagebionetworks/schematic:latest` is the name of the image chosen
3. Run Schematic Command with `docker run `.
- For more information on flags for `docker run` and what they do, visit the [Docker Documentation](https://docs.docker.com/engine/reference/commandline/run/)
- These example commands assume that you have navigated to the directory you want to run schematic from. To specify your working directory, use `$(pwd)` on MacOS/Linux or `%cd%` on Windows.
- - If not using the latest image, then the full name should be specified: ie `sagebionetworks/schematic:commit-e611e4a`
- - If using local image created by `docker compose up`, then the docker image name should be changed: i.e. `schematic_schematic`
+ - If not using the latest image, then the full name should be specified: ie `sagebionetworks/schematic:commit-e611e4a`
+ - If using local image created by `docker compose up`, then the docker image name should be changed: i.e. `schematic_schematic`
+ - Using the `--name` flag sets the name of the container running locally on your machine
### Example For REST API
From 64d0b281c6985961f990a98dcc052e9b5ffe6ed7 Mon Sep 17 00:00:00 2001
From: afwillia
Date: Tue, 8 Nov 2022 14:14:49 -0800
Subject: [PATCH 31/35] Update image name in docker instructions
---
README.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/README.md b/README.md
index f667ddeec..f488413e2 100644
--- a/README.md
+++ b/README.md
@@ -229,7 +229,7 @@ docker run \
-w /schematic \
-e SCHEMATIC_CONFIG=/schematic/config.yml \
-e GE_HOME=/usr/src/app/great_expectations/ \
- aws_dca_schematic_deploy-schematic schematic model \
+ sagebionetworks/schematic schematic model \
-c /schematic/config.yml validate \
-mp /schematic/tests/data/mock_manifests/Valid_Test_Manifest.csv \
-dt MockComponent \
From a19ca2d0ba84f999f1d504e37ede8f8871b35851 Mon Sep 17 00:00:00 2001
From: linglp
Date: Tue, 8 Nov 2022 18:11:56 -0500
Subject: [PATCH 32/35] remove SCHEMATIC_CONFIG env variable when using config
content to to avoid confusion
---
README.md | 1 -
1 file changed, 1 deletion(-)
diff --git a/README.md b/README.md
index f488413e2..486262f27 100644
--- a/README.md
+++ b/README.md
@@ -212,7 +212,6 @@ docker run --rm -p 3001:3001 \
```
docker run --rm -p 3001:3001 \
-v $(pwd):/schematic -w /schematic --name schematic \
- -e SCHEMATIC_CONFIG=/schematic/config.yml \
-e GE_HOME=/usr/src/app/great_expectations/ \
-e SCHEMATIC_CONFIG_CONTENT=$SCHEMATIC_CONFIG_CONTENT \
sagebionetworks/schematic \
From 8825d5eb65514ba30e004dda6d6ecb54c6143bc1 Mon Sep 17 00:00:00 2001
From: linglp
Date: Thu, 10 Nov 2022 14:31:06 -0500
Subject: [PATCH 33/35] correct typo
---
schematic/store/synapse.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/schematic/store/synapse.py b/schematic/store/synapse.py
index b72b6e372..ca57490ba 100644
--- a/schematic/store/synapse.py
+++ b/schematic/store/synapse.py
@@ -87,11 +87,11 @@ def __init__(
except KeyError:
raise MissingConfigValueError(("synapse", "master_fileview"))
- # check if "master_basename" has been set
+ # check if "manifest_basename" has been set
try:
self.manifest = CONFIG["synapse"]["manifest_basename"]
except KeyError:
- raise MissingConfigValueError(("synapse", "master_basename"))
+ raise MissingConfigValueError(("synapse", "manifest_basename"))
try:
self.storageFileview = CONFIG["synapse"]["master_fileview"]
From 17c30b76bee82da727ac1b26064310ec802fe3fa Mon Sep 17 00:00:00 2001
From: linglp
Date: Thu, 10 Nov 2022 14:42:09 -0500
Subject: [PATCH 34/35] remove using develop-pydoc branch to deploy
---
.github/workflows/pdoc.yml | 1 -
1 file changed, 1 deletion(-)
diff --git a/.github/workflows/pdoc.yml b/.github/workflows/pdoc.yml
index 0038ccbf8..654b6cb5c 100644
--- a/.github/workflows/pdoc.yml
+++ b/.github/workflows/pdoc.yml
@@ -5,7 +5,6 @@ on:
push:
branches:
- develop
- - develop-pydoc # for experiment
workflow_dispatch: # Allow manually triggering the workflow
# security: restrict permissions for CI jobs.
From e83c18f2d4217aab47f78dc2db74e723c0e63634 Mon Sep 17 00:00:00 2001
From: linglp
Date: Thu, 10 Nov 2022 14:48:11 -0500
Subject: [PATCH 35/35] remove changes of indentation
---
schematic/schemas/df_parser.py | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/schematic/schemas/df_parser.py b/schematic/schemas/df_parser.py
index fd0a52fab..d4e6c7081 100644
--- a/schematic/schemas/df_parser.py
+++ b/schematic/schemas/df_parser.py
@@ -162,13 +162,13 @@ def get_property(
"""Constructs a new schema.org compliant property of an existing schema.org object/class; note that the property itself is a schema.org object class.
Args:
- se: a schema explorer object allowing the traversal and modification of a schema graph
- property_display_name: human readable label for the schema object/attribute: key characteristic X of the assay, related protocol, or downstream data that we want to record as metadata feature
- property_class_name: *schema* label of the class/object that this is a property of
- description: definition or a reference containing the definition of attribute X. Preferably provide a source ontology link or code in addition to the definition.
- requires_range: what is the set/domain of values that this attribute can be assigned to; currently only used to specify primitive types. TODO: extend to reg exp patterns
- requires_dependencies: important characteristics, if any, of property X that need to be recorded as metadata features given property X is specified. These characteristics are attributes themselves and need to pre-exist in the schema as such
- validation_rules: a list of validation rules defined for this class (e.g. defining what is a valid object of this property)
+ se: a schema explorer object allowing the traversal and modification of a schema graph
+ property_display_name: human readable label for the schema object/attribute: key characteristic X of the assay, related protocol, or downstream data that we want to record as metadata feature
+ property_class_name: *schema* label of the class/object that this is a property of
+ description: definition or a reference containing the definition of attribute X. Preferably provide a source ontology link or code in addition to the definition.
+ requires_range: what is the set/domain of values that this attribute can be assigned to; currently only used to specify primitive types. TODO: extend to reg exp patterns
+ requires_dependencies: important characteristics, if any, of property X that need to be recorded as metadata features given property X is specified. These characteristics are attributes themselves and need to pre-exist in the schema as such
+ validation_rules: a list of validation rules defined for this class (e.g. defining what is a valid object of this property)
Returns: a json schema.org property object