.

smaht-dac · Dec 4, 2023 · 493d696 · 493d696
1 parent e3abace
commit 493d696
Show file tree

Hide file tree

Showing 15 changed files with 71 additions and 355 deletions.
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -4,8 +4,6 @@ Change Log
 ==========
 
 
-3.0.0
+0.0.1
 =====
-* 2023-10-10
-* Added this CHANGELOG.rst file.
-* Upgrade to Python 3.11.
+* Initial release
diff --git a/LOG.md b/LOG.md
diff --git a/README.md b/README.md
@@ -21,6 +21,6 @@ To install from source:
     make update
     make build
 
-To check that the software is correctly installed, try to run `pipeline_utils`. If installed from source, this command may fail with a bash “command not found” error, try `poetry run pipeline_utils` instead.
+To check that the software is correctly installed, try to run `smaht_pipeline_utils`. If installed from source, this command may fail with a bash “command not found” error, try `poetry run smaht_pipeline_utils` instead.
 
 See `make info` for details on make targets.
diff --git a/docs/conf.py b/docs/conf.py
@@ -18,11 +18,11 @@
 # -- Project information -----------------------------------------------------
 
 project = 'smaht-pipeline-utils'
-copyright = '2021, HMS DBMI'
+copyright = '2023, HMS DBMI'
 author = 'Michele Berselli, SMaHT Team'
 
 # The full version, including alpha/beta/rc tags
-release = '2.1.0'
+release = '0.0.1'
 
 
 # -- General configuration ---------------------------------------------------

diff --git a/docs/deploy_pipeline.rst b/docs/deploy_pipeline.rst
@@ -142,7 +142,7 @@ Example of a key-pair entry:
     }
   }
 
-``<namespace>`` is the namespace for the environment and can be found in the portal health page (e.g., cgap-wolf).
+``<namespace>`` is the namespace for the environment and can be found in the portal health page (e.g., smaht-wolf).
 
 .. _account_vars:
 
@@ -154,21 +154,21 @@ Finally we need to setup the information to identify the target environment to u
 .. code-block:: bash
 
   # Set the namespace of the target environment
-  #   e.g., cgap-wolf
+  #   e.g., smaht-wolf
   export ENV_NAME=
 
   # Set the bucket used to store the worklow description files
-  #   e.g., cgap-biotest-main-application-tibanna-cwls
+  #   e.g., smaht-wolf-application-tibanna-cwls
   export WFL_BUCKET=
 
   # Set the path to the keypair file with the portal credential
   export KEYDICTS_JSON=~/.cgap-keys.json
 
-  # Set up project and institution
-  #   Project and institution need to correspond to metadata present on the portal
-  #   e.g., cgap-core and hms-dbmi
-  export PROJECT=
-  export INSTITUTION=
+  # Set up consortia and submission centers
+  #   consortia and submission_centers need to correspond to metadata present on the portal
+  #   e.g., ['smaht'] and ['smaht_dac']
+  export CONSORTIA=
+  export SUBMISSION_CENTERS=
 
   # If running sentieon code,
   #   specify the address for the server that validate the software license
@@ -194,8 +194,8 @@ by the ``--repos`` argument.
     --wfl-bucket ${WFL_BUCKET} \
     --account ${AWS_ACCOUNT_NUMBER} \
     --region ${TIBANNA_AWS_REGION} \
-    --project ${PROJECT} \
-    --institution ${INSTITUTION} \
+    --consortia ${CONSORTIA} \
+    --submission-centers ${SUBMISSION_CENTERS} \
     --sentieon-server ${SENTIEON_LICENSE} \
     --post-software \
     --post-file-format \

diff --git a/docs/functions.rst b/docs/functions.rst
@@ -2,49 +2,4 @@
 Functions
 =========
 
-Collection of utilities available as functions:
-
-  - :ref:`check_lines <check_lines>`
-
-.. _check_lines:
-
-check_lines
-+++++++++++
-
-*check_lines* function can be used to check that line counts are matching between the output of two steps where lines should not be dropped (i.e., any steps that modify without filtering), or between an output ``bam`` and the input ``fastq`` files.
-Requires uuid for the *MetaWorkflowRun* object to check and ff_key to access the metadata on the portal. The steps to compare are specified as dictionaries, examples below.
-
-.. code-block:: python
-
-    from pipeline_utils.lib import check_lines
-
-    result <bool> = check_lines.check_lines(metawfr_uuid, ff_key, steps=steps_dict, fastqs=fastqs_dict)
-
-    # metawfr_uuid
-    #   -> uuid for MetaWorkflowRun object
-
-    # ff_key
-    #   -> key to authenticate on the portal
-
-    ## steps_dict example
-    # steps_dict = {
-    #     'workflow_add-readgroups-check': {
-    #                         'dependency': 'workflow_bwa-mem_no_unzip-check',
-    #                         'output': 'bam_w_readgroups',
-    #                         'output_match': 'raw_bam',
-    #                         'key': 'Total Reads',
-    #                         'key_match': 'Total Reads'
-    #                         },
-    #     ...
-    # }
-
-    ## fastqs_dict example
-    # fastqs_dict = {
-    #     'workflow_bwa-mem_no_unzip-check': {
-    #                          'output': 'raw_bam',
-    #                          'input_match': ['fastq_R1', 'fastq_R2'],
-    #                          'key': 'Total Reads',
-    #                          'key_match': 'Total Sequences'
-    #                          },
-    #     ...
-    # }
+In development.
diff --git a/docs/index.rst b/docs/index.rst
@@ -1,8 +1,8 @@
-=========================
-Portal Pipeline Utilities
-=========================
+===============================
+SMaHT Portal Pipeline Utilities
+===============================
 
-Documentation for smaht-pipeline-utils_, a collection of utilities for deploying pipelines and interfacing with portal infrastructure.
+Documentation for smaht-pipeline-utils_, a collection of utilities for deploying pipelines and interfacing with SMaHT portal infrastructure.
 
 .. _smaht-pipeline-utils: https://github.com/smaht-dac/smaht-pipeline-utils
 

diff --git a/docs/pipeline_utils.rst b/docs/pipeline_utils.rst
@@ -61,10 +61,10 @@ Usage:
     - AWS account to use for deployment
   * - *-\-region*
     - AWS account region to use for deployment
-  * - *-\-project*
-    - Project to use for deployment [cgap-core]
-  * - *-\-institution*
-    - Institution to use for deployment [hms-dbmi]
+  * - *-\-consortia*
+    - List of consortia to use for deployment [smaht]
+  * - *-\-submission-centers*
+    - List of centers to use for deployment [smaht_dac]
   * - *-\-post-software*
     - DEPLOY | UPDATE Software objects (.yaml or .yml)
   * - *-\-post-file-format*

diff --git a/docs/yaml_file_format.rst b/docs/yaml_file_format.rst
@@ -22,10 +22,9 @@ Template
 
     # All the following fields are optional and provided as example,
     #   can be expanded to anything accepted by the schema
-    #   https://github.com/dbmi-bgm/cgap-portal/tree/master/src/encoded/schemas
+    #   https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas
     secondary_formats:
       - <format>              # bam, fastq, bwt, ...
-    status: <status>          # shared
 
 
 Fields Definition
@@ -49,7 +48,7 @@ Description of the file format.
 
 Optional
 ^^^^^^^^
-All the following fields are optional and provided as example. Can be expanded to anything accepted by the schema, see `schemas <https://github.com/dbmi-bgm/cgap-portal/tree/master/src/encoded/schemas>`__.
+All the following fields are optional and provided as example. Can be expanded to anything accepted by the schema, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
 
 secondary_formats
 -----------------

diff --git a/docs/yaml_file_reference.rst b/docs/yaml_file_reference.rst
@@ -21,9 +21,14 @@ Template
     format: <format>              # bam, fastq, bwt, ...
     version: <string>
 
+    category:
+      - <category>                # Reference Genome, ...
+    type:
+      -  <type>                   # Reference Sequence, ...
+
     # All the following fields are optional and provided as example,
     #   can be expanded to anything accepted by the schema
-    #   https://github.com/dbmi-bgm/cgap-portal/tree/master/src/encoded/schemas
+    #   https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas
     secondary_files:
       - <format>                  # bam, fastq, bwt, ...
     status: <status>              # uploading, uploaded
@@ -60,7 +65,7 @@ Version of the reference file.
 
 Optional
 ^^^^^^^^
-All the following fields are optional and provided as example. Can be expanded to anything accepted by the schema, see `schemas <https://github.com/dbmi-bgm/cgap-portal/tree/master/src/encoded/schemas>`__.
+All the following fields are optional and provided as example. Can be expanded to anything accepted by the schema, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
 
 secondary_files
 ---------------
@@ -78,3 +83,11 @@ Most likely you don't want to set this field and just use the default logic auto
 license
 -------
 License information.
+
+category
+--------
+Categories for the reference file, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
+
+type
+----
+Types for the reference file, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
diff --git a/docs/yaml_metaworkflow.rst b/docs/yaml_metaworkflow.rst
@@ -20,6 +20,9 @@ Template
     name: <string>
     description: <string>
 
+    category:
+      - <category>                          # Alignment, ...
+
     ## General arguments ########################################
     #     Pipeline input, reference files, and general arguments
     #       define all arguments for the pipeline here
@@ -61,7 +64,7 @@ Template
         #   Allows to force a fixed shards structure ignoring
         #     the input structure, scatter and gather dimensions
         ####################################
-        shards: [[<string>], ..]               # e.g., [['0'], ['1'], ['2']]
+        shards: [[<string>], ..]             # e.g., [['0'], ['1'], ['2']]
 
         ## Lock version ####################
         #   Specific version to use
@@ -81,7 +84,7 @@ Template
 
           # File argument
           <file_argument_name>:
-            argument_type: file.<format>      # bam, fastq, bwt ...
+            argument_type: file.<format>     # bam, fastq, bwt ...
             # Linking fields
             #   These are optional fields
             #   Check https://magma-suite.readthedocs.io/en/latest/meta-workflow.html
@@ -157,6 +160,10 @@ description
 -----------
 Description of the pipeline.
 
+category
+--------
+Categories for the pipeline, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
+
 input
 -----
 Description of general input files and parameters for the pipeline. See :ref:`Input Definition <input>`.
@@ -167,11 +174,7 @@ Description of workflows that are steps of the pipeline. See :ref:`Workflows Def
 
 Optional
 ^^^^^^^^
-All the following fields are optional and provided as example. Can be expanded to anything accepted by the schema, see `schemas <https://github.com/dbmi-bgm/cgap-portal/tree/master/src/encoded/schemas>`__.
-
-title
------
-Title of the pipeline.
+All the following fields are optional and provided as example. Can be expanded to anything accepted by the schema, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
 
 
 .. _workflows:
@@ -210,7 +213,7 @@ output
 Description of expected output files for the workflow.
 
 Each output is defined by its name. Additional subfields can be specified.
-See `schemas <https://github.com/dbmi-bgm/cgap-portal/tree/master/src/encoded/schemas>`__.
+See `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
 
 Each output name needs to match an output name that has been previously defined in the corresponding workflow, see :ref:`Workflow <workflow>`.
 

diff --git a/docs/yaml_software.rst b/docs/yaml_software.rst
@@ -22,10 +22,12 @@ Template
     version: <string>
     commit: <string>
 
+    category:
+      - <category>                    # Quality Control
+
     # All the following fields are optional and provided as example,
     #   can be expanded to anything accepted by the schema
-    #   https://github.com/dbmi-bgm/cgap-portal/tree/master/src/encoded/schemas
-    title: <string>
+    #   https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas
     source_url: <string>
     description: <string>
     license: <string>                 # MIT, GPLv3, ...
@@ -51,13 +53,13 @@ commit
 ------
 Commit of the software.
 
+category
+--------
+Categories for the software, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
+
 Optional
 ^^^^^^^^
-All the following fields are optional and provided as example. Can be expanded to anything accepted by the schema, see `schemas <https://github.com/dbmi-bgm/cgap-portal/tree/master/src/encoded/schemas>`__.
-
-title
------
-Title for the software.
+All the following fields are optional and provided as example. Can be expanded to anything accepted by the schema, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
 
 source_url
 ----------

diff --git a/docs/yaml_workflow.rst b/docs/yaml_workflow.rst
@@ -26,11 +26,12 @@ Template
       child:
         - <file>                          # .cwl or .wdl file
 
+    category:
+      - <category>                        # Annotation
+
     # All the following fields are optional and provided as example,
     #   can be expanded to anything accepted by the schema
-    #   https://github.com/dbmi-bgm/cgap-portal/tree/master/src/encoded/schemas
-    title: <string>
-
+    #   https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas
     software:
       - <software>@<version|commit>
 
@@ -99,6 +100,10 @@ Several subfields need to be specified:
 
 At the moment we support two standards, `Common Workflow Language <https://www.commonwl.org>`__ (CWL) and `Workflow Description Language <https://openwdl.org>`__ (WDL).
 
+category
+--------
+Categories for the workflow, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
+
 input
 -----
 Description of input files and parameters for the workflow. See :ref:`Input Definition <input_a>`.
@@ -109,11 +114,7 @@ Description of expected outputs for the workflow. See :ref:`Output Definition <o
 
 Optional
 ^^^^^^^^
-All the following fields are optional and provided as example. Can be expanded to anything accepted by the schema, see `schemas <https://github.com/dbmi-bgm/cgap-portal/tree/master/src/encoded/schemas>`__.
-
-title
------
-Title of the workflow.
+All the following fields are optional and provided as example. Can be expanded to anything accepted by the schema, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
 
 software
 --------