updated docs and deploy code. Added flag to specify a single version …

…file to use for all the repos deployed

docs/contribute_pipeline.rst

@@ -38,7 +38,7 @@ Each step codified as a workflow description file will execute on a single EC2 m
 
 *Note: the workflow description file must have a .wdl or .cwl extension to be recognized during the automated deployment.*
 
-The following example implement the steps ``foo`` and ``bar`` for the ``foo_bar`` pipeline.
+The following example implements the steps ``foo`` and ``bar`` for the ``foo_bar`` pipeline.
 Each step will execute independently on a single EC2 machine.
 
 ::
@@ -54,10 +54,10 @@ Typically, when creating a workflow description file, the code will make referen
 To store these containers, we use private ECR repositories that are specific to each AWS account.
 To ensure that the description file points to the appropriate image, we utilize two placeholders, **VERSION** and **ACCOUNT**,
 which will be automatically substituted in the file with the relevant account information during deployment.
-If the code runs Sentieon software and requires the *SENTIEON_LICENSE* environmental variable to be set,
+If the code runs Sentieon software and requires the *SENTIEON_LICENSE* environment variable to be set,
 the **LICENSEID** placeholder will be substituted by the code with the server address provided to the :ref:`deploy command <pipeline_deploy>`.
 
-Example of a CWL code with the placeholders
+Example of a CWL code with the placeholders:
 
 .. code-block:: yaml
 

docs/deploy_pipeline.rst

@@ -14,7 +14,7 @@ This step may be skipped if you have an EC2 already set up.
 
 We recommend using the following configuration:
 
-  * AMI: Use a linux distribution (64-bit, x86)
+  * AMI: Use a Linux distribution (64-bit, x86)
   * Instance Type: t3.large or higher
   * Storage: 50+GB in main volume
 
@@ -87,7 +87,7 @@ Once the virtual environment is set up and activated, we can proceed to :ref:`in
 To check that the software is correctly installed, try to run ``smaht_pipeline_utils``.
 If installed from source, this command may fail with a bash "command not found" error, try ``poetry run smaht_pipeline_utils`` instead.
 
-Set Up Credentials and Environmental Variables
+Set Up Credentials and Environment Variables
 ==============================================
 
 .. _auth_vars:
@@ -96,7 +96,7 @@ AWS Auth Credentials
 --------------------
 
 To deploy pipelines components in a specific AWS account,
-we need to setup the following environmental variables to authenticate to the account.
+we need to setup the following environment variables to authenticate to the account.
 
 .. code-block:: bash
 

docs/pipeline_utils.rst

@@ -17,7 +17,7 @@ Usage:
 pipeline_deploy
 +++++++++++++++
 
-Utility to automatically deploy pipeline's components from a target repository.
+Utility to automatically deploy a pipeline's components from a target repository.
 It is possible to specify multiple target repositories to deploy multiple pipelines at the same time.
 
 Usage:
@@ -88,5 +88,7 @@ Usage:
     - Validate YAML objects against schemas. Turn off DEPLOY | UPDATE action
   * - *-\-sentieon-server*
     - Address for Sentieon license server
+  * - *-\-version-file*
+    - Path to VERSION file to use. This will override the version for all the repositories
   * -
     -

docs/repo.rst

@@ -4,7 +4,7 @@
 Pipeline's Repository Structure
 ===============================
 
-To be picked up correctly by some of the commands, a repository needs to be set up as follow:
+To be picked up correctly by some of the commands, a repository needs to be set up as follows:
 
 - A **descriptions** folder to store workflow description files (CWL and WDL).
 - A **dockerfiles** folder to store Docker images.

docs/yaml_file_format.rst

@@ -26,6 +26,8 @@ Template
     secondary_formats:
       - <format>              # bam, fastq, bwt, ...
 
+    file_types:
+      - <string>              # ReferenceFile, OutputFile, ...
 
 Fields Definition
 +++++++++++++++++
@@ -54,3 +56,7 @@ secondary_formats
 -----------------
 List of secondary ``<format>`` available for the file format.
 Each ``<format>`` needs to match a file format that has been previously defined.
+
+file_types
+----------
+List of file types that can be associated with the file format, see schemas for available options.

docs/yaml_file_reference.rst

@@ -63,6 +63,14 @@ version
 -------
 Version of the reference file.
 
+category
+--------
+Categories for the pipeline, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
+
+type
+----
+Types for the pipeline, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.
+
 Optional
 ^^^^^^^^
 All the following fields are optional and provided as example. Can be expanded to anything accepted by the schema, see `schemas <https://github.com/smaht-dac/smaht-portal/tree/main/src/encoded/schemas>`__.

docs/yaml_workflow.rst

@@ -159,8 +159,8 @@ For a **QC** (Quality Control) output, the argument type is defined as ``qc``.
 For a QC, it is possible to generate two different types of output: a key-value pairs JSON file and a compressed file.
 The JSON file can be used to create a summary report of the quality metrics generated by the QC process.
 The compressed file can be used to store the original output for the QC, including additional data or graphs.
-Both the JSON file and compressed file will be attached to the file specified as target by ``argument_to_be_attached_to`` with a ``QualityMetric`` object.
-The content of the JSON file will be patched directly on the object, while the compressed file will be made available for download via a link.
+Both the JSON file and compressed file will be attached to the file specified as the target by ``argument_to_be_attached_to`` with a ``QualityMetric`` object.
+The contents of the JSON file will be patched directly on the object, while the compressed file will be made available for download via a link.
 The output type can be specified by setting ``json: True`` or ``zipped: True`` in the the QC output definition.
 
 Template for key-value pairs JSON:

pipeline_utils/__main__.py

@@ -64,6 +64,7 @@ def main(args=None):
     pipeline_deploy_parser.add_argument('--post-wfl', action='store_true', help='Upload Workflow Description files (.cwl, .wdl)')
     pipeline_deploy_parser.add_argument('--post-ecr', action='store_true', help='Build Docker container images and push to AWS ECR. By default will use AWS CodeBuild unless --local-build flag is set')
 
+    pipeline_deploy_parser.add_argument('--version-file', required=False, help='Path to version file to use. This will override the version for all the repositories')
     pipeline_deploy_parser.add_argument('--debug', action='store_true', help='Turn off POST|PATCH action')
     pipeline_deploy_parser.add_argument('--verbose', action='store_true', help='Print the JSON structure created for the objects')
 

pipeline_utils/pipeline_deploy.py

@@ -56,17 +56,19 @@ class PostPatchRepo(object):
     """Class to handle deployment of pipeline components.
     """
 
-    def __init__(self, args, repo, version='VERSION', pipeline='PIPELINE'):
+    def __init__(self, args, repo, version_file='VERSION', pipeline_file='PIPELINE', version=None):
         """Constructor method.
 
             :param args: Command line arguments
             :type args: object returned by ArgumentParser.parse_args() method
             :param repo: Name of the repository
             :type repo: str
-            :param version: Name of the file storing pipeline version information
+            :param version_file: Name of the file storing pipeline version information
+            :type version_file: str
+            :param pipeline_file: Name of the file storing pipeline name information
+            :type pipeline_file: str
+            :param version: Pipeline version to use
             :type version: str
-            :param pipeline: Name of the file storing pipeline name information
-            :type pipeline: str
         """
         # Init attributes
         self.ff_key = None
@@ -103,10 +105,13 @@ def __init__(self, args, repo, version='VERSION', pipeline='PIPELINE'):
             setattr(self, key, val)
 
         # Get pipeline version
-        with open(f'{self.repo}/{version}') as f:
-            self.version = f.readlines()[0].strip()
+        if not version:
+            with open(f'{self.repo}/{version_file}') as f:
+                self.version = f.readlines()[0].strip()
+        else:
+            self.version = version
         # Get pipeline name
-        with open(f'{self.repo}/{pipeline}') as f:
+        with open(f'{self.repo}/{pipeline_file}') as f:
             self.pipeline = f.readlines()[0].strip()
 
         # Load credentials
@@ -466,7 +471,12 @@ def main(args):
             error = 'MISSING ARGUMENT, --post-wfl | --post-workflow | --post-ecr requires --region argument.\n'
             sys.exit(error)
 
+    # Get override version if flag is set
+    if args.version_file:
+        with open(args.version_file) as f:
+            version = f.readlines()[0].strip()
+    else: version = None
     # Run
     for repo in args.repos:
-        pprepo = PostPatchRepo(args, repo)
+        pprepo = PostPatchRepo(args, repo, version=version)
         pprepo.run_post_patch()