Merge branch 'develop' into genomicVariations

ga4gh-beacon · Nov 8, 2024 · c8d3a7a · c8d3a7a
2 parents 4ce7b1d + 56ff359
commit c8d3a7a
Show file tree

Hide file tree

Showing 146 changed files with 1,288 additions and 951 deletions.
diff --git a/.github/workflows/mk-docs.yaml b/.github/workflows/mk-docs.yaml
@@ -0,0 +1,19 @@
+name: mk-progenetix-docs 
+on:
+  push:
+    branches:
+      - website-docs
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.x
+      - run: pip install mkdocs-material 
+      - run: pip install mkdocs-macros-plugin
+      - run: pip install pymdown-extensions
+      - run: pip install mkdocs-mermaid2-plugin
+      - run: pip install mdx_gh_links
+      - run: mkdocs gh-deploy --force
diff --git a/.gitignore b/.gitignore
@@ -2,4 +2,3 @@ site
 .DS_Store
 models/.DS_Store
 /.vs
-docs/schemas-md
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -17,7 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
  * Updated `docs`:
     - `filters.md`
     - `variant-queries.md`
-    - `bugs-changes-log.md` 
+    - `changes-todo.md` 
     - `ComplexValue.md`
     - `README.md`
     - Added missing descriptions to models properties (see issue #42)
@@ -34,7 +34,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Fixed
 
  * Fixed `POST`queries for `g_variant` (w/ examples)
- * Removed 'json' references inside the yaml version (PR [#43] (https://github.com/ga4gh-beacon/beacon-v2/pull/43))
+ * Removed 'json' references inside the yaml version (PR [#43](https://github.com/ga4gh-beacon/beacon-v2/pull/43))
+ * added missing `type: object` to `ResultsetInstance` (PR [#82](https://github.com/ga4gh-beacon/beacon-v2/pull/82))
 
 ### Deprecated
 

diff --git a/README.md b/README.md
@@ -8,7 +8,7 @@ This repository is a unified repository representing the different parts of the
 * [models](models)
 * Beacon v2 Documentation
     - authoritive source already in this repository [`/docs`](docs)
-    - rendered version through [here](https://beacon-project.io/beacon-v2/) (alternative address is [docs.genomebeacons.org](http://docs.genomebeacons.org))
+    - rendered version through [here](https://beacon-project.io/beacon-v2/) (alternative address is [docs.genomebeacons.org](https://docs.genomebeacons.org))
 
 As with other schema projects, here we separate between the schema source files (in `src`; JSON-Schema written in YAML) and the generated versions for referencing. The current setup allows already the direct referencing of the generated JSON schemas. Examples:
 
@@ -28,12 +28,34 @@ As with other schema projects, here we separate between the schema source files
 
 There is a set of tools in [`/bin`](./bin/) to facilitate the conversion. ATM, after editing `...yaml` schema files somewhere in the `/src` tree, a (local) run of `bin/yamlerRunner.sh` - which re-generates the `....json` files in the `/json` tree) has to be performed before pushing changes.
 
-### Changes
+### Changelog
 
-* change notes with respect to the repository & documentation are now in [docs.genomebeacons.org](http://docs.genomebeacons.org/bugs-changes-log/)
+## 2.1.0
+
+*Released, July, 19, 2024*
+
+* Relocated TypedQuantity required to proper level of the schema for complexValue
+* Added end and start entities for ageRange and iso8601duration for age
+* Filtering terms scopes changed from string to array of strings
+
+## 2.0.1
+
+*Released July, 16, 2024*
+
+* Replaced ENSGLOSSARY for SO ontology family in documentation examples
+* Moved CURIE to beaconCommonComponents
+* Created filtering terms entity
+* Removed validation directories
+* Several fixes to entity types, typos and other non-breaking changes
+
+## 2.0.0
+
+*Released June, 21, 2022*
+
+* change notes with respect to the repository & documentation are now in [docs.genomebeacons.org](https://docs.genomebeacons.org/changes-todo/)
 * NOTE: on 2022-06-20 the previous development repositories have been archived:
     - ARCHIVE - [beacon-framework-v2](https://github.com/ga4gh-beacon/beacon-framework-v2)
-    - ARCHIVE - [[beacon-v2-Models](https://github.com/ga4gh-beacon/beacon-v2-Models)
+    - ARCHIVE - [beacon-v2-Models](https://github.com/ga4gh-beacon/beacon-v2-Models)
 
 
 ## Directory structure

diff --git a/bin/SCHEMAS2MD.md b/bin/SCHEMAS2MD.md
@@ -32,11 +32,11 @@ YAMLs schemas. Each time the original MS Word document was edited, someone had t
 
 This script inverts the process, i.e., **it enforces modifying the schema specification directly at the YAML/JSON level**.
 
-Editing the schemas directly at the YAML/JSON level has two advantages, the first is that because we follow [OpenAPI](https://swagger.io/specification/) specification (along with JSON schema), _a priori_ we could use [SWAGGER UI](https://swagger.io/docs/open-source-tools/swagger-ui/usage/installation). The second is that the YAML/JSON files can be converted to Markdown tables in order to create [Markdown based documentation](http://docs.genomebeacons.org) documentation. This script **transforms YAML/JSON to Markdown tables**, including their nested objects **up to a third degree of hierarchy**.
+Editing the schemas directly at the YAML/JSON level has two advantages, the first is that because we follow [OpenAPI](https://swagger.io/specification/) specification (along with JSON schema), _a priori_ we could use [SWAGGER UI](https://swagger.io/docs/open-source-tools/swagger-ui/usage/installation). The second is that the YAML/JSON files can be converted to Markdown tables in order to create [Markdown based documentation](https://docs.genomebeacons.org) documentation. This script **transforms YAML/JSON to Markdown tables**, including their nested objects **up to a third degree of hierarchy**.
 
-The **Markdown** format can be directly rendered as tables at the GitHub repository, and it can be used with [MkDocs](https://www.mkdocs.org/) to create [HTML](http://docs.genomebeacons.org) documentation. 
+The **Markdown** format can be directly rendered as tables at the GitHub repository, and it can be used with [MkDocs](https://www.mkdocs.org/) to create [HTML](https://docs.genomebeacons.org) documentation. 
 
-Everytime a `git push` is performed to the [repo](https://github.com/ga4gh-beacon/beacon-v2) the documentation at [Github Pages](http://docs.genomebeacons.org) gets updated. Note that only content under directory `docs/` will make it to [Github Pages](http://docs.genomebeacons.org).
+Everytime a `git push` is performed to the [repo](https://github.com/ga4gh-beacon/beacon-v2) the documentation at [Github Pages](https://docs.genomebeacons.org) gets updated. Note that only content under directory `docs/` will make it to [Github Pages](https://docs.genomebeacons.org).
 
 Before creating this tool, the author made an exhaustive search on what had been dveloped by the _community_ to automatically convert YAML/JSON to Markdown tables and found that there were many ways to go from YAML/JSON to HTML (e.g., CPAN, Python, Node.js), but not much from YAML/JSON to Markdown. Obviously, even in the case we had found something, some major tweaking will be needed in order to display things the way we want.
 
@@ -132,7 +132,7 @@ _NB:_ The script was built to work with the Beacon v2 Model schemas and the auth
 
 _NB:_ The decission to take YAMLs (and not JSON) as an input is deliberate and made by the author.
 
-_NB:_ The script only processes the `Terms` nested **up to 3 degrees of hierarchy**. Before Adoption of VRS/PHX that limit was OK.
+_NB:_ The script only processes the `Terms` nested **up to 3 degrees of hierarchy**. Before Adoption of VRS/PXF that limit was OK.
 
 _NB:_ The script also includes the Beacon v2 Models examples from [beacon-v2 repo](https://github.com/ga4gh-beacon/beacon-v2) in JSON format.
 

diff --git a/bin/jsonref2json.py → bin/_jsonref2json.py b/bin/jsonref2json.py → bin/_jsonref2json.py
diff --git a/bin/beacon_yaml2md.pl b/bin/beacon_yaml2md.pl
@@ -2,11 +2,11 @@
 #
 #   Script to convert Beacon v2 Models schemas to Markdown tables
 #
-#   Last Modified: May/05/2022
+#   Last Modified: Mar/26/2024
 #
 #   Version 2.0.0
 #
-#   Copyright (C) 2021-2022 Manuel Rueda (manuel.rueda@crg.eu)
+#   Copyright (C) 2021-2024 Manuel Rueda (manuel.rueda@cnag.eu)
 #
 #   This program is free software; you can redistribute it and/or modify
 #   it under the terms of the GNU General Public License as published by
@@ -236,6 +236,10 @@ sub yaml2md_obj {
         # We parse $yaml to get paths and more...
         my ( $base, $dir, $ext ) = fileparse( $yaml, '.yaml' );
         $ext =~ s/\.yaml/.md/;
+
+        # Ad hoc fix for two files that have same namex except for uc/lc
+        # AgeRange == ageRange  and Value == value on MacOS cwAPFS (Case insensitive)
+        $base = $base . '_PXF' if ( $base eq 'AgeRange' || $base eq 'Value' );
         my $file = catfile( $mo_dir, $base . $ext );    # Note -> $base.$ext
         write_file( $file, $out_str );
 
@@ -278,11 +282,11 @@ sub yaml_slicer {
     # one YAML file for each property and then re-use code from the 'main' schema
 
     ##########################################
-    # **** Note about VRS / PHX adoption *** #
+    # **** Note about VRS / PXF adoption *** #
     ##########################################
 
     # The adoption of those standards had technical implications. The script expects objects to have
-    #  <key> for the object and then <properties>. VRS/PHX follow JSON schemas that include /oneOf allOf anyOf/
+    #  <key> for the object and then <properties>. VRS/PXF follow JSON schemas that include /oneOf allOf anyOf/
     # plus other complex intructions such as <if:> <else:>.
     # This becomes a real challenge with $ref as, for instance, in <g_v.variation> we can not find the key for
     # 'MolecularVariation', 'SystemicVariation', 'LegacyVariation'
@@ -352,7 +356,7 @@ sub yaml_slicer {
 sub table_content {
 
     my ( $yaml_properties, $ra_properties, $headers, $obj, $link ) = @_;
-    my @lc_headers = map { lc } @$headers; # Copy array uc to avoid modifying original $ref
+    my @lc_headers = map { lc } @$headers;    # Copy array uc to avoid modifying original $ref
     my $out_str    = '';
 
     #---------------------------------------------------------|
@@ -394,10 +398,10 @@ sub table_content {
                   if $header eq 'example';
 
                 # Slice differentely if $object->{type} eq 'array'
-                if ($object->{type} eq 'array' ) {
-                  for ('description', 'properties'){
-                      $value_header = $object->{items}{$_} if $header eq $_;
-                  }
+                if ( $object->{type} eq 'array' ) {
+                    for ( 'description', 'properties' ) {
+                        $value_header = $object->{items}{$_} if $header eq $_;
+                    }
                 }
 
                 # Now convert data structure to string
@@ -454,7 +458,7 @@ sub ref2str {
 
         # string or undef
         else {
-            $out_str = defined $data->[0] ? join ', ', @$data : 'NA'; # Note ', ' to allow HTML column rendering
+            $out_str = defined $data->[0] ? join ', ', @$data : 'NA';    # Note ', ' to allow HTML column rendering
         }
     }
     elsif ( ref $data eq 'HASH' ) {
@@ -480,15 +484,20 @@ sub add_external_links {
     my ( $tmp_str, $key ) = @_;
 
     # Note: This is an ad hoc solution to fix errors with deeply-nested data
-    my @phx = qw( typedQuantities days weeks Quantity high low);
-    my @vrs = qw(_id state type CURIE Location);
+    my @pxf       = qw( typedQuantities days weeks Quantity high low);
+    my @vrs       = qw(_id state type CURIE Location);
     my @framework = ("ontologyTerm");
-    return ( any { ( $_ eq $key ) } @phx )
+
+    return ( any { ( $_ eq $key ) } @pxf )
       ? "[$key](https://phenopacket-schema.readthedocs.io/en/latest/building-blocks.html)"
       : ( any { ( $_ eq $key ) } @vrs )
       ? "[$key](https://vrs.ga4gh.org/en/stable/terms_and_model.html#$key)"
-      : ( any { ( $_ eq $key ) } @framework ) 
-      ? "[$key](https://github.com/ga4gh-beacon/beacon-v2/blob/main/framework/src/common/$key.yaml)" 
+      : ( any { ( $_ eq $key ) } @framework )
+      ? "[$key](https://github.com/ga4gh-beacon/beacon-v2/blob/main/framework/src/common/$key.yaml)"
+
+      # NB: Ad hoc solution for properties having equal name (lc)
+      : ( $key eq 'AgeRange' || $key eq 'Value' )
+      ? "[$key]($tmp_str/${key}_PXF.md)"
       : "[$key]($tmp_str/$key.md)";
 }
 
@@ -582,13 +591,13 @@ sub create_str_yaml {
   description: References to the tool
   examples:
     - bio.toolsId: https://bio.tools/vep
-    - url: http://www.ensembl.org/vep
+    - url: https://www.ensembl.org/vep
   type: object
 EOF
 
 ## ontologyTerm.yaml is needed due to a bug with jsonref2json.js that overrided "parent" <description> field
 
-   my $str_ontologyTerm = <<EOF;
+    my $str_ontologyTerm = <<EOF;
 ---
 additionalProperties: true
 description: Definition of an ontology term.
@@ -676,10 +685,10 @@ sub parse_json_keywords {
         'variation' =>
           [ 'MolecularVariation', 'SystemicVariation', 'LegacyVariation' ],
         'SystemicVariation'  => ['CopyNumber'],
-        'MolecularVariation' => [ 'Allele', 'Haplotype' ],
-        'location'           => [ 'CURIE', 'Location' ],
+        'MolecularVariation' => [ 'Allele',        'Haplotype' ],
+        'location'           => [ 'CURIE',         'Location' ],
         'state'              => [ 'SequenceState', 'SequenceExpression' ],
-        'Value'              => [ 'Quantity', 'ontologyTerm' ]
+        'Value'              => [ 'Quantity',      'ontologyTerm' ]
     };
 
     # We'll be checking <oneOf allOf anyOf>
@@ -699,14 +708,17 @@ sub parse_json_keywords {
                 #  my $const = $pointer->get("/$keyword/$property/$count/properties/type/const");
                 #   $tmp_hash->{properties}{$const} = $elements;
                 #} else{
-                my $tmp_term = ( $pointer->contains("/$keyword/$count/title") && $pointer->get("/$keyword/$count/title") ne 'Ontology Term' )
+                my $tmp_term =
+                  (      $pointer->contains("/$keyword/$count/title")
+                      && $pointer->get("/$keyword/$count/title") ne
+                      'Ontology Term' )
                   ? $pointer->get("/$keyword/$count/title")
                   : @{ $terms->{$property} }[$count];
-                $tmp_hash->{properties}{$tmp_term} = $elements if $tmp_term; # Ad-hoc some terms appear duplicated and come empty....
-                                                                             #}
+                $tmp_hash->{properties}{$tmp_term} = $elements if $tmp_term;   # Ad-hoc some terms appear duplicated and come empty....
+                                                                               #}
                 $count++;
             }
-            $data = $tmp_hash;    # Adding new reference
+            $data = $tmp_hash;                                                 # Adding new reference
         }
     }
     return $data;
@@ -769,11 +781,11 @@ =head1 MOTIVATION
 
 This script inverts the process, i.e., B<it enforces modifying the schema specification directly at the YAML/JSON level>.
 
-Editing the schemas directly at the YAML/JSON level has two advantages, the first is that because we follow L<OpenAPI|https://swagger.io/specification/> specification (along with JSON schema), I<a priori> we could use L<SWAGGER UI|https://swagger.io/docs/open-source-tools/swagger-ui/usage/installation>. The second is that the YAML/JSON files can be converted to Markdown tables in order to create L<Markdown based documentation|http://docs.genomebeacons.org> documentation. This script B<transforms YAML/JSON to Markdown tables>, including their nested objects B<up to a third degree of hierarchy>.
+Editing the schemas directly at the YAML/JSON level has two advantages, the first is that because we follow L<OpenAPI|https://swagger.io/specification/> specification (along with JSON schema), I<a priori> we could use L<SWAGGER UI|https://swagger.io/docs/open-source-tools/swagger-ui/usage/installation>. The second is that the YAML/JSON files can be converted to Markdown tables in order to create L<Markdown based documentation|https://docs.genomebeacons.org> documentation. This script B<transforms YAML/JSON to Markdown tables>, including their nested objects B<up to a third degree of hierarchy>.
 
-The B<Markdown> format can be directly rendered as tables at the GitHub repository, and it can be used with L<MkDocs|https://www.mkdocs.org/> to create L<HTML|http://docs.genomebeacons.org> documentation. 
+The B<Markdown> format can be directly rendered as tables at the GitHub repository, and it can be used with L<MkDocs|https://www.mkdocs.org/> to create L<HTML|https://docs.genomebeacons.org> documentation. 
 
-Everytime a C<git push> is performed to the L<repo|https://github.com/ga4gh-beacon/beacon-v2> the documentation at L<Github Pages|http://docs.genomebeacons.org> gets updated. Note that only content under directory C<docs/> will make it to L<Github Pages|http://docs.genomebeacons.org>.
+Everytime a C<git push> is performed to the L<repo|https://github.com/ga4gh-beacon/beacon-v2> the documentation at L<Github Pages|https://docs.genomebeacons.org> gets updated. Note that only content under directory C<docs/> will make it to L<Github Pages|https://docs.genomebeacons.org>.
 
 Before creating this tool, the author made an exhaustive search on what had been dveloped by the I<community> to automatically convert YAML/JSON to Markdown tables and found that there were many ways to go from YAML/JSON to HTML (e.g., CPAN, Python, Node.js), but not much from YAML/JSON to Markdown. Obviously, even in the case we had found something, some major tweaking will be needed in order to display things the way we want.
 
@@ -872,7 +884,7 @@ =head1 HOW TO RUN BEACON_YAML2MD
 
 I<NB:> The decission to take YAMLs (and not JSON) as an input is deliberate and made by the author.
 
-I<NB:> The script only processes the C<Terms> nested B<up to 3 degrees of hierarchy>. Before Adoption of VRS/PHX that limit was OK.
+I<NB:> The script only processes the C<Terms> nested B<up to 3 degrees of hierarchy>. Before Adoption of VRS/PXF that limit was OK.
 
 I<NB:> The script also includes the Beacon v2 Models examples from L<beacon-v2 repo|https://github.com/ga4gh-beacon/beacon-v2> in JSON format.
 

diff --git a/bin/deref_schemas/datasets/defaultSchema.json b/bin/deref_schemas/datasets/defaultSchema.json
@@ -247,7 +247,7 @@
         "externalUrl": {
             "description": "URL to an external system providing more dataset information (RFC 3986 format).",
             "examples": [
-                "http://example.org/wiki/Main_Page"
+                "https://example.org/wiki/Main_Page"
             ],
             "type": "string"
         },

diff --git a/bin/deref_schemas/datasets/defaultSchema.yaml b/bin/deref_schemas/datasets/defaultSchema.yaml
@@ -165,7 +165,7 @@ properties:
   externalUrl:
     description: URL to an external system providing more dataset information (RFC 3986 format).
     examples:
-      - http://example.org/wiki/Main_Page
+      - https://example.org/wiki/Main_Page
     type: string
   id:
     description: Unique identifier of the dataset