beacon_yaml2md: Script to convert Beacon v2 Models schemas to Markdown tables
beacon_yaml2md.pl
Options:
-schema-dir Directory with YAML schemas
-markdown-dir Directory where to write .md
-D|delete Delete schema-dir/obj/*yaml
-h|help Brief help message
-man Full documentation
-debug Print debugging (from 1 to 5, being 5 max)
-verbose Verbosity on
To be defined.
beacon_yaml2md: Script to convert Beacon v2 Models schemas to Markdown tables
NOTE: This script is UNDER CONSTRUCTION and improvements are added often.
Prior to the creation of this script (i.e.,June-2021), the Beacon documentation for the schemas was first created in a text-type document (e.g., Word) and then MANUALLY transformed into YAMLs schemas. Each time the original MS Word document was edited, someone had to manually edit the YAML.
This script inverts the process, i.e., it enforces modifying the schema specification directly at the YAML/JSON level.
Editing the schemas directly at the YAML/JSON level has two advantages, the first is that because we follow OpenAPI specification (along with JSON schema), a priori we could use SWAGGER UI. The second is that the YAML/JSON files can be converted to Markdown tables in order to create Markdown based documentation documentation. This script transforms YAML/JSON to Markdown tables, including their nested objects up to a third degree of hierarchy.
The Markdown format can be directly rendered as tables at the GitHub repository, and it can be used with MkDocs to create HTML documentation.
Everytime a git push is performed to the repo the documentation at Github Pages gets updated. Note that only content under directory docs/ will make it to Github Pages.
Before creating this tool, the author made an exhaustive search on what had been dveloped by the community to automatically convert YAML/JSON to Markdown tables and found that there were many ways to go from YAML/JSON to HTML (e.g., CPAN, Python, Node.js), but not much from YAML/JSON to Markdown. Obviously, even in the case we had found something, some major tweaking will be needed in order to display things the way we want.
In the Beacon context, mbaudis has developed a nice framework for schemablocks which creates HTML from YAML schemas to be used with Yekyll. However, personalizing his tools to work in our scenario will still not solve the initial objective of creating Markdown tables from the YAML/JSON.
All of the above lead to the creation of this tool, which was written in Perl language.
There are several steps that need to be peformed to update the documentation:
1 - Install C<nodejs>.
$ sudo apt install nodejs
The reason for installing JS is that we need it to convert JSON files with JSON references ($ref) to JSON files with no references (i.e., all references will be embeded in the file).
2 - Now run the script:
$ # cd bin # skip this step if you are already at bin directory
$ ./transform_json2md.sh
$ cd ..
$ git add docs/schemas-md
3 - Finally you need to push beacon-v2 repo to GitHub.
4 - Documentation will get automatically updated via GitHub actions.
The script is written in Perl and runs on Linux (tested on Debian-based distribution). Perl 5 is installed by default on Linux, but you might need to install some CPAN module(s).
First we install cpanminus (with sudo privileges):
$ sudo apt-get install cpanminus
Then the module(s):
$ cpanm --sudo --installdeps .
If you prefer to have the dependencies in a "virtual environment" (i.e., install the CPAN modules in the directory of the application) we recommend using the module Carton.
$ cpanm --sudo Carton
Then, we can install our dependencies:
$ carton install
The script takes YAMLs as input file and when no arguments is used it will read them from ./deref_schemas/ directory.
NB: We recommend running the script with the provided bash file transform_yaml2md.sh (see ADDENDUM: How to update Documentation).
Example 1:
$ ./beacon_yaml2md.pl
$ carton exec -- ./beacon_yaml2md.pl # if using Carton
If the script is run directly at bin/ directory (default) and with no arguments, then it will create contents in:
* Markdowns at ../docs/schemas-md/
Example 2:
Here we are providind arguments for --schema-dir and for markdown-dir.
$ $path/beacon_yaml2md.pl --schema-dir ./deref_schemas --markdown-dir ../docs/schemas-md --D
NB: if the YAML is not well defined (e.g., wrong indentation, etc.) the script will complain about it. Thus, it indirectly serves as a YAML validator.
Notes:
The argument -D|delete deletes all ./deref_schemas/obj/*yaml files prior to doing anything else. It's harmless so you may want to use it you're unsure of who created those files.
See the directory tree below:
```
beacon-v2/
|-- bin
|-- docs
| |-- img
| `-- schemas-md
| `-- obj
```
NB: The script was built to work with the Beacon v2 Model schemas and the author deliberately chose to hard-code some variables (e.g., the schemas) to avoid extra arguments or a config.file. However, this may change in future versions depending on user's adoption/feedback.
NB: The decission to take YAMLs (and not JSON) as an input is deliberate and made by the author.
NB: The script only processes the Terms nested up to 3 degrees of hierarchy. Before Adoption of VRS/PXF that limit was OK.
NB: The script also includes the Beacon v2 Models examples from beacon-v2 repo in JSON format.
Written by Manuel Rueda, PhD. Info about EGA can be found at https://ega-archive.org/.
* If two entities share a term/object (e.g., id, label) the text will correspond to the latest one (alphabetical order). "Ideally" all shared-objects should have a common description, shouldn't them? This way we avoid having to create sub-objects for every entity.
* As of April 2022 we're only processing 3 levels of nesting and adding external URLs for deeper levels.
* We are not processing <if: then:> statements in JSON Schema.
Report bugs or comments to [email protected].
This PERL file is copyrighted. See the LICENSE file included in this distribution.