gradle vodmlDoc
+will generate standard documentation into the directory build/generated/docs/vodml/ (this can be changed with the outputDocDir setting)
This will produce a model diagram, latex and html formatted documentation, as well as a graphml representation of the model +that can be hand edited with yEd for nicer looking model diagrams.
+gradle vodmlSite
+Will generate a whole static site describing the model that is intended to be +further processed with mkdocs tool that is configured with the material theme.
+The site is generated at in the build/generated/docs/vodml-site/ directory (which can be
+changed with the outputSiteDir setting). If you are creating full documentation site,
+then it is likely that you will want to add content other than the autogenerated model
+description. In this case the tools have an assumption that the outputSiteDir is set to a
+generated sub-directory of your configured mkdocs top-level directory.
The plugin will create an allnav.yaml file in the project directory which can be added to the mkdocs navigation using yq by adding the following to the build.gradle.kts
tasks.register<Exec>("siteNav")
+{
+ commandLine("yq", "-i", "(.nav.[]|select(has(\"AutoGenerated Documentation\"))|.[\"AutoGenerated Documentation\"]) += load(\"allnav.yml\")", "mkdocs.yml")
+ dependsOn("vodmlSite")
+}
+which in turn assumes that there is an "AutoGenerated Documentation" section in the mkdocs nav that can be added to.
+nav:
+ - Home: index.md
+ - AutoGenerated Documentation:
+ - Javadoc: generated/javadoc
+The DataModel Template has an example setup.
+ + + + + + + + + + + + + +The VO-DML tooling is based around gradle (currently version 8) which itself
+is based on Java. It is recommended that a minimum of JDK 17 is installed
+using a package manager for your OS and
+similarly use a package manager for gradle installation. Although if you are working
+with a repository that already has a gradelw file at the top level, then that can be used
+in place of the gradle command, and it will handle the downloading and running of the correct gradle version.
The functionality of the tooling is then encapsulated with a gradle plugin which +is configured in the quickstart instructions
+Note the documentation tasks of the tools that produce the overall model diagram also require that graphviz be installed.
+If full documentation site generation is required then mkdocs material theme is needed as an external installation dependency along with yq that can be used to automate the mkdocs navigation menu creation.
+ + + + + + + + + + + + + +The tooling is capable of generating Java classes that can be used to store instances of the +models. The code is annotated to allow JAXB and JPA to operate, which mean that it is easy to +read and write model instances to XML and standard relational databases. It should be noted that +the generated code uses java 1.8 constructs.
+The generated Java code depends on the VO-DML java runtime library, which the plugin will automatically add to the +dependencies along with the necessary JAXB and JPA libraries.
+The command +
gradle vodmlJavaGenerate
+./build/generated/sources/vodml/java which is on the build classpath and so
+gradle build
+gradle build will
+automatically run gradle vodmlJavaGenerate if the code is out of date because the model has been updated.
+In general the code creates POJOs or data classes - i.e. the classes have very little functionality +apart from being stores of the data model. The functionality that they do have is described below.
+To be JPA and JAXB compliant, the classes are Java beans with a no argument constructor, so they can be +default constructed and then getX/setX can be used on the properties.
+A a = new A();
+a.setX(x);
+a.setY(y);
+In addition, there are several other ways of creating objects
+A constructor with all the possible properties included is
+A a = new A(x,y);
+Each property has a withX(X x) function
+A a = new A().withX(x).withY(y);
+A static builder that takes a functional argument.
+A a = A.createA( b -> {
+ b.x = x;
+ b.y = y;
+ }
+ );
+As well as all the individual Enum, ObjectType, and DataType classes, there is +an overall ${modelname}Model class generated, that is intended to act as +a 'container' for the individual elements. This is especially useful in the case +where the model has references, as then there are some convenience methods for dealing with the +automatic setting of reference IDs if necessary.
+For each of the concrete objectTypes in the model there is
+an overloaded addContent() method, which will add the content to the
+overall instance and find any references.
Once all the content has been added, then there is a processReferences() method
+which will go through the whole model and automatically assign IDs for any
+references that do not already have them.
At this point the overall model object is suitable to be serialized.
+On reading in a model instance, it is possible to extract any top level ObjectTypes
+with the public <T> List<T> getContent(Class<T> c) method.
contained references
+There is some provisional support for "contained references" when cloning an object - the API for this
+is subject to change, but an example is used in copyTest using the createContext() and updateClonedReferences() methods either side of an object clone with a copy constructor.
Finally, there is a static public static boolean hasReferences() method which
+can be used to check if the model has any references - if it does not then much of the
+machinery above (apart from the JAXBContext) is necessary, and individual ObjectTypes may be
+written.
The unit tests for this project show most of the various code features being used
+A static function to create a suitable JAXBContext is present +
JAXBContext jc = MyModel.contextFactory();
+The JSON serialization is implemented with the Jackson library
+A suitable ObjectMapper is obtained with +
ObjectMapper mapper = MyModel.jsonMapper();
+The generated code has JPA annotations to allow storing in RDBs with compliant systems.
+Operations are not in general cascaded into references - so that the references need to be explicitly managed. In most cases this will be the "natural" way to do things
+for the model - however at creation time it might be inconvenient to do this so that there is a method
+persistRefs(jakarta.persistence.EntityManager _em) on the model that will do a deep persist of any references in the child objects, which in turn then will allow an error free persist of the content. Note that in general it is only possible to run the persistRefs once all of the content has been added to the model, and only for the first time a reference is created - for subsequent updates of the model it will be necessary to manage the references manually.
In general collections are marked for lazy loading, and as a convenience there is a forceLoad()
+method generated that will do a deep walk of all the collections in a particular type, which will force the loading of the whole instance tree if that is desired.
This extra JPA functionality is described by the JPAManipulations interface.
+Some convenience methods are created on the object that is the parent of compositions to make dealing with JPA updates easier.
+class X {
+ public void replaceInY(final Y _p) {...}
+}
+Internally this makes use of another convenience function
+class Y {
+ public void updateUsing ( final Y other){...}
+}
+The most natural way to represent dataTypes in JPA is as embeddable, this means that they do +not have a separate "identity" and are simply represented as columns within the parent entity table. +The problem with this is that JPA does not specifically allow inheritance of embeddables (though nor does it disallow the use). +As a consequence the support for inherited embeddables is not uniform in JPA providers.
+Hibernate seems to support the concept of embeddable hierarchies reasonably well by
+naturally using the @MappedSuperclass annotation - although there is an irritation in that
+the full flexibility of having optional attributes that are dataTypes is not supported as all columns are
+made non-nullable - a bug has been submitted https://hibernate.atlassian.net/browse/HHH-14818
There are also eclipselink bugs that mean that the suggested way of doing inherited embeddables does not seem to work.
+The java runtime has a number of base classes that aid the testing of model instances - there is an example for the mock coords model. +Although it is not obvious from the source code presented because most of the behaviour is inherited +from the base test class, this test will actually
+simply by running
+gradle test
+will generate the actual model code (if not already done) and run the tests as long as
+tasks.test {
+ useJUnitPlatform()
+}
+build.gradle.kts file.
+Much of the functionality described above is defined in two interfaces
+ModelManagement an
+instance of which can be obtained with the management() method on the model class and
+ModelDescription an
+instance of which can be obtained with the description() method on the model class.
+These interfaces allow generic model handling code to be written.
NB The python code generation should currently be regarded as alpha quality in that it does not serialize model instances to something that is interoperable with the Java generated code. Indeed it is not yet guaranteed to be representing the VO-DML model fully.
+The command +
gradle vodmlPythonGenerate
+./build/generated/sources/vodml/python directory by default (changeable with the outputPythonDir vodml setting).
+The code uses xsdata for XML and JSON serialization and SQLAlchemy for the RDB serialization.
+Although the code generation itself does not need python, to run any tests in the project +relies on (https://github.com/xvik/gradle-use-python-plugin) to manage python virtualenv (i.e. https://virtualenv.pypa.io/en/latest/index.html - not all the other ones). This creates the virtualenv +in venv directory at top level - it does not seem to do that properly, so it is best to do that manually beforehand...
+gradle pipInstall
+Depends on python 3.10+
+The project is set up with a single python test at the moment that can be run with
+gradle pytest
+Latest published version
In general, a new data model should be started in its own git repository and configured +as below (see ProposalDM for a complete example separate from this repository). +If starting a completely new data model then the Template DM Project is probably the easiest way to get going.
+If adapting an existing data model repository then
+
build.gradle.kts file with reference to the plugin (note substitute 
below)
+plugins {
+ id("net.ivoa.vo-dml.vodmltools") version "0.x.x"
+}
+settings.gradle.kts - it is possible just to copy the template version and just edit the rootProject.name.There is nothing else that needs to be done if the VO-DML files in the default place +(see sample build file for some more +hints on how gradle flexibility allows finding of the files +to be configured in a variety of ways).
+If the configuration is successful then
+gradle vodmlValidate
+If the validation is successful you can produce various derived products. Developing your VO-DML model further is discussed here.
+The vodml tools are all configured within a +
vodml {
+
+}
+build.gradle.kts file.
+The various sub-properties that can be set are
+src/main/vo-dml
+ vodmlDir.set(file("vo-dml"))
+vo-dml*.vo-dml.xml files in the vodmlDir, but can be individually overridden.bindingFiles - the files that specify the mapping details between the models and the generated code. N.B. there is no default for this setting - if generating code it must be specified.
+outputDocDir - where the generated documentation is created by the gradle vodmlDoc command- default build/generated/docs/vodml/.
gradle vodmlSite command - default build/generated/docs/vodml-site.build/generated/sources/vodml/java/ and it should not
+ be necessary to ever alter this as gradle will integrate this automatically into the various source paths.build/generated/sources/vodml/schema/ - this is automatically included in the classpath and the output jar.<?xml version="1.0"?>
+<catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
+ <group prefer="system" >
+ <uri
+ name="IVOA-v1.0.vo-dml.xml"
+ uri="src/main/vo-dml/IVOA-v1.0.vo-dml.xml"/>
+ </group>
+</catalog>
+The binding file is used to set up some properties for various functions on the model - the +most basic of which is that it provides a connection between the model name and the filename +which contains the VO-DML model.
+<m:mappedModels xmlns:m="http://www.ivoa.net/xml/vodml-binding/v0.9.1">
+<!-- ========================================
+This is a minimal sample file for mapping VO-DML models to XSD or Java using the gradle tooling
+ -->
+
+<model>
+<name>sample</name>
+<file>Sample.vo-dml.xml</file>
+<java-package>org.ivoa.dm.sample</java-package>
+<xml-targetnamespace prefix="simp" >http://ivoa.net/dm/models/vo-dml/xsd/sample/sample</xml-targetnamespace>
+</model>
+</m:mappedModels>
+The schema for the binding file shows what elements are allowed. The binding file for the base IVOA model +shows extensive use of the binding features, where it is possible to ignore the automated code generation entirely and substitute +hand-written code.
+ + + + + + + + + + + + + +The design choices made in the serialization of the models to various formats has been driven by the capabilities +of the various libraries/standards that are applicable to the Java generated code. Other language implementations +will follow the choices that were made in order to be interoperable.
+For the serializations (other than the relational database), the general idea is that a "natural" serialization for the model +has been chosen, in contrast to the approach of MIVOT +where the idea is that the model is coerced into a table based model - which of course is similar to the relational +database serialization below. This "natural" serialization means that objects are enclosed within their parents to whatever depth +is necessary. The only exception to this is that referenced objects that are not otherwise contained within another object in the model are separated out into their own section early in the +serialization so that they can easily be referenced.
+classDiagram
+ Model <-- References
+ Model <-- ContentThe aim of the top level container object is to contain all the referred to objects as well as the general content within +a single document. References external to the model would need to be incorporated into the model design with an explicit external reference (hence this proposal for VO-DML 1.1 and ). It should be noted that this is a different methodology to the way that the +XML was produced in the previous (ant based) versions of this tooling, and as such the XML target namespaces have been changed.
+The schema for the serializations can be created with the gradle vodmlSchema command (more detail).
For the small example model, the overall model object will produce xml like
+<ser:myModelModel xmlns:ser="http://ivoa.net/vodml/sample/serialization" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >
+ <refs>
+ <refa _id="MyModel-Refa_1000">
+ <val>a value</val>
+ </refa>
+ <refb>
+ <name>naturalkey</name>
+ <val>another val</val>
+ </refb>
+ </refs>
+ <someContent>
+ <zval>some</zval>
+ <zval>z</zval>
+ <zval>values</zval>
+ <con xsi:type="ser:Dcont" >
+ <bname>dval</bname>
+ <dval>a D</dval>
+ </con>
+ <con xsi:type="ser:Econt" >
+ <bname>eval</bname>
+ <evalue>cube</evalue>
+ </con>
+ <ref1>MyModel-Refa_1000</ref1>
+ <ref2>naturalkey</ref2>
+ </someContent>
+</ser:myModelModel>
+JSON does not natively have an equivalent to the XML-ID/IDREF mechanism, however, it is possible to distinguish between
+a named object as the value of a field or a string or integer literal value for the same field which could be interpreted
+as a reference to the object if the data model is known. An _id property is added to referenced types which can then used to make a reference. Unlike the XML case where the ID values need to be unique across the whole document, for the JSON serialization the IDs need only be unique for each type - this makes it somewhat easier to use the native database keys directly.
{
+ "MyModelModel" : {
+ "refs" : {
+ "MyModel:Refa" : [ {
+ "_id" : 1000,
+ "val" : "a value"
+ } ],
+ "MyModel:Refb" : [ {
+ "name" : "naturalkey",
+ "val" : "another val"
+ } ]
+ },
+ "content" : [ {
+ "@type" : "MyModel:SomeContent",
+ "_id" : 0,
+ "zval" : [ "some", "z", "values" ],
+ "con" : [ {
+ "@type" : "MyModel:Dcont",
+ "_id" : 0,
+ "bname" : "dval",
+ "dval" : "a D"
+ }, {
+ "@type" : "MyModel:Econt",
+ "_id" : 0,
+ "bname" : "eval",
+ "evalue" : "cube"
+ } ],
+ "ref1" : 1000,
+ "ref2" : "naturalkey"
+ } ]
+ }
+}
+@type with the UType as value is added.
+The object relational mapping has been done with the capabilities offered by JPA. The general design +decisions that have been made for the mapping are.
+The default inheritance strategy is "JOINED" - which means that there will be a table per sub-type that has to be joined. This strategy the default as it allows for the widest application of "NOT NULL" constraints within the database, at the expense of more complex joins being required. As an alternative a "SINGLE_TABLE" strategy can be adopted, by specifying +
<rdb inheritance-strategy="single-table"/>
+DataTypes become embedded as extra columns within the table.
+Generating the actual DDL for the database does necessarily depend on some differences between vendors. +However, running the test will produce DDL.
+ + + + + + + + + + + + + +A VO-DML transformation is something that takes the model and expresses it in another way.
+The most basic form of transformation is to make human-readable documentation.
+Here the model is transformed into source code in various languages, which can then be used to hold instances +of the data model and then serialize in various formats - currently supported
+The languages supported are;
+ +The models are also transformed into schema that describe the various serializations. The overall aim of the +VO-DML tooling is to be able to exchange instances of the models between different computer languages, with +all the source code and schema automatically generated.
+The gradle task
+gradle vodmlSchema
+outputSchemaDir property (default build/generated/sources/vodml/schema/).
+These schema files will automatically be included within the jar file for the model, so that instance validation can be automatically be done without reference to external files.
+The transformation of other data model representations to VO-DML is discussed elsewhere.
+ + + + + + + + + + + + + +