diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 8e8bb0c2..e28c45fc 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -26,23 +26,30 @@ jobs: java-version: '11' distribution: 'temurin' - name: Build java runtime with Gradle - uses: gradle/gradle-build-action@v2.1.3 + uses: gradle/actions/setup-gradle@v3 with: cache-disabled: false arguments: | :java:publishToMavenLocal -PskipSigning=True - name: Build ivoa model with Gradle - uses: gradle/gradle-build-action@v2.1.3 + uses: gradle/actions/setup-gradle@v3 with: cache-disabled: false arguments: | :ivoa:publishToMavenLocal -PskipSigning=True - name: Test sample model with Gradle - uses: gradle/gradle-build-action@v2.1.3 + uses: gradle/actions/setup-gradle@v3 with: cache-disabled: false arguments: | - :gradletooling:vodml-sample:test + :sample:test + -PskipSigning=True + - name: Validate sample model with Gradle + uses: gradle/actions/setup-gradle@v3 + with: + cache-disabled: false + arguments: | + :sample:vodmlValidate -PskipSigning=True diff --git a/README.md b/README.md index d7b80c1d..f4ca5faa 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,8 @@ language. There is a [Guide to creating models with VO-DML tooling](https://ivoa.github.io/vo-dml/) +[![PDF-Preview](https://img.shields.io/badge/VODML_WD_1.1-PDF-blue)](../../releases/download/WD-1.1/VO-DML.pdf) + The various subdirectories * [doc](./doc) : The IVOA standard document source and the vodml tools guide. @@ -21,5 +23,6 @@ The various subdirectories ![main test](https://github.com/ivoa/vo-dml/actions/workflows/test.yml/badge.svg) ![site build](https://github.com/ivoa/vo-dml/actions/workflows/site.yml/badge.svg) [![gradle plugin](https://img.shields.io/gradle-plugin-portal/v/net.ivoa.vo-dml.vodmltools?label=gradle%20plugin)](https://plugins.gradle.org/plugin/net.ivoa.vo-dml.vodmltools) +[![Standard Document PDF Preview generation](https://github.com/ivoa/vo-dml/actions/workflows/std.yml/badge.svg)](https://github.com/ivoa/vo-dml/actions/workflows/std.yml) _note that this project was moved from https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/_ diff --git a/build.gradle.kts b/build.gradle.kts index 7496cafb..1e195346 100644 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -1,10 +1,10 @@ //IMPL cannot get this composite build to work - have resorted to sequence of commands in github CI -tasks.register("doAll"){ - dependsOn(gradle.includedBuild("ivoa").task(":jar")) - finalizedBy(gradle.includedBuild("gradletooling").task(":vodml-sample:test")) - description = "builds and installs the runtime library and then runs unit tests on code generated from sample model" -} +//tasks.register("doAll"){ +// dependsOn(gradle.includedBuild("ivoa").task(":jar")) +// finalizedBy(gradle.includedBuild("gradletooling").task(":vodml-sample:test")) +// description = "builds and installs the runtime library and then runs unit tests on code generated from sample model" +//} // would like rto do something like this //tasks.register("doAll"){ diff --git a/doc/guide/JavaCodeGeneration.md b/doc/guide/JavaCodeGeneration.md index b3fd8364..f3694d76 100644 --- a/doc/guide/JavaCodeGeneration.md +++ b/doc/guide/JavaCodeGeneration.md @@ -98,7 +98,7 @@ At this point the overall model object is suitable to be serialized. On reading in a model instance, it is possible to extract any top level ObjectTypes with the `public List getContent(Class c)` method. -!!! Note "contained references" +!!! note "contained references" There is some provisional support for "contained references" when cloning an object - the API for this is subject to change, but an example is used in [copyTest](https://github.com/ivoa/vo-dml/blob/519f213830c0a946a0d63be65552067bfc707f1d/tools/gradletooling/sample/src/test/java/org/ivoa/dm/lifecycle/LifeCycleDetailedTest.java#L99C10-L99C18) using the `createContext()` and `updateClonedReferences()` methods either side of an object clone with a copy constructor. diff --git a/doc/guide/QuickStart.md b/doc/guide/QuickStart.md index eb83319a..d169365b 100644 --- a/doc/guide/QuickStart.md +++ b/doc/guide/QuickStart.md @@ -10,14 +10,12 @@ If starting a completely new data model then the [Template DM Project](https://g If adapting an existing data model repository then 1. Edit a `build.gradle.kts` file with reference to the plugin (note substitute ![latest published version](https://img.shields.io/gradle-plugin-portal/v/net.ivoa.vo-dml.vodmltools?label=latest%20published%20version) below) - ```kotlin plugins { id("net.ivoa.vo-dml.vodmltools") version "0.x.x" } ``` 2. create a `settings.gradle.kts` - it is possible just to copy the [template version](https://github.com/ivoa/DataModelTemplate/blob/master/settings.gradle.kts) and just edit the `rootProject.name`. - 3. create the binding file for the model (see [below](#binding-files)) for more detail There is nothing else that needs to be done if the VO-DML files in the default place @@ -32,10 +30,8 @@ gradle vodmlValidate ``` will attempt to validate the model and print any errors. -* vodmlValidate - runs validation on the models. -* -* vodmlJavaGenerate - generate java classes. See [generated code guide](JavaCodeGeneration.md) for details of how to use the generated java code to serialize instances to XML and RDB. - Note that it is also possible to test models using the generated code. +If the validation is successful you can produce [various derived products](Transformers.md). Developing your VO-DML model further is discussed [here](modelling/modellingIntro.md). + ## Detailed configuration @@ -47,6 +43,8 @@ vodml { ``` section in the `build.gradle.kts` file. +The various sub-properties that can be set are + * _vodmlDir_ - the default is `src/main/vo-dml` ```kotlin vodmlDir.set(file("vo-dml")) @@ -59,6 +57,7 @@ section in the `build.gradle.kts` file. * _outputSiteDir_ - where the [mkdocs](https://www.mkdocs.org) suitable model description is created by the `gradle vodmlSite` command - default `build/generated/docs/vodml-site`. * _outputJavaDir_ - where the generated Java is created - the default is `build/generated/sources/vodml/java/` and it should not be necessary to ever alter this as gradle will integrate this automatically into the various source paths. +* _outputSchemaDir_ - where the XML and JSON schema are generated to - the default is `build/generated/sources/vodml/schema/` - this is automatically included in the classpath and the output jar. * _catalogFile_ - in general it is not necessary to set this, as the plugin will create a catalogue file automatically from the vodmlDir and vodmlFiles properties (as well as including files in any dependencies that also contain VO-DML models) A catalogue file is necessary as the rest of the tooling is designed to use only the filename (no path) for inclusions and references. If it is desired to create a file manually for a special purpose, then the file should have the format as below - it should be noted that all references to model files will have to be specified if this is done. diff --git a/doc/guide/Serialization.md b/doc/guide/Serialization.md index 73690833..6063e416 100644 --- a/doc/guide/Serialization.md +++ b/doc/guide/Serialization.md @@ -30,7 +30,7 @@ For the [small example model](https://github.com/ivoa/vo-dml/tree/master/models/ ```xml - + a value @@ -45,6 +45,8 @@ For the [small example model](https://github.com/ivoa/vo-dml/tree/master/models/ ``` +Note that because the integer parts of the id might have come from database indices, which +typically are only unique per table, then the string part of the id is made up from the part of the vodml-id of the type. ## JSON diff --git a/doc/guide/Transformers.md b/doc/guide/Transformers.md index 6bcee5ae..c362eb57 100644 --- a/doc/guide/Transformers.md +++ b/doc/guide/Transformers.md @@ -26,6 +26,13 @@ The models are also transformed into schema that describe the various serializat VO-DML tooling is to be able to exchange instances of the models between different computer languages, with all the source code and schema automatically generated. -The gradle plugin does not currently have a task directly to generate XML and RDB schema from the models, however, this can be done -indirectly from the generated Java code as can be seen from the [Small java example](https://github.com/ivoa/vo-dml/tree/master/gradletooling/sample/src/main/java/WriteSampleSchema.java). +The gradle task +```shell +gradle vodmlSchema +``` +will generate XML and JSON schema for the model. These schema files will automatically be included withing the jar file for the model. + +# Transformation to VO-DML + +The transformation of other data model representations to VO-DML is discussed elsewhere. diff --git a/doc/guide/modelling/TransformingToVODML.md b/doc/guide/modelling/TransformingToVODML.md new file mode 100644 index 00000000..cb940111 --- /dev/null +++ b/doc/guide/modelling/TransformingToVODML.md @@ -0,0 +1,8 @@ +Creating VO-DML from other Data Models +====================================== + +It is possible to start with an XML schema that can be transferred to VODSL with + +```shell +gradle vodmlXsdToVodsl --xsd mymodel.xsd --dsl mymodel.vodsl +``` \ No newline at end of file diff --git a/doc/guide/modelling/UML.md b/doc/guide/modelling/UML.md index 921bf1cc..f30e21c4 100644 --- a/doc/guide/modelling/UML.md +++ b/doc/guide/modelling/UML.md @@ -3,8 +3,8 @@ UML ## Configuring the UML tool -In general it is necessary to configure a UML tool with the IVOA "profile" to restrict the metamodel to that -used by VO-DML. +In general it is necessary to configure a UML tool with the IVOA "profile" to restrict +the metamodel to that used by VO-DML. TBC diff --git a/doc/guide/modelling/designIntro.md b/doc/guide/modelling/designIntro.md index 11c13ef7..2a03d8f7 100644 --- a/doc/guide/modelling/designIntro.md +++ b/doc/guide/modelling/designIntro.md @@ -1,4 +1,11 @@ Designing Models ================ -## Testing serialization. \ No newline at end of file +Creating "good" data model designs is a art rather than a science. Within the context of the IVOA reusing existing recommended models is a prerequisite for a model be considered good. + +TBC + +## Testing serialization. + +The java runtime has some functionality for [roundtrip testing](../JavaCodeGeneration.md#testing-models) the various serializations +which can be a good first level test as to whether your model is a "good design". \ No newline at end of file diff --git a/doc/guide/modelling/modellingIntro.md b/doc/guide/modelling/modellingIntro.md index 4b7a9c29..ccd58497 100644 --- a/doc/guide/modelling/modellingIntro.md +++ b/doc/guide/modelling/modellingIntro.md @@ -13,3 +13,29 @@ by editing in a text editor (and the [model schema](https://github.com/ivoa/vo-d * [VODSL](VODSL.md) * [UML](UML.md) + +It is possible to import existing data models by; + +1. Adding the published jar as a dependency in the `build.gradle.kts` file, e.g. +```kotlin +dependencies { + implementation("org.javastro.ivoa.vo-dml:ivoa-base:1.0") +} +``` +the tooling then makes sure that the model files are available for local import. +1. Importing into the model file, and there are two cases whether VO-DML or VODSL is considered the 'working source' + + 1. VO-DML - the tooling only requires the file name (it automatically creates an XML catalog which can resolve to the absolute path) +```xml + + ivoa + IVOA-v1.0.vo-dml.xml + + +``` + + 1. VODSL - a relative path import should be used - the tooling automatically creates a vodsl file to import into the `build/tmp` directory so that the + exact form of the relative path depends on where the including source file is located. +``` +include "../../build/tmp/IVOA-v1.0.vodsl" +``` \ No newline at end of file diff --git a/doc/mkdocs.yml b/doc/mkdocs.yml index 682c03a9..b07098ea 100644 --- a/doc/mkdocs.yml +++ b/doc/mkdocs.yml @@ -13,7 +13,7 @@ nav: - Getting Started: - Installation: Installation.md - Configuration: QuickStart.md - - Transformers: + - Derived Products: - Introduction: Transformers.md - Documentation: Documentation.md - Java Generation: JavaCodeGeneration.md @@ -24,10 +24,13 @@ nav: - VODSL : modelling/VODSL.md - Eclipse Editor: modelling/EclipseVODSL.md - UML : modelling/UML.md + - Generation from other sources : modelling/TransformingToVODML.md - Model Design: - Introduction : modelling/designIntro.md markdown_extensions: + - admonition + - pymdownx.details - pymdownx.superfences: custom_fences: - name: mermaid diff --git a/doc/std/ivoatex b/doc/std/ivoatex index 242faedd..4cb591cb 160000 --- a/doc/std/ivoatex +++ b/doc/std/ivoatex @@ -1 +1 @@ -Subproject commit 242faeddf442f03163a53c7bee8062d464ad8ed5 +Subproject commit 4cb591cb784bd06378b537cff05a02e658aad143 diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar index 033e24c4..e6441136 100644 Binary files a/gradle/wrapper/gradle-wrapper.jar and b/gradle/wrapper/gradle-wrapper.jar differ diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties index 9f4197d5..b82aa23a 100644 --- a/gradle/wrapper/gradle-wrapper.properties +++ b/gradle/wrapper/gradle-wrapper.properties @@ -1,6 +1,6 @@ distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists -distributionUrl=https\://services.gradle.org/distributions/gradle-8.2.1-bin.zip +distributionUrl=https\://services.gradle.org/distributions/gradle-8.7-bin.zip networkTimeout=10000 validateDistributionUrl=true zipStoreBase=GRADLE_USER_HOME diff --git a/gradlew b/gradlew index fcb6fca1..1aa94a42 100755 --- a/gradlew +++ b/gradlew @@ -83,7 +83,8 @@ done # This is normally unused # shellcheck disable=SC2034 APP_BASE_NAME=${0##*/} -APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit +# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036) +APP_HOME=$( cd "${APP_HOME:-./}" > /dev/null && pwd -P ) || exit # Use the maximum available, or set MAX_FD != -1 to use that value. MAX_FD=maximum @@ -144,7 +145,7 @@ if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then case $MAX_FD in #( max*) # In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked. - # shellcheck disable=SC3045 + # shellcheck disable=SC2039,SC3045 MAX_FD=$( ulimit -H -n ) || warn "Could not query maximum file descriptor limit" esac @@ -152,7 +153,7 @@ if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then '' | soft) :;; #( *) # In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked. - # shellcheck disable=SC3045 + # shellcheck disable=SC2039,SC3045 ulimit -n "$MAX_FD" || warn "Could not set maximum file descriptor limit to $MAX_FD" esac @@ -201,11 +202,11 @@ fi # Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"' -# Collect all arguments for the java command; -# * $DEFAULT_JVM_OPTS, $JAVA_OPTS, and $GRADLE_OPTS can contain fragments of -# shell script including quotes and variable substitutions, so put them in -# double quotes to make sure that they get re-expanded; and -# * put everything else in single quotes, so that it's not re-expanded. +# Collect all arguments for the java command: +# * DEFAULT_JVM_OPTS, JAVA_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments, +# and any embedded shellness will be escaped. +# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be +# treated as '${Hostname}' itself on the command line. set -- \ "-Dorg.gradle.appname=$APP_BASE_NAME" \ diff --git a/gradlew.bat b/gradlew.bat index 93e3f59f..25da30db 100644 --- a/gradlew.bat +++ b/gradlew.bat @@ -43,11 +43,11 @@ set JAVA_EXE=java.exe %JAVA_EXE% -version >NUL 2>&1 if %ERRORLEVEL% equ 0 goto execute -echo. -echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. -echo. -echo Please set the JAVA_HOME variable in your environment to match the -echo location of your Java installation. +echo. 1>&2 +echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2 +echo. 1>&2 +echo Please set the JAVA_HOME variable in your environment to match the 1>&2 +echo location of your Java installation. 1>&2 goto fail @@ -57,11 +57,11 @@ set JAVA_EXE=%JAVA_HOME%/bin/java.exe if exist "%JAVA_EXE%" goto execute -echo. -echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% -echo. -echo Please set the JAVA_HOME variable in your environment to match the -echo location of your Java installation. +echo. 1>&2 +echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2 +echo. 1>&2 +echo Please set the JAVA_HOME variable in your environment to match the 1>&2 +echo location of your Java installation. 1>&2 goto fail diff --git a/models/ivoa/build.gradle.kts b/models/ivoa/build.gradle.kts index c1f178b8..80fab779 100644 --- a/models/ivoa/build.gradle.kts +++ b/models/ivoa/build.gradle.kts @@ -1,5 +1,5 @@ plugins { - id("net.ivoa.vo-dml.vodmltools") version "0.4.4" + id("net.ivoa.vo-dml.vodmltools") version "0.5.1" // id ("com.diffplug.spotless") version "5.17.1" `maven-publish` id("io.github.gradle-nexus.publish-plugin") version "1.3.0" diff --git a/models/ivoa/vo-dml/ivoa_base.vodml-binding.xml b/models/ivoa/vo-dml/ivoa_base.vodml-binding.xml index 8c25fc05..bf4f8d72 100644 --- a/models/ivoa/vo-dml/ivoa_base.vodml-binding.xml +++ b/models/ivoa/vo-dml/ivoa_base.vodml-binding.xml @@ -1,71 +1,77 @@ - - ivoa -IVOA-v1.0.vo-dml.xml + IVOA-v1.0.vo-dml.xml org.ivoa.dm.ivoa org.ivoa.dm http://ivoa.net/vodml/ivoa http://ivoa.net/dm/models/vo-dml/xsd/ivoa - + real Double float xsd:float + number string String str xsd:string + string boolean Boolean boolean xsd:boolean + boolean integer Integer int xsd:int + number nonnegativeInteger Integer int xsd:nonNegativeInteger + number anyURI String xsd:anyURI + string datetime java.util.Date datetime.datetime xsd:dateTime + string decimal java.math.BigDecimal xsd:decimal + number rational org.ivoa.vodml.stdtypes.Rational xsd:string + number complex org.ivoa.vodml.stdtypes.Complex - xsd:float + xsd:float + diff --git a/runtime/java/build.gradle.kts b/runtime/java/build.gradle.kts index 935eefe5..904363aa 100644 --- a/runtime/java/build.gradle.kts +++ b/runtime/java/build.gradle.kts @@ -6,12 +6,12 @@ plugins { signing } group = "org.javastro.ivoa.vo-dml" -version = "0.6.0" +version = "0.7.0" dependencies { // xjcPlugins("net.codesup.util:jaxb2-rich-contract-plugin:2.1.0") - // implementation("jakarta.persistence:jakarta.persistence-api:3.0.0") // more modern, but perhaps not quite ready + implementation("org.xmlresolver:xmlresolver:6.0.4") // for xml catalogues - note that the apache xml-commons resolver is out of date implementation("jakarta.xml.bind:jakarta.xml.bind-api:4.0.0") // implementation("org.glassfish.jaxb:jaxb-runtime:2.3.6") implementation("jakarta.persistence:jakarta.persistence-api:3.1.0") @@ -19,7 +19,7 @@ dependencies { implementation("org.hibernate.orm:hibernate-core:6.2.7.Final") implementation("org.slf4j:slf4j-api:1.7.36") - api("org.javastro:jaxbjpa-utils:0.2.0") + api("org.javastro:jaxbjpa-utils:0.2.1") compileOnly("org.junit.jupiter:junit-jupiter-api:5.9.2")// have put the base test classes in the runtime main - naughty, but easier to make everything work without changing dependencies diff --git a/runtime/java/src/main/java/org/ivoa/vodml/ModelManagement.java b/runtime/java/src/main/java/org/ivoa/vodml/ModelManagement.java index e5e9824e..193faef5 100644 --- a/runtime/java/src/main/java/org/ivoa/vodml/ModelManagement.java +++ b/runtime/java/src/main/java/org/ivoa/vodml/ModelManagement.java @@ -9,7 +9,6 @@ package org.ivoa.vodml; -import java.io.IOException; import java.util.List; import jakarta.xml.bind.JAXBContext; @@ -50,6 +49,9 @@ public interface ModelManagement { String pu_name(); /** * Write XML schema for the model. + * Note that these schema are internally generated by JAXB, and as such should not be regarded as the definitive schema to be published - + * they are used mainly in automated testing. + * @deprecated - should only use the schema generated by the top level. */ void writeXMLSchema(); @@ -67,7 +69,7 @@ public interface ModelManagement { boolean hasReferences(); /** - * Return a Jackson objectMapper suitable for JSON serialzation. + * Return a Jackson objectMapper suitable for JSON serialization. * @return the objectmapper. */ ObjectMapper jsonMapper(); diff --git a/runtime/java/src/main/java/org/ivoa/vodml/VodmlModel.java b/runtime/java/src/main/java/org/ivoa/vodml/VodmlModel.java index cd08a86d..9237819f 100644 --- a/runtime/java/src/main/java/org/ivoa/vodml/VodmlModel.java +++ b/runtime/java/src/main/java/org/ivoa/vodml/VodmlModel.java @@ -13,6 +13,7 @@ * Marks as a vodml model. Also provides some useful management interfaces. * @author Paul Harrison (paul.harrison@manchester.ac.uk) * @since 3 May 2023 + * IMPL it might be nicer just to inherit all of the interfaces. */ public interface VodmlModel extends org.ivoa.vodml.jaxb.JaxbManagement { ModelManagement management(); diff --git a/runtime/java/src/main/java/org/ivoa/vodml/jaxb/XmlIdManagement.java b/runtime/java/src/main/java/org/ivoa/vodml/jaxb/XmlIdManagement.java index 7ec32db2..5b1583bc 100644 --- a/runtime/java/src/main/java/org/ivoa/vodml/jaxb/XmlIdManagement.java +++ b/runtime/java/src/main/java/org/ivoa/vodml/jaxb/XmlIdManagement.java @@ -38,11 +38,13 @@ public interface XmlIdManagement { /** * Create and XMLID. * @param i the key value; + * @param clazz the class for which the ID is to be generated. * @return the XMLID as NCName. + * The idea is to create a globally unique id */ - static String createXMLId(long i) + static String createXMLId(long i, Class clazz) { - return new StringBuffer("id_").append(i).toString(); // XML ids must be NCNames. + return clazz.getDeclaredAnnotation(org.ivoa.vodml.annotation.VoDml.class).id().replace(":","-") + "_" + i; // XML ids must be NCNames. } /** diff --git a/runtime/java/src/main/java/org/ivoa/vodml/json/JsonManagement.java b/runtime/java/src/main/java/org/ivoa/vodml/json/JsonManagement.java index 5660dfcf..738941b7 100644 --- a/runtime/java/src/main/java/org/ivoa/vodml/json/JsonManagement.java +++ b/runtime/java/src/main/java/org/ivoa/vodml/json/JsonManagement.java @@ -14,12 +14,14 @@ import com.fasterxml.jackson.annotation.PropertyAccessor; import com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility; +import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.ObjectMapper.DefaultTyping; import com.fasterxml.jackson.databind.SerializationFeature; import com.fasterxml.jackson.databind.json.JsonMapper; import com.fasterxml.jackson.databind.jsontype.BasicPolymorphicTypeValidator; import com.fasterxml.jackson.databind.jsontype.PolymorphicTypeValidator; +import com.fasterxml.jackson.databind.util.StdDateFormat; import org.ivoa.vodml.ModelDescription; @@ -46,6 +48,7 @@ static public ObjectMapper jsonMapper(ModelDescription md) { return JsonMapper.builder() .visibility(PropertyAccessor.FIELD, Visibility.ANY) .visibility(PropertyAccessor.GETTER, Visibility.NONE) + .defaultTimeZone(utc) .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false) .configure(SerializationFeature.WRITE_DATES_WITH_ZONE_ID, true) @@ -53,7 +56,8 @@ static public ObjectMapper jsonMapper(ModelDescription md) { .configure(SerializationFeature.WRAP_ROOT_VALUE, false) .configure(SerializationFeature.WRITE_SINGLE_ELEM_ARRAYS_UNWRAPPED, false) .handlerInstantiator(new VodmlHandlerInstantiator(md)) - .build(); + .build().setSerializationInclusion(Include.NON_NULL) + .setDateFormat(new StdDateFormat().withColonInTimeZone(true)); } diff --git a/runtime/java/src/main/java/org/ivoa/vodml/nav/Util.java b/runtime/java/src/main/java/org/ivoa/vodml/nav/Util.java index cf52869d..e2e8ce84 100644 --- a/runtime/java/src/main/java/org/ivoa/vodml/nav/Util.java +++ b/runtime/java/src/main/java/org/ivoa/vodml/nav/Util.java @@ -32,10 +32,7 @@ */ public class Util { - /** ID_0. - */ - private static final String ID_0 = XmlIdManagement.createXMLId(0); - + static final AtomicLong NEXT_ID = new AtomicLong(1000); /** logger for this class */ @@ -114,13 +111,13 @@ public static void makeUniqueIDs(List els ) { Set currentValues = els.stream().map(p->p.getXmlId()).collect(Collectors.toSet()); for (XmlIdManagement el : els) { final String id = el.getXmlId(); - if(id == null || id.isEmpty()|| id.trim().equals(ID_0)) + if(id == null || id.isEmpty()|| id.trim().equals(XmlIdManagement.createXMLId(0,el.getClass()))) { String newval; do { final long l = NEXT_ID.getAndIncrement(); - newval = XmlIdManagement.createXMLId(l); + newval = XmlIdManagement.createXMLId(l,el.getClass()); } while(currentValues.contains(newval)); @@ -140,7 +137,7 @@ public static > List cloneList(List ModelManagement getModelManagement(); - public > T modelRoundTripXMLwithTest(T model) throws PropertyException, TransformerConfigurationException, ParserConfigurationException, JAXBException, TransformerFactoryConfigurationError, TransformerException + public > T modelRoundTripXMLwithTest( VodmlModel model) throws PropertyException, TransformerConfigurationException, ParserConfigurationException, JAXBException, TransformerFactoryConfigurationError, TransformerException { - RoundTripResult result = roundtripXML(model.management()); + RoundTripResult result = roundtripXML(model); assertTrue(result.isValid, "reading xml back had errors"); assertNotNull(result.retval,"returned object from XML serialization null"); return result.retval; @@ -45,7 +45,7 @@ public > T modelRoundTripXMLwithTest(T model) throws Pro public > T modelRoundTripJSONwithTest(T model) throws JsonProcessingException { - RoundTripResult result = roundTripJSON(model.management()); + RoundTripResult result = roundTripJSON(model); assertTrue(result.isValid, "reading xml back had errors"); assertNotNull(result.retval,"returned object from JSON serialization null"); return result.retval; diff --git a/runtime/java/src/main/java/org/ivoa/vodml/testing/AutoRoundTripTest.java b/runtime/java/src/main/java/org/ivoa/vodml/testing/AutoRoundTripTest.java index d1cb8055..ed81b537 100644 --- a/runtime/java/src/main/java/org/ivoa/vodml/testing/AutoRoundTripTest.java +++ b/runtime/java/src/main/java/org/ivoa/vodml/testing/AutoRoundTripTest.java @@ -52,7 +52,7 @@ public abstract class AutoRoundTripTest > extends Abstra void testXmlRoundTrip() throws JAXBException, TransformerConfigurationException, ParserConfigurationException, TransformerFactoryConfigurationError, TransformerException, IOException { M model = createModel(); - RoundTripResult result = roundtripXML(model.management()); + RoundTripResult result = roundtripXML(model); assertTrue(result.isValid, "reading XML back had errors"); assertNotNull(result.retval,"returned object from XML serialization null"); testModel(result.retval); @@ -62,7 +62,7 @@ void testXmlRoundTrip() throws JAXBException, TransformerConfigurationException, void testJSONRoundTrip() throws JsonProcessingException { M model = createModel(); - RoundTripResult result = roundTripJSON(model.management()); + RoundTripResult result = roundTripJSON(model); assertTrue(result.isValid, "reading JSON back had errors"); assertNotNull(result.retval,"returned object from JSON serialization null"); testModel(result.retval); diff --git a/runtime/java/src/main/java/org/ivoa/vodml/testing/AutoRoundTripWithValidationTest.java b/runtime/java/src/main/java/org/ivoa/vodml/testing/AutoRoundTripWithValidationTest.java index 5e96ec0c..0a387041 100644 --- a/runtime/java/src/main/java/org/ivoa/vodml/testing/AutoRoundTripWithValidationTest.java +++ b/runtime/java/src/main/java/org/ivoa/vodml/testing/AutoRoundTripWithValidationTest.java @@ -26,7 +26,7 @@ public abstract class AutoRoundTripWithValidationTest> e @Test void validationTest() throws JAXBException { final M model = createModel(); - model.management().writeXMLSchema(); + //model.management().writeXMLSchema(); ValidationResult vr = validateModel(model); if(!vr.isOk) diff --git a/runtime/java/src/main/java/org/ivoa/vodml/validation/AbstractBaseValidation.java b/runtime/java/src/main/java/org/ivoa/vodml/validation/AbstractBaseValidation.java index 38e8fa56..e3a327e0 100644 --- a/runtime/java/src/main/java/org/ivoa/vodml/validation/AbstractBaseValidation.java +++ b/runtime/java/src/main/java/org/ivoa/vodml/validation/AbstractBaseValidation.java @@ -40,11 +40,11 @@ import org.ivoa.vodml.validation.ModelValidator.ValidationResult; public abstract class AbstractBaseValidation { - protected RoundTripResult roundTripJSON(ModelManagement m) throws JsonProcessingException { - T model = m.theModel(); + protected RoundTripResult roundTripJSON(VodmlModel m) throws JsonProcessingException { + T model = m.management().theModel(); @SuppressWarnings("unchecked") Class clazz = (Class) model.getClass(); - ObjectMapper mapper = m.jsonMapper(); + ObjectMapper mapper = m.management().jsonMapper(); String json = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(model); System.out.println("JSON output"); System.out.println(json); @@ -62,13 +62,17 @@ public static class RoundTripResult { } } - protected RoundTripResult roundtripXML(ModelManagement modelManagement) throws ParserConfigurationException, JAXBException, + protected > RoundTripResult roundtripXML(VodmlModel vodmlModel) throws ParserConfigurationException, JAXBException, PropertyException, TransformerFactoryConfigurationError, TransformerConfigurationException, TransformerException { - T model = modelManagement.theModel(); + T model = vodmlModel.management().theModel(); + if(vodmlModel.management().hasReferences()) + { + vodmlModel.processReferences(); + } @SuppressWarnings("unchecked") Class clazz = (Class) model.getClass(); - JAXBContext jc = modelManagement.contextFactory(); + JAXBContext jc = vodmlModel.management().contextFactory(); StringWriter sw = new StringWriter(); Marshaller m = jc.createMarshaller(); @@ -173,7 +177,7 @@ protected ValidationResult validateModel(VodmlModel m) throws JAXBExcepti final ModelDescription desc = m.descriptor(); File schemaFile = new File(desc.schemaMap().get(desc.xmlNamespace())); - ModelValidator v = new ModelValidator(schemaFile, m.management().contextFactory()); + ModelValidator v = new ModelValidator(m); return v.validate(m); } diff --git a/runtime/java/src/main/java/org/ivoa/vodml/validation/LSResourceAdapter.java b/runtime/java/src/main/java/org/ivoa/vodml/validation/LSResourceAdapter.java new file mode 100644 index 00000000..dbd0657f --- /dev/null +++ b/runtime/java/src/main/java/org/ivoa/vodml/validation/LSResourceAdapter.java @@ -0,0 +1,68 @@ +package org.ivoa.vodml.validation; + +import org.w3c.dom.ls.LSInput; +import org.w3c.dom.ls.LSResourceResolver; +import org.xmlresolver.*; +import org.xmlresolver.logging.AbstractLogger; +import org.xmlresolver.logging.ResolverLogger; +import org.xmlresolver.sources.ResolverLSInput; + +/** + * This class implements the {@link LSResourceResolver} API. + *

It's a separate class in order to avoid a compile-time dependency on the DOM + * API for users of {@link XMLResolver} who don't use it.

+ * + * This is modified from the similarly named class in XMLResolver because it does not do namspace lookup + */ + +public class LSResourceAdapter implements LSResourceResolver { + private final XMLResolver resolver; + private final ResolverLogger logger; + + public LSResourceAdapter(XMLResolver resolver) { + if (resolver == null) { + throw new NullPointerException(); + } + this.resolver = resolver; + this.logger = resolver.getConfiguration().getFeature(ResolverFeature.RESOLVER_LOGGER); + } + + @Override + public LSInput resolveResource(String type, String namespaceURI, String publicId, String systemId, String baseURI) { + if (systemId == null) { + return null; + } + + final ResourceRequest request; + if (type == null || "http://www.w3.org/TR/REC-xml".equals(type)) { + logger.log(AbstractLogger.REQUEST, "resolveResource: XML: %s (baseURI: %s, publicId: %s)", + systemId, baseURI, publicId); + // This isn't DTD_NATURE because there's no name in this API + request = resolver.getRequest(systemId, baseURI, ResolverConstants.EXTERNAL_ENTITY_NATURE, ResolverConstants.VALIDATION_PURPOSE); + request.setPublicId(publicId); + } else { + logger.log(AbstractLogger.REQUEST, "resolveResource: %s, %s (namespace: %s, baseURI: %s, publicId: %s)", + type, systemId, namespaceURI, baseURI, publicId); + + String purpose = null; + // If it looks like it's going to be used for validation, ... + if (ResolverConstants.NATURE_XML_SCHEMA.equals(type) + || ResolverConstants.NATURE_XML_SCHEMA_1_1.equals(type) + || ResolverConstants.NATURE_RELAX_NG.equals(type)) { + purpose = ResolverConstants.PURPOSE_SCHEMA_VALIDATION; + } + + request = resolver.getRequest(namespaceURI); + } + + ResourceResponse resp = resolver.resolve(request); + + LSInput input = null; + if (resp != null && resp.isResolved()) { + input = new ResolverLSInput(resp, namespaceURI); + } + + return input; + + } +} diff --git a/runtime/java/src/main/java/org/ivoa/vodml/validation/ModelValidator.java b/runtime/java/src/main/java/org/ivoa/vodml/validation/ModelValidator.java index 50d8b94e..28e899ae 100644 --- a/runtime/java/src/main/java/org/ivoa/vodml/validation/ModelValidator.java +++ b/runtime/java/src/main/java/org/ivoa/vodml/validation/ModelValidator.java @@ -12,24 +12,37 @@ import java.io.File; import java.io.IOException; import java.io.PrintStream; +import java.io.StringReader; +import java.io.StringWriter; +import java.io.FileInputStream; +import java.net.URI; +import java.net.URL; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.stream.Collectors; import javax.xml.XMLConstants; import jakarta.xml.bind.JAXBContext; import jakarta.xml.bind.JAXBException; import jakarta.xml.bind.util.JAXBSource; + import javax.xml.transform.Source; import javax.xml.transform.stream.StreamSource; import javax.xml.validation.Schema; import javax.xml.validation.SchemaFactory; import javax.xml.validation.Validator; +import org.ivoa.vodml.VodmlModel; import org.xml.sax.ErrorHandler; +import org.xml.sax.InputSource; import org.xml.sax.SAXException; import org.xml.sax.SAXParseException; +import org.xmlresolver.ResolverFeature; +import org.xmlresolver.XMLResolverConfiguration; +import org.xmlresolver.XMLResolver; +import org.xmlresolver.catalog.entry.EntryCatalog; /** * A Model Validator. @@ -40,21 +53,50 @@ public class ModelValidator { private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory .getLogger(ModelValidator.class); + private final String schemaCat; + private JAXBContext jc; + private final Source[] schemaFiles; - private final JAXBContext jc; /** * Create modelValidator from XML Schema. - * @param schemaFile the XML schema for the model - * @param jc the J + * @param model the model description. */ - public ModelValidator(File schemaFile, JAXBContext jc) { - schemaFiles = new Source[] {new StreamSource(schemaFile)}; - this.jc = jc; + public ModelValidator(VodmlModel model) { + schemaFiles = model.descriptor().schemaMap().entrySet().stream() + .map(s -> new StreamSource(this.getClass().getResourceAsStream("/"+s.getValue()),s.getKey())) + .toArray(Source[]::new); + schemaCat = makeCatalogue( model.descriptor().schemaMap()); + try { + this.jc = model.management().contextFactory(); + } catch (JAXBException e) { + this.jc = null; + logger.error("unable to create a model validator", e); + } } - + + + static private String makeCatalogue(Map schemaMap) { + StringWriter writer = new StringWriter(); + + writer.write("\n"); + schemaMap.forEach((k,v) -> { + writer.append("\n"); + }); + + writer.write(""); + + + return writer.toString(); + } + + public enum ErrorKind { Unknown, Warning, @@ -176,30 +218,95 @@ public void fatalError(SAXParseException exception) } - ValidationResult validate (T p) { + /** + * validate an object from the model. + * @param The type of the object + * @param p The type to be validated. + * @return the result of the validation. + */ +public ValidationResult validate (T p) { + try { + JAXBSource source = new JAXBSource(jc, p); + validateJAXB(source); + } catch (JAXBException e) { + ErrorDescription d = new ErrorDescription(new RuntimeException(e)); + put(errorMap, d.kind, d);; + } + return new ValidationResult(errorMap.isEmpty(), errorMap); + } + + /** + * Validate the file content against the model + * @param file containing xml instance of the model. + * @return the validation + */ +public ValidationResult validate(File file) { + return validateInternal(new StreamSource(file)); + } - try { - errorMap = new HashMap<>(); - SchemaFactory schemaFactory = SchemaFactory - .newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI); - Schema schema = schemaFactory.newSchema(schemaFiles); - Validator validator = schema.newValidator(); - validator.setErrorHandler(new SimpleErrorHandler()); - JAXBSource source = new JAXBSource(jc, p); validator.validate(source);; +public ValidationResult validate(String s) { + return validateInternal(new StreamSource(new StringReader(s))); +} + + ValidationResult validateInternal(Source source) { + try { + + Validator validator = initValidator(); + + validator.validate(source); } catch (SAXException e) { ErrorDescription d = new ErrorDescription(e); put(errorMap, d.kind, d); - } catch (IOException | JAXBException e) { + } catch (IOException e) { ErrorDescription d = new ErrorDescription(new RuntimeException(e)); put(errorMap, d.kind, d);; } return new ValidationResult(errorMap.isEmpty(), errorMap); + + } + + + ValidationResult validateJAXB (JAXBSource source) { + return validateInternal(source); } + + + private Validator initValidator() throws SAXException { + errorMap = new HashMap<>(); + SchemaFactory schemaFactory = SchemaFactory + .newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI); + schemaFactory.setResourceResolver(new LSResourceAdapter(makeXMLResolver())); + Schema schema = schemaFactory.newSchema(schemaFiles); + Validator validator = schema.newValidator(); + validator.setErrorHandler(new SimpleErrorHandler()); + return validator; + } + + XMLResolver makeXMLResolver() { + XMLResolverConfiguration config = new XMLResolverConfiguration(); + config.setFeature(ResolverFeature.DEFAULT_LOGGER_LOG_LEVEL, "info"); + config.setFeature(ResolverFeature.ACCESS_EXTERNAL_DOCUMENT, ""); + config.setFeature(ResolverFeature.THROW_URI_EXCEPTIONS, true); + config.setFeature(ResolverFeature.ALWAYS_RESOLVE, false); + config.setFeature(ResolverFeature.PREFER_PUBLIC, false); + config.setFeature(ResolverFeature.CLASSPATH_CATALOGS, true); + + org.xmlresolver.CatalogManager manager = config + .getFeature(ResolverFeature.CATALOG_MANAGER); + URI caturi = URI.create("https://ivoa.net/vodml/catalog.xml");//IMPL - not sure is this should be more obviously false. + config.addCatalog(caturi.toString()); + EntryCatalog cat = manager.loadCatalog(caturi, new InputSource(new StringReader(schemaCat))); + + XMLResolver resolver = new XMLResolver(config); + return resolver; + } + + private static void put (Map> map, KEY key, VALUE value) { map.computeIfAbsent(key, k -> new ArrayList<>()).add(value); } diff --git a/settings.gradle.kts b/settings.gradle.kts index d02934f6..426f253c 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -1,7 +1,7 @@ /* this is an empty umbrella build to include the various component builds */ - +rootProject.name="vodml" includeBuild("runtime/java") //fixed code needed by the generated code for java includeBuild("models/ivoa") // the ivoa base model -includeBuild("tools/gradletooling") //gradle plugin +includeBuild("tools/gradletooling/sample") //sample project diff --git a/tools/ChangeLog.md b/tools/ChangeLog.md index 2abe74b7..c243bbf8 100644 --- a/tools/ChangeLog.md +++ b/tools/ChangeLog.md @@ -31,4 +31,11 @@ * 0.4.1 Improved list API * 0.4.2 Update the vodsl support to 0.4.6 * 0.4.3 Make work with windows -* 0.4.4 initial contained references support \ No newline at end of file +* 0.4.4 initial contained references support +* 0.4.5 + * New binding customizations - length and isAttribute + * new XML Schema generation + * xml id handling includes vodmlid + * take into account packages. +* 0.5.0 Json Schema Generation (+ new style XSD schema generation) +* 0.5.1 Fix validation regression diff --git a/tools/binding_filter_model.xml b/tools/binding_filter_model.xml index f6cda4fb..fb1e8e2f 100644 --- a/tools/binding_filter_model.xml +++ b/tools/binding_filter_model.xml @@ -8,7 +8,7 @@ Filter.vo-dml.xml org.ivoa.dm.filter org.ivoa.dm.filter - http://ivoa.net/vodml/sample/filter + http://ivoa.net/dm/sample/filter/v1 http://ivoa.net/dm/sample/filter diff --git a/tools/binding_notcoords_model.xml b/tools/binding_notcoords_model.xml index 057a1155..6cfbf6ea 100644 --- a/tools/binding_notcoords_model.xml +++ b/tools/binding_notcoords_model.xml @@ -8,7 +8,7 @@ note that this is just a copy of what like_coords.vo-dml.xml org.ivoa.dm.notstccoords org.ivoa.dm.notstccoords - http://ivoa.net/vodml/sample/notstc/coords/ + http://notstc/coords/v1 http://ivoa.net/dm/sample/notstc/coords/ Epoch diff --git a/tools/binding_sample_model.xml b/tools/binding_sample_model.xml index 08d6a33e..e7a7de10 100644 --- a/tools/binding_sample_model.xml +++ b/tools/binding_sample_model.xml @@ -3,16 +3,20 @@ - sample Sample.vo-dml.xml org.ivoa.dm.sample org.ivoa.dm.samplemodel - http://ivoa.net/vodml/sample/sample + http://ivoa.net/dm/sample/sample/v1 http://ivoa.net/dm/sample/sample + + 32 + + + diff --git a/tools/binding_test_model.xml b/tools/binding_test_model.xml index e5657310..55f36249 100644 --- a/tools/binding_test_model.xml +++ b/tools/binding_test_model.xml @@ -9,7 +9,7 @@ jpatest.vo-dml.xml org.ivoa.dm.jpatest org.ivoa.dm.jpatest - http://ivoa.net/vodml/sample/jpatest + http://ivoa.net/dm/sample/jpatest http://ivoa.net/dm/sample/jpatest diff --git a/tools/gradletooling/TODO.md b/tools/gradletooling/TODO.md index 95f26e58..32c29d77 100644 --- a/tools/gradletooling/TODO.md +++ b/tools/gradletooling/TODO.md @@ -24,15 +24,18 @@ VODML Tooling TODO * idea of a oneOf/Choice * idea of an Any type... * idea of how unique the natural key is - see provenance model (for whole model or just the type) - * could make xmlid depend on type.... + * ~~could make xmlid depend on type....~~ * constraints * add more simple ones - min, max, regexp.. * VODML Metamodel * different serializations of VO-DML itself (json top of list) * allow attributes to be + * perhaps standardize on the binding of the ivoa std primitives so that they do not need to be specified in the files + * specialized "hand written" IVOA base schema + * allow "schema fragments" in binding to cope with specialist overriding of primitives and some base dataTypes... * Anything that might help in mapping to OPENAPI * idea of a "view" - eg. jobsummary in uws - + * STC * epoch - not really defined as something that is used properly * equinox in spaceframe (only used for a few..) @@ -107,7 +110,12 @@ VODML Tooling TODO * JSON * allow refs to be serialized/deserialized as ids always.... - for use in APIs.... https://stackoverflow.com/questions/51172496/how-to-dynamically-ignore-a-property-on-jackson-serialization + * perhaps have custom written ivoa base schema.... express some better rules... e.g. non neg integer... + # Python production -* using dataclasses - need python 3.10 for the kw_only field specifier - might do better just generating multiple `__init__()` rather than relying on the dataclass generation. \ No newline at end of file +* using dataclasses - need python 3.10 for the kw_only field specifier - might do better just generating multiple `__init__()` rather than relying on the dataclass generation. + +# Distribution + * need better directory structure on IVOA site.... \ No newline at end of file diff --git a/tools/gradletooling/gradle-plugin/build.gradle.kts b/tools/gradletooling/gradle-plugin/build.gradle.kts index 817d7b35..84ec01d2 100644 --- a/tools/gradletooling/gradle-plugin/build.gradle.kts +++ b/tools/gradletooling/gradle-plugin/build.gradle.kts @@ -8,13 +8,13 @@ plugins { `java-gradle-plugin` // Apply the Kotlin JVM plugin to add support for Kotlin. - id("org.jetbrains.kotlin.jvm") version "1.8.20" + id("org.jetbrains.kotlin.jvm") version "1.8.22" `maven-publish` id("com.gradle.plugin-publish") version "1.1.0" } group = "net.ivoa.vo-dml" -version = "0.4.4" +version = "0.5.1" repositories { mavenLocal() // FIXME remove this when releasing - just here to pick up local vodsl updates @@ -73,7 +73,7 @@ gradlePlugin { java { toolchain { - languageVersion.set(JavaLanguageVersion.of(11)) + languageVersion.set(JavaLanguageVersion.of(11))//NB needs to stay at 11 for gradle plugin ATM } } diff --git a/tools/gradletooling/gradle-plugin/settings.gradle.kts b/tools/gradletooling/gradle-plugin/settings.gradle.kts new file mode 100644 index 00000000..143213b7 --- /dev/null +++ b/tools/gradletooling/gradle-plugin/settings.gradle.kts @@ -0,0 +1 @@ +rootProject.name="gradle-plugin" \ No newline at end of file diff --git a/tools/gradletooling/gradle-plugin/src/main/java/net/ivoa/vodml/gradle/plugin/VodmlToVodslTask.java b/tools/gradletooling/gradle-plugin/src/main/java/net/ivoa/vodml/gradle/plugin/VodmlToVodslTask.java index 0435b24f..ceb6d4d4 100644 --- a/tools/gradletooling/gradle-plugin/src/main/java/net/ivoa/vodml/gradle/plugin/VodmlToVodslTask.java +++ b/tools/gradletooling/gradle-plugin/src/main/java/net/ivoa/vodml/gradle/plugin/VodmlToVodslTask.java @@ -6,11 +6,19 @@ import java.io.File; +/** + * a task to convert VO-DML to VODSL. + */ public class VodmlToVodslTask extends DefaultTask { private File dml, dsl; + + /** + * Set the input file. + * @param dml the input VODML file + */ @Option(option = "dml", description = "The VO-DML input file") public void setDml(String dml) { this.dml = new File(getProject().getProjectDir(), dml); @@ -18,6 +26,11 @@ public void setDml(String dml) { ); } + + /** + * Set the ooutput file. + * @param dsl the output dsl file name. + */ @Option(option = "dsl", description = "The VODSL output file") public void setDsl(String dsl){ this.dsl = new File(getProject().getProjectDir(), dsl); diff --git a/tools/gradletooling/gradle-plugin/src/main/java/net/ivoa/vodml/gradle/plugin/XsdToVodslTask.java b/tools/gradletooling/gradle-plugin/src/main/java/net/ivoa/vodml/gradle/plugin/XsdToVodslTask.java index 316c2e2a..21c74fff 100644 --- a/tools/gradletooling/gradle-plugin/src/main/java/net/ivoa/vodml/gradle/plugin/XsdToVodslTask.java +++ b/tools/gradletooling/gradle-plugin/src/main/java/net/ivoa/vodml/gradle/plugin/XsdToVodslTask.java @@ -15,7 +15,10 @@ public class XsdToVodslTask extends DefaultTask { private File xsd, dsl; - + /** + * Set the input XSD file. + * @param xsd the input xsd file. + */ @Option(option = "xsd", description = "The XML schema file to be converted (path to)") public void setXsd(String xsd) { this.xsd = new File(xsd); @@ -23,7 +26,11 @@ public void setXsd(String xsd) { ); } - @Option(option = "dsl", description = "The VODSL output filey") + /** + * Set the output VODSL file + * @param dsl the output filename. + */ + @Option(option = "dsl", description = "The VODSL output file") public void setDsl(String dsl){ this.dsl = new File( dsl); } diff --git a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlBaseTask.kt b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlBaseTask.kt index 83410909..3daba871 100644 --- a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlBaseTask.kt +++ b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlBaseTask.kt @@ -47,8 +47,9 @@ class ExternalModelHelper constructor (private val project: Project, private val private val buildDir = project.layout.buildDirectory.asFile.get() init { - logger.info("external models=${externalModelJars.joinToString { f -> f.name }}") + logger.debug("external models=${externalModelJars.joinToString { f -> f.name }}") } + fun makeCatalog(vodmlFiles:ConfigurableFileCollection, catalogFile:RegularFileProperty): File { val tmpdir = project.mkdir(Paths.get(buildDir.absolutePath, "tmp")) @@ -60,6 +61,7 @@ class ExternalModelHelper constructor (private val project: Project, private val } ) ) + return actualCatalog } @@ -70,8 +72,9 @@ class ExternalModelHelper constructor (private val project: Project, private val } toConvert.forEach{ val outvodsl = project.file(Paths.get(tmpdir.absolutePath, it.name.replace(".vo-dml.xml", ".vodsl"))) - logger.info("writing VODSL for external model to ${outvodsl}") + logger.debug("writing VODSL for external model to ${outvodsl}") Vodml2Vodsl.doTransform(it, outvodsl) + } } diff --git a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlExtension.kt b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlExtension.kt index cd9d5718..b00bdf38 100644 --- a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlExtension.kt +++ b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlExtension.kt @@ -1,11 +1,9 @@ package net.ivoa.vodml.gradle.plugin import org.gradle.api.NamedDomainObjectContainer -import org.gradle.api.file.ConfigurableFileCollection import org.gradle.api.file.DirectoryProperty import org.gradle.api.file.ProjectLayout import org.gradle.api.model.ObjectFactory -import org.gradle.api.provider.ListProperty import org.gradle.api.provider.Property import javax.inject.Inject @@ -22,7 +20,7 @@ open class VodmlExtension @Inject constructor(objects: ObjectFactory, layout: Pr override val outputPythonDir = objects.directoryProperty() override val outputDocDir = objects.directoryProperty() override val outputSiteDir = objects.directoryProperty() - override val outputResourcesDir = objects.directoryProperty() + override val outputSchemaDir = objects.directoryProperty() override val defaultPackage = objects.property(String::class.java) override val generateEpisode = objects.property(Boolean::class.java) override val bindingFiles = objects.fileCollection() @@ -45,7 +43,7 @@ open class VodmlExtension @Inject constructor(objects: ObjectFactory, layout: Pr outputPythonDir.set(layout.buildDirectory.dir("generated/sources/vodml/python/")) outputDocDir.set(layout.buildDirectory.dir("generated/docs/vodml/")) outputSiteDir.set(layout.buildDirectory.dir("generated/docs/vodml-site/")) - outputResourcesDir.set(layout.buildDirectory.dir("generated/sources/vodml/resources/")) + outputSchemaDir.set(layout.buildDirectory.dir("generated/sources/vodml/schema/")) defaultPackage.set("vodml.generated") generateEpisode.set(false) vodslDir.set(layout.projectDirectory.dir("src/main/vodsl")) @@ -54,7 +52,7 @@ open class VodmlExtension @Inject constructor(objects: ObjectFactory, layout: Pr vodmlDir.set(this@VodmlExtension.vodmlDir) // note that files not explicitly set outputJavaDir.set(layout.buildDirectory.dir("generated/sources/vodml-$name/java")) outputPythonDir.set(layout.buildDirectory.dir("generated/sources/vodml-$name/python")) - outputResourcesDir.set(layout.buildDirectory.dir("generated/sources/vodml-$name/resources")) + outputSchemaDir.set(layout.buildDirectory.dir("generated/sources/vodml-$name/schema")) outputDocDir.set(layout.buildDirectory.dir("generated/docs/vodml-$name/")) outputSiteDir.set(layout.buildDirectory.dir("generated/docs/vodml-site-$name/")) defaultPackage.set(this@VodmlExtension.defaultPackage) diff --git a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlExtensionGroup.kt b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlExtensionGroup.kt index 5337f693..f78653ec 100644 --- a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlExtensionGroup.kt +++ b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlExtensionGroup.kt @@ -19,7 +19,7 @@ interface VodmlExtensionGroup { val outputPythonDir: DirectoryProperty val outputDocDir: DirectoryProperty val outputSiteDir: DirectoryProperty - val outputResourcesDir: DirectoryProperty + val outputSchemaDir: DirectoryProperty val defaultPackage: Property val generateEpisode: Property val bindingFiles: ConfigurableFileCollection diff --git a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlGradlePlugin.kt b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlGradlePlugin.kt index bb3b74e4..a2e4c797 100644 --- a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlGradlePlugin.kt +++ b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlGradlePlugin.kt @@ -11,6 +11,7 @@ import org.gradle.api.plugins.JavaPluginExtension import org.gradle.api.tasks.SourceSet import org.gradle.api.tasks.SourceSetContainer import org.gradle.api.tasks.TaskProvider +import org.gradle.api.tasks.Copy import org.gradle.api.tasks.util.PatternSet import org.gradle.jvm.tasks.Jar import org.gradle.jvm.toolchain.JavaLanguageVersion @@ -59,10 +60,10 @@ class VodmlGradlePlugin: Plugin { it.modelsToDocument.set(extension.modelsToDocument) } // register the schame task - project.tasks.register(VODML_SCHEMA_TASK_NAME,VodmlXsdTask::class.java) { + val schematask: TaskProvider = project.tasks.register(VODML_SCHEMA_TASK_NAME,VodmlSchemaTask::class.java) { it.description = "create schema for VO-DML models" setVodmlFiles(it,extension,project) - it.schemaDir.set(extension.outputDocDir) + it.schemaDir.set(extension.outputSchemaDir) it.modelsToGenerate.set(extension.modelsToDocument) } // register the validate task @@ -97,12 +98,19 @@ class VodmlGradlePlugin: Plugin { setVodmlFiles(task,extension,project) task.javaGenDir.set(extension.outputJavaDir) - //add the generated source directory to the list of sources to compile IMPL - this feels a bit hacky + //add the generated source directory to the list of sources to compile + // IMPL - recommendation of https://docs.gradle.org/current/dsl/org.gradle.api.tasks.SourceSetOutput.html#org.gradle.api.tasks.SourceSetOutput does not seem to work + // val sourceSets = project.properties["sourceSets"] as SourceSetContainer sourceSets.named(SourceSet.MAIN_SOURCE_SET_NAME) { it.java.srcDir(task.javaGenDir) it.resources.srcDir(task.javaGenDir) + + } + //IMPL this seems hacky as the tests should just pick up the normal resources above, but they do not seem to + sourceSets.named(SourceSet.TEST_SOURCE_SET_NAME){ + it.resources.srcDir(schematask) } // add the vo-dml and binding files to the jar setup val jartask = project.tasks.named(JavaPlugin.JAR_TASK_NAME).get() as Jar @@ -115,19 +123,25 @@ class VodmlGradlePlugin: Plugin { )) } + } + //IMPL this is part of the hack to try to get the generated schema on the classpath - have been copied all over the place!! + // some of the behaviour might be because the default place to generate the schema is into the build directory?? + val processResources = project.tasks.named(JavaPlugin.PROCESS_RESOURCES_TASK_NAME).get() as Copy + processResources.from(schematask) + processResources.dependsOn.add(schematask) + //using java 11 minimum val toolchain = project.extensions.getByType(JavaPluginExtension::class.java).toolchain - toolchain.languageVersion.set(JavaLanguageVersion.of(11)) + toolchain.languageVersion.set(JavaLanguageVersion.of(17)) // force java compile to depend on this task project.tasks.named(JavaPlugin.COMPILE_JAVA_TASK_NAME) { it.dependsOn.add(vodmlJavaTask) + it.dependsOn.add(schematask) } - project.tasks.named(JavaPlugin.PROCESS_RESOURCES_TASK_NAME) - { - it.dependsOn.add(vodmlJavaTask) - } + + //register a task with the old task name as an alias project.tasks.register(VODML_JAVA_TASK_NAME_OLD,DefaultTask::class.java) { @@ -145,7 +159,7 @@ class VodmlGradlePlugin: Plugin { //add the dependencies for JAXB and JPA - using the hibernate implementation - listOf("org.javastro.ivoa.vo-dml:vodml-runtime:0.6.0", + listOf("org.javastro.ivoa.vo-dml:vodml-runtime:0.7.0", "jakarta.xml.bind:jakarta.xml.bind-api:4.0.0", "org.glassfish.jaxb:jaxb-runtime:4.0.2", // "org.eclipse.persistence:org.eclipse.persistence.jpa:2.7.10", // supports JPA 2.2 @@ -172,6 +186,7 @@ class VodmlGradlePlugin: Plugin { JavaPlugin.IMPLEMENTATION_CONFIGURATION_NAME, // don't want exported project.objects.property(Dependency::class.java).convention( project.dependencies.create(it) + ) ) } diff --git a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlSchemaTask.kt b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlSchemaTask.kt new file mode 100644 index 00000000..d6c935c3 --- /dev/null +++ b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlSchemaTask.kt @@ -0,0 +1,74 @@ +package net.ivoa.vodml.gradle.plugin + +import org.gradle.api.file.ArchiveOperations +import org.gradle.api.file.DirectoryProperty +import org.gradle.api.provider.Property +import org.gradle.api.tasks.* +import java.io.File +import javax.inject.Inject +import com.fasterxml.jackson.databind.ObjectMapper + + + + + +/* + * Created on 13/09/2022 by Paul Harrison (paul.harrison@manchester.ac.uk). + */ + +/** + * Task to generate XML schema files from vodsl. + */ +open class VodmlSchemaTask @Inject constructor(ao1: ArchiveOperations) : VodmlBaseTask(ao1) +{ + + @get:OutputDirectory + val schemaDir : DirectoryProperty = project.objects.directoryProperty() + + @Input + @Optional + val modelsToGenerate : Property = project.objects.property(String::class.java) + + @TaskAction + fun doXsdGenerate() { + logger.info("Generating XML schema ") + logger.debug("Looked in ${vodmlDir.get()}") + val eh = ExternalModelHelper(project, ao, logger) + val actualCatalog = eh.makeCatalog(vodmlFiles, catalogFile) + val allBinding = bindingFiles.files.plus(eh.externalBinding()) + vodmlFiles.forEach { + val shortname = it.nameWithoutExtension + val outfile = schemaDir.file("$shortname.xsd") + logger.debug("Generating XML schema from ${it.name} to ${outfile.get().asFile.absolutePath}") + Vodml2xsdNew.doTransform(it.absoluteFile, mapOf( + "binding" to allBinding.joinToString(separator = ",") { it.toURI().toURL().toString() } + ), + actualCatalog, outfile.get().asFile) + } + + logger.info("Generating JSON schema") + vodmlFiles.forEach { + val shortname = it.nameWithoutExtension + val outfile = schemaDir.file("$shortname.json") + logger.debug("Generating JSON schema from ${it.name} to ${outfile.get().asFile.absolutePath}") + val s = Vodml2json.doTransformToString(it.absoluteFile, mapOf( + "binding" to allBinding.joinToString(separator = ",") { it.toURI().toURL().toString() } + ), + actualCatalog) + //prettyprint the generated JSON - i.e. going via jackson + val mapper = ObjectMapper() + val pretty = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(mapper.readTree(s)); + outfile.get().asFile.writeText(pretty) + } + + logger.info("generating Catalogues") + Vodml2Catalogues.doTransform(mapOf( + "binding" to bindingFiles.joinToString(separator = ",") { it.toURI().toURL().toString() }, + "xml-catalogue" to schemaDir.file("xmlcat.xml").get().asFile.absolutePath, + "json-catalogue" to schemaDir.file("jsoncat.txt").get().asFile.absolutePath + + ) + ) + + } +} \ No newline at end of file diff --git a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlXsdTask.kt b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlXsdTask.kt deleted file mode 100644 index 1bc2fa58..00000000 --- a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlXsdTask.kt +++ /dev/null @@ -1,47 +0,0 @@ -package net.ivoa.vodml.gradle.plugin - -import org.gradle.api.DefaultTask -import org.gradle.api.file.ArchiveOperations -import org.gradle.api.file.ConfigurableFileCollection -import org.gradle.api.file.DirectoryProperty -import org.gradle.api.file.RegularFileProperty -import org.gradle.api.provider.Property -import org.gradle.api.tasks.* -import javax.inject.Inject - - -/* - * Created on 13/09/2022 by Paul Harrison (paul.harrison@manchester.ac.uk). - */ - -/** - * Task to generate XML schema files from vodsl. - */ -open class VodmlXsdTask @Inject constructor(ao1: ArchiveOperations) : VodmlBaseTask(ao1) -{ - - @get:OutputDirectory - val schemaDir : DirectoryProperty = project.objects.directoryProperty() - - @Input - @Optional - val modelsToGenerate : Property = project.objects.property(String::class.java) - - @TaskAction - fun doXsdGenerate() { - logger.info("Generating XML schema for VO-DML files ${vodmlFiles.files.joinToString { it.name }}") - logger.info("Looked in ${vodmlDir.get()}") - val eh = ExternalModelHelper(project, ao, logger) - val actualCatalog = eh.makeCatalog(vodmlFiles, catalogFile) - val allBinding = bindingFiles.files.plus(eh.externalBinding()) - vodmlFiles.forEach { - val shortname = it.nameWithoutExtension - val outfile = schemaDir.file("$shortname.xsd") - logger.info("Generating XML schema from ${it.name} to ${outfile.get().asFile.absolutePath}") - Vodml2xsd.doTransform(it.absoluteFile, mapOf( - "binding" to allBinding.joinToString(separator = ",") { it.toURI().toURL().toString() } - ), - actualCatalog, outfile.get().asFile) - } - } -} \ No newline at end of file diff --git a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/XSLTTransform.kt b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/XSLTTransform.kt index 47ac4078..54136436 100644 --- a/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/XSLTTransform.kt +++ b/tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/XSLTTransform.kt @@ -5,12 +5,14 @@ import net.sf.saxon.s9api.Processor import net.sf.saxon.s9api.QName import net.sf.saxon.s9api.Serializer import net.sf.saxon.s9api.XdmAtomicValue +import net.sf.saxon.s9api.XsltExecutable import org.gradle.api.GradleException import org.slf4j.LoggerFactory import org.xmlresolver.XMLResolver import org.xmlresolver.ResolverFeature import org.xmlresolver.XMLResolverConfiguration import java.io.File +import java.io.StringWriter import javax.xml.transform.URIResolver import javax.xml.transform.stream.StreamSource @@ -19,46 +21,62 @@ import javax.xml.transform.stream.StreamSource * Runs XSLT transformations. * Created on 27/07/2021 by Paul Harrison (paul.harrison@manchester.ac.uk). */ - open class XSLTTransformer( val script: String, val method: String) { - - private val logger = LoggerFactory.getLogger(this.javaClass.name) - +abstract class BaseTransformer( val script: String ) { + protected val logger = LoggerFactory.getLogger(this.javaClass.name) + protected val processor : Processor + protected val defaultResolver: URIResolver + protected val stylesheet: XsltExecutable init { - logger.info("initializing transformer for ${script}") + logger.debug("initializing transformer for ${script}") + processor = Processor(false) + defaultResolver = processor.underlyingConfiguration.uriResolver + processor.underlyingConfiguration.setURIResolver { href, base -> + logger.debug("XSLT Transform uri resolver href={} base={}", href, base) + if (base.isBlank()) //IMPL is this the correct heuristic? + StreamSource(this::class.java.getResourceAsStream("/xslt/$href")) + else + defaultResolver.resolve(href, base) + } + val compiler = processor.newXsltCompiler() + val streamSource = StreamSource(this::class.java.getResourceAsStream("/xslt/$script")) + stylesheet = compiler.compile(streamSource) } +} + + open class XSLTTransformer( val xslscript: String, val method: String) : BaseTransformer(xslscript) { + + fun doTransform(vodmlFile: File, output: File) { doTransform(vodmlFile, emptyMap(), null, output) } fun doTransform(vodmlFile: File, params: Map ,output: File) { doTransform(vodmlFile, params, null, output) } - fun doTransform(vodmlFile: File, params: Map, catalog: File?, output: File) { - logger.info("doing $script transform with params") + + fun doTransform(vodmlFile: File, params: Map, catalog: File?, output: File) + { + val out = processor.newSerializer(output) + doTransformInternal(vodmlFile,params,catalog,out) + + } + fun doTransformToString(vodmlFile: File, params: Map, catalog: File?) : String + { + val sw = java.io.StringWriter() + val out = processor.newSerializer(sw) + doTransformInternal(vodmlFile,params,catalog,out) + return sw.toString() + } + + private fun doTransformInternal(vodmlFile: File, params: Map, catalog: File?, out: Serializer) { + logger.debug("doing $script transform with params") params.forEach{ - logger.info("parameter ${it.key}, val=${it.value}") + logger.debug("parameter ${it.key}, val=${it.value}") } if(!vodmlFile.exists()) { throw GradleException("input file "+vodmlFile+ " does not exist") } - - - - //TODO push some of this to static construction? - val processor = Processor(false) - val defaultResolver: URIResolver = processor.underlyingConfiguration.uriResolver - processor.underlyingConfiguration.setURIResolver { href, base -> - logger.debug("XSLT Transform uri resolver href={} base={}", href, base) - if (base.isBlank()) //IMPL is this the correct heuristic? - StreamSource(this::class.java.getResourceAsStream("/xslt/$href")) - else - defaultResolver.resolve(href, base) - } - val compiler = processor.newXsltCompiler() - val streamSource = StreamSource(this::class.java.getResourceAsStream("/xslt/$script")) - val stylesheet = compiler.compile(streamSource) - val out = processor.newSerializer(output) out.setOutputProperty(Serializer.Property.METHOD, method) out.setOutputProperty(Serializer.Property.INDENT, "yes") val trans = stylesheet.load30() @@ -81,17 +99,39 @@ import javax.xml.transform.stream.StreamSource } +/** + * a transformer that has a stylesheet that only uses xsl:result-document ans starts from the named template. + */ +open class XSLTExecutionOnlyTransformer( val xslscript: String, val templateName: String) : BaseTransformer(xslscript) { + + fun doTransform(params: Map) { + val trans = stylesheet.load30() + val sparams = params.entries.associate{ + QName(it.key) to XdmAtomicValue(it.value) + } + trans.setStylesheetParameters(sparams) + val tmp = java.io.File.createTempFile("ignore","silly") + trans.setBaseOutputURI(tmp.toURI().toString()) // IMPL - not the real output - only done to avoid net.sf.saxon.s9api.SaxonApiException: The system identifier of the principal output file is unknown + trans.callTemplate(QName(templateName)) + } + +} + object Vodml2Gml : XSLTTransformer("vo-dml2gml.xsl", "xml") object Vodml2Gvd : XSLTTransformer("vo-dml2gvd.xsl", "text") object Vodml2Html : XSLTTransformer("vo-dml2html.xsl", "html") object Vodml2rdf : XSLTTransformer("vo-dml2rdf.xsl", "text") object Vodml2xsd : XSLTTransformer("vo-dml2xsd.xsl", "xml") + +object Vodml2xsdNew : XSLTTransformer("vo-dml2xsdNew.xsl", "xml") + object Vodml2Java : XSLTTransformer("vo-dml2java.xsl", "text") object Vodml2Latex : XSLTTransformer("vo-dml2Latex.xsl", "text") object Vodml2Vodsl : XSLTTransformer("vo-dml2dsl.xsl", "text") object Vodml2Python : XSLTTransformer("vo-dml2python.xsl", "text") object Xsd2Vodsl : XSLTTransformer("xsd2dsl.xsl", "text") object Vodml2json : XSLTTransformer("vo-dml2jsonschema.xsl", "text") +object Vodml2Catalogues : XSLTExecutionOnlyTransformer("create-catalogues.xsl", "main") object Vodml2md : XSLTTransformer("vo-dml2md.xsl", "text") diff --git a/tools/gradletooling/sample/build.gradle.kts b/tools/gradletooling/sample/build.gradle.kts index 1cf16960..a2c0a42c 100644 --- a/tools/gradletooling/sample/build.gradle.kts +++ b/tools/gradletooling/sample/build.gradle.kts @@ -6,8 +6,8 @@ import ru.vyarus.gradle.plugin.python.task.PythonTask * */ plugins { - id("net.ivoa.vo-dml.vodmltools") version "0.4.4" -// id ("com.diffplug.spotless") version "5.17.1" + id("net.ivoa.vo-dml.vodmltools") version "0.5.1" + id("com.diffplug.spotless") version "6.25.0" id("ru.vyarus.use-python") version "3.0.0" } @@ -43,6 +43,7 @@ vodml { ) outputDocDir.set(layout.projectDirectory.dir("docs")) outputSiteDir.set(outputDocDir.dir("generated")) + outputSchemaDir.set(outputDocDir.dir("schema")) vodslDir.set(vodmlDir) // same place for source models modelsToDocument.set("sample,filter,coords,jpatest,lifecycleTest") outputPythonDir.set(layout.projectDirectory.dir("pythontest/generated")) @@ -79,12 +80,13 @@ tasks.test { dependencies { implementation("org.javastro.ivoa.vo-dml:ivoa-base") testImplementation("org.junit.jupiter:junit-jupiter:5.9.2") + testImplementation("com.networknt:json-schema-validator:1.4.0") testRuntimeOnly("org.junit.platform:junit-platform-launcher") implementation("org.slf4j:slf4j-api:2.0.9") testRuntimeOnly("ch.qos.logback:logback-classic:1.4.7") testImplementation("com.h2database:h2:2.1.214") // try out h2 // testImplementation("org.apache.derby:derby:10.14.2.0") - compileOnly("com.google.googlejavaformat:google-java-format:1.16.0") + compileOnly("com.google.googlejavaformat:google-java-format:1.22.0") } @@ -104,6 +106,44 @@ python { } + +tasks.register("tpath") { + group = "Other" + description = "looking at various paths" + +dependsOn("vodmlJavaGenerate") + doLast{ + + println(sourceSets.main.get().java.sourceDirectories.asPath) + println(sourceSets.main.get().resources.sourceDirectories.asPath) + println(sourceSets.main.get().output.classesDirs.asPath) + println(sourceSets.test.get().resources.sourceDirectories.asPath) + + sourceSets.test.get().compileClasspath.files.forEach{ + println(it.path) + } + SourceSet.TEST_SOURCE_SET_NAME + + } +} + +//TODO +configure { + // optional: limit format enforcement to just the files changed by this feature branch + // ratchetFrom 'origin/main' + + + java { + // don't need to set target, it is inferred from java + target("build/generated/sources/vodml/java/**/*.java") + // apply a specific flavor of google-java-format + googleJavaFormat("1.22.0").reflowLongStrings().skipJavadocFormatting() + // fix formatting of type annotations + formatAnnotations() + + } +} + tasks.register("pytest", PythonTask::class.java) { command = "pythontest/src/SourceCatalogueTest.py" // command = "-c \"import sys; print(sys.path)\"" diff --git a/tools/gradletooling/settings.gradle.kts b/tools/gradletooling/sample/settings.gradle.kts similarity index 59% rename from tools/gradletooling/settings.gradle.kts rename to tools/gradletooling/sample/settings.gradle.kts index 986ede44..76c44f3c 100644 --- a/tools/gradletooling/settings.gradle.kts +++ b/tools/gradletooling/sample/settings.gradle.kts @@ -5,7 +5,7 @@ pluginManagement { gradlePluginPortal() mavenCentral() } - includeBuild("gradle-plugin") //get the gradle plugin + includeBuild("../gradle-plugin") //get the gradle plugin } // == Define locations for components == @@ -16,8 +16,5 @@ dependencyResolutionManagement { } } -includeBuild("../../models/ivoa") -rootProject.name="gradletooling" -include("sample") - -project(":sample").name = "vodml-sample" +includeBuild("../../../models/ivoa") +rootProject.name="vodml-sample" diff --git a/tools/gradletooling/sample/src/main/java/WriteSampleSchema.java b/tools/gradletooling/sample/src/main/java/WriteSampleSchema.java index 93b24a88..b48ffbb9 100644 --- a/tools/gradletooling/sample/src/main/java/WriteSampleSchema.java +++ b/tools/gradletooling/sample/src/main/java/WriteSampleSchema.java @@ -1,4 +1,3 @@ - /* * Created on 26/09/2022 by Paul Harrison (paul.harrison@manchester.ac.uk). */ @@ -6,7 +5,7 @@ import org.ivoa.dm.sample.SampleModel; public class WriteSampleSchema { - public static void main(String[] args) { - SampleModel.writeXMLSchema(); - } + public static void main(String[] args) { + SampleModel.writeXMLSchema(); + } } diff --git a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/jpatest/JPATestModelTest.java b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/jpatest/JPATestModelTest.java index 56e46f57..b35087b1 100644 --- a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/jpatest/JPATestModelTest.java +++ b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/jpatest/JPATestModelTest.java @@ -1,29 +1,23 @@ /* - * Created on 21 Oct 2022 + * Created on 21 Oct 2022 * Copyright 2022 Paul Harrison (paul.harrison@manchester.ac.uk) * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License in file LICENSE - */ + */ package org.ivoa.dm.jpatest; import static org.junit.jupiter.api.Assertions.*; -import java.io.IOException; -import java.util.Arrays; -import java.util.List; - +import com.fasterxml.jackson.core.JsonProcessingException; import jakarta.xml.bind.JAXBContext; import jakarta.xml.bind.JAXBException; +import java.util.List; import javax.xml.parsers.ParserConfigurationException; -import javax.xml.transform.TransformerConfigurationException; import javax.xml.transform.TransformerException; import javax.xml.transform.TransformerFactoryConfigurationError; - -import com.fasterxml.jackson.core.JsonProcessingException; - import org.ivoa.dm.sample.SampleModel; import org.ivoa.vodml.testing.AbstractTest; import org.junit.jupiter.api.AfterEach; @@ -33,132 +27,133 @@ /** * . - * @author Paul Harrison (paul.harrison@manchester.ac.uk) + * @author Paul Harrison (paul.harrison@manchester.ac.uk) * @since 21 Oct 2022 */ class JPATestModelTest extends AbstractTest { - private Parent atest; - - /** - * @throws java.lang.Exception - */ - @BeforeAll - static void setUpBeforeClass() throws Exception { - } - - /** - * @throws java.lang.Exception - */ - @BeforeEach - void setUp() throws Exception { - final ReferredTo1 referredTo = new ReferredTo1("top level ref"); - final ReferredTo2 referredToin = new ReferredTo2("lower ref"); - Child refcont = new Child(referredToin); - List ll = List.of(new LChild("First",1), new LChild("Second", 2), new LChild("Third",3)); - - atest = Parent.createParent(a -> { - ReferredTo3 ref3 = new ReferredTo3("ref in dtype"); - a.dval = new ADtype(1.1, "astring", ref3 ); - a.rval = referredTo; - a.cval = refcont; - a.lval = ll; - }); - - } - - /** - * @throws java.lang.Exception - */ - @AfterEach - void tearDown() throws Exception { - } - - @Test - void jaxbtest() throws JAXBException, ParserConfigurationException, TransformerFactoryConfigurationError, TransformerException{ - - JAXBContext jc = JpatestModel.contextFactory(); - JpatestModel model = new JpatestModel(); - model.addContent(atest); - model.processReferences(); - assertTrue(atest.cval.rval._id != 0, "id setting did not work"); - JpatestModel modelin = modelRoundTripXMLwithTest(model); - System.out.println("generating schema"); - JpatestModel.writeXMLSchema(); - } - @Test - void jpaInitialCreateTest() { - jakarta.persistence.EntityManager em = setupH2Db(SampleModel.pu_name());//the persistence unit is all under the one file.... - em.getTransaction().begin(); - atest.persistRefs(em); //IMPL need to save references explicitly as they are new. - em.persist(atest); - em.getTransaction().commit(); - Long id = atest.getId(); - - //flush any existing entities - em.clear(); - em.getEntityManagerFactory().getCache().evictAll(); - - // now read back - em.getTransaction().begin(); - List par = em.createNamedQuery("Parent.findById", Parent.class) - .setParameter("id", id).getResultList(); - em.getTransaction().commit(); - assertEquals(1, par.size()); - assertEquals("top level ref",par.get(0).rval.sval); - assertEquals("lower ref",par.get(0).cval.rval.sval); - assertEquals("ref in dtype",par.get(0).dval.dref.sval); - - - - } - @Test - void jpaUpdateOrderedTest() throws JsonProcessingException { - jakarta.persistence.EntityManager em = setupH2Db(SampleModel.pu_name());//the persistence unit is all under the one file.... - em.getTransaction().begin(); - atest.persistRefs(em); //IMPL need to save references explicitly as they are new. - em.persist(atest); - em.getTransaction().commit(); - Long id = atest.getId(); - - //flush any existing entities - em.clear(); - em.getEntityManagerFactory().getCache().evictAll(); - - // now read back - Parent par = em.createNamedQuery("Parent.findById", Parent.class) - .setParameter("id", id).getResultList().get(0); - List ll = par.getLval(); - assertEquals(3, ll.size()); - LChild a = ll.get(1); - assertEquals(2, a.getIval()); - - // create an update object - String injson = "{\"sval\":\"Seconded\",\"ival\":2000,\"_id\":2}"; // assumes the id is 2 - LChild arepl = SampleModel.jsonMapper().readValue(injson, LChild.class); - - em.getTransaction().begin(); - - par.replaceInLval(arepl); - em.merge(par); - em.getTransaction().commit(); - //flush any existing entities - em.clear(); - em.getEntityManagerFactory().getCache().evictAll(); - - - Parent par2= em.createNamedQuery("Parent.findById", Parent.class) - .setParameter("id", id).getResultList().get(0); - List ll2 = par2.getLval(); - assertEquals(3, ll2.size()); - LChild ain = ll2.get(1); - assertEquals(2000, a.getIval()); - - - - - } - + private Parent atest; + + /** + * @throws java.lang.Exception + */ + @BeforeAll + static void setUpBeforeClass() throws Exception {} + + /** + * @throws java.lang.Exception + */ + @BeforeEach + void setUp() throws Exception { + final ReferredTo1 referredTo = new ReferredTo1("top level ref"); + final ReferredTo2 referredToin = new ReferredTo2("lower ref"); + Child refcont = new Child(referredToin); + List ll = + List.of(new LChild("First", 1), new LChild("Second", 2), new LChild("Third", 3)); + + atest = + Parent.createParent( + a -> { + ReferredTo3 ref3 = new ReferredTo3("ref in dtype"); + a.dval = new ADtype(1.1, "astring", ref3); + a.rval = referredTo; + a.cval = refcont; + a.lval = ll; + }); + } + + /** + * @throws java.lang.Exception + */ + @AfterEach + void tearDown() throws Exception {} + + @Test + void jaxbtest() + throws JAXBException, + ParserConfigurationException, + TransformerFactoryConfigurationError, + TransformerException { + + JAXBContext jc = JpatestModel.contextFactory(); + JpatestModel model = new JpatestModel(); + model.addContent(atest); + model.processReferences(); + assertTrue(atest.cval.rval._id != 0, "id setting did not work"); + JpatestModel modelin = modelRoundTripXMLwithTest(model); + } + + @Test + void jpaInitialCreateTest() { + jakarta.persistence.EntityManager em = + setupH2Db(SampleModel.pu_name()); // the persistence unit is all under the one file.... + em.getTransaction().begin(); + atest.persistRefs(em); // IMPL need to save references explicitly as they are new. + em.persist(atest); + em.getTransaction().commit(); + Long id = atest.getId(); + + // flush any existing entities + em.clear(); + em.getEntityManagerFactory().getCache().evictAll(); + + // now read back + em.getTransaction().begin(); + List par = + em.createNamedQuery("Parent.findById", Parent.class).setParameter("id", id).getResultList(); + em.getTransaction().commit(); + assertEquals(1, par.size()); + assertEquals("top level ref", par.get(0).rval.sval); + assertEquals("lower ref", par.get(0).cval.rval.sval); + assertEquals("ref in dtype", par.get(0).dval.dref.sval); + } + + @Test + void jpaUpdateOrderedTest() throws JsonProcessingException { + jakarta.persistence.EntityManager em = + setupH2Db(SampleModel.pu_name()); // the persistence unit is all under the one file.... + em.getTransaction().begin(); + atest.persistRefs(em); // IMPL need to save references explicitly as they are new. + em.persist(atest); + em.getTransaction().commit(); + Long id = atest.getId(); + + // flush any existing entities + em.clear(); + em.getEntityManagerFactory().getCache().evictAll(); + + // now read back + Parent par = + em.createNamedQuery("Parent.findById", Parent.class) + .setParameter("id", id) + .getResultList() + .get(0); + List ll = par.getLval(); + assertEquals(3, ll.size()); + LChild a = ll.get(1); + assertEquals(2, a.getIval()); + + // create an update object + String injson = "{\"sval\":\"Seconded\",\"ival\":2000,\"_id\":2}"; // assumes the id is 2 + LChild arepl = SampleModel.jsonMapper().readValue(injson, LChild.class); + + em.getTransaction().begin(); + + par.replaceInLval(arepl); + em.merge(par); + em.getTransaction().commit(); + // flush any existing entities + em.clear(); + em.getEntityManagerFactory().getCache().evictAll(); + + Parent par2 = + em.createNamedQuery("Parent.findById", Parent.class) + .setParameter("id", id) + .getResultList() + .get(0); + List ll2 = par2.getLval(); + assertEquals(3, ll2.size()); + LChild ain = ll2.get(1); + assertEquals(2000, a.getIval()); + } } - - diff --git a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/lifecycle/LifeCycleDetailedTest.java b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/lifecycle/LifeCycleDetailedTest.java index 86e5ae8c..5995be84 100644 --- a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/lifecycle/LifeCycleDetailedTest.java +++ b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/lifecycle/LifeCycleDetailedTest.java @@ -1,26 +1,19 @@ /* - * Created on 27 Jun 2023 + * Created on 27 Jun 2023 * Copyright 2023 Paul Harrison (paul.harrison@manchester.ac.uk) * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License in file LICENSE - */ + */ package org.ivoa.dm.lifecycle; import static org.junit.jupiter.api.Assertions.*; -import java.util.AbstractMap; import java.util.Arrays; -import java.util.HashSet; import java.util.List; -import java.util.Map; -import java.util.Set; - import org.ivoa.dm.sample.SampleModel; -import org.ivoa.dm.sample.catalog.inner.SourceCatalogue; -import org.ivoa.vodml.ModelContext; import org.ivoa.vodml.testing.AbstractTest; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeAll; @@ -29,106 +22,107 @@ /** * . - * @author Paul Harrison (paul.harrison@manchester.ac.uk) + * @author Paul Harrison (paul.harrison@manchester.ac.uk) * @since 27 Jun 2023 */ public class LifeCycleDetailedTest extends AbstractTest { - private ATest atest; - private ATest2 atest2; - private LifecycleTestModel model; - private ATest3 atest3; - - /** - * @throws java.lang.Exception - */ - @BeforeAll - static void setUpBeforeClass() throws Exception { - } - - /** - * @throws java.lang.Exception - */ - @BeforeEach - void setUp() throws Exception { - final ReferredTo referredTo = new ReferredTo(3); - List contained = Arrays.asList(new Contained("firstcontained"),new Contained("secondContained")); - List refcont = Arrays.asList(new ReferredLifeCycle("rc1"), new ReferredLifeCycle("rc2")); - atest = ATest.createATest(a -> { - a.ref1 = referredTo; - a.contained = contained; - a.refandcontained = refcont; - - }); - atest2 = new ATest2(atest, referredTo, refcont.get(0)); - atest3 = new ATest3(contained, refcont.get(0));//TODO this will create contradictions.... how best to test - - model = new LifecycleTestModel(); - model.addContent(atest); - model.addContent(atest2); - } - - /** - * @throws java.lang.Exception - */ - @AfterEach - void tearDown() throws Exception { - - - } - - @Test - void MultiContainedJPATest() { - jakarta.persistence.EntityManager em = setupH2Db(SampleModel.pu_name());//IMPL build means that everything is in one persistence unit. - em.getTransaction().begin(); - atest2.persistRefs(em); - atest.persistRefs(em); - em.persist(atest2); - em.persist(atest); - em.persist(atest3); - em.getTransaction().commit(); - Long id = atest2.getId(); - - //flush any existing entities - em.clear(); - em.getEntityManagerFactory().getCache().evictAll(); - - // now read back - em.getTransaction().begin(); - List ats = em.createNamedQuery("ATest2.findById", ATest2.class) - .setParameter("id", id).getResultList(); - em.getTransaction().commit(); - dumpDbData(em, "lifecycle_dump.sql"); - } - - @Test - void copyTest() { - model.createContext(); - ATest atestprime = new ATest(atest); - atest.ref1.test1 = 4; - assertEquals(4, atestprime.ref1.test1); //the reference should have changed - atest.contained.get(0).test2 = "changed"; - assertEquals("firstcontained",atestprime.contained.get(0).test2); // new objects created for the contained so changing original should not affect the prime - - - - - - ATest2 atest2prime = new ATest2(atest2); - - - atest2.atest.refandcontained.get(0).test3 = "changed2"; - assertEquals("changed2", atest2.refcont.test3); // this is in atest3 - - atest2prime.updateClonedReferences(); - assertEquals("rc1" ,atest2prime.atest.refandcontained.get(0).test3);// this is what we want the copied atest2 has its "own" contained references - assertEquals("rc1", atest2prime.refcont.test3); // should be pointing to above - - // assertEquals("rc1" ,atest3.refBad.test3);//TODO not sure which way we want these to work. - - - } - + private ATest atest; + private ATest2 atest2; + private LifecycleTestModel model; + private ATest3 atest3; -} + /** + * @throws java.lang.Exception + */ + @BeforeAll + static void setUpBeforeClass() throws Exception {} + + /** + * @throws java.lang.Exception + */ + @BeforeEach + void setUp() throws Exception { + final ReferredTo referredTo = new ReferredTo(3); + List contained = + Arrays.asList(new Contained("firstcontained"), new Contained("secondContained")); + List refcont = + Arrays.asList(new ReferredLifeCycle("rc1"), new ReferredLifeCycle("rc2")); + atest = + ATest.createATest( + a -> { + a.ref1 = referredTo; + a.contained = contained; + a.refandcontained = refcont; + }); + atest2 = new ATest2(atest, referredTo, refcont.get(0)); + atest3 = + new ATest3( + contained, refcont.get(0)); // TODO this will create contradictions.... how best to test + + model = new LifecycleTestModel(); + model.addContent(atest); + model.addContent(atest2); + } + + /** + * @throws java.lang.Exception + */ + @AfterEach + void tearDown() throws Exception {} + + @Test + void MultiContainedJPATest() { + jakarta.persistence.EntityManager em = + setupH2Db(SampleModel.pu_name()); // IMPL build means that everything is in one + // persistence unit. + em.getTransaction().begin(); + atest2.persistRefs(em); + atest.persistRefs(em); + em.persist(atest2); + em.persist(atest); + em.persist(atest3); + em.getTransaction().commit(); + Long id = atest2.getId(); + // flush any existing entities + em.clear(); + em.getEntityManagerFactory().getCache().evictAll(); + // now read back + em.getTransaction().begin(); + List ats = + em.createNamedQuery("ATest2.findById", ATest2.class).setParameter("id", id).getResultList(); + em.getTransaction().commit(); + dumpDbData(em, "lifecycle_dump.sql"); + } + + @Test + void copyTest() { + model.createContext(); + ATest atestprime = new ATest(atest); + atest.ref1.test1 = 4; + assertEquals(4, atestprime.ref1.test1); // the reference should have changed + atest.contained.get(0).test2 = "changed"; + assertEquals( + "firstcontained", + atestprime.contained.get(0) + .test2); // new objects created for the contained so changing original + // should not affect the prime + + ATest2 atest2prime = new ATest2(atest2); + + atest2.atest.refandcontained.get(0).test3 = "changed2"; + assertEquals("changed2", atest2.refcont.test3); // this is in atest3 + + atest2prime.updateClonedReferences(); + assertEquals( + "rc1", + atest2prime.atest.refandcontained.get(0) + .test3); // this is what we want the copied atest2 has its "own" contained + // references + assertEquals("rc1", atest2prime.refcont.test3); // should be pointing to above + + // assertEquals("rc1" ,atest3.refBad.test3);//TODO not sure which way we want these to work. + + } +} diff --git a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/lifecycle/LifecycleTestModelTest.java b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/lifecycle/LifecycleTestModelTest.java index bcd2105f..4bdbddfe 100644 --- a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/lifecycle/LifecycleTestModelTest.java +++ b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/lifecycle/LifecycleTestModelTest.java @@ -1,11 +1,11 @@ /* - * Created on 21 Oct 2022 + * Created on 21 Oct 2022 * Copyright 2022 Paul Harrison (paul.harrison@manchester.ac.uk) * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License in file LICENSE - */ + */ package org.ivoa.dm.lifecycle; @@ -13,91 +13,81 @@ import java.util.Arrays; import java.util.List; - import org.ivoa.vodml.testing.AutoRoundTripTest; import org.junit.jupiter.api.AfterEach; -import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.BeforeEach; /** * Test for lifecycle proposal . - * + * * JSON and Rdb serialization work with this model - however XML serialization does not - the ReferredLifeCycle("rc1") object gets represented twice. * The XML serialization appears to work properly, but the XML is not valid because there are two objects with the same id (they really are the same object of course). - * @author Paul Harrison (paul.harrison@manchester.ac.uk) + * @author Paul Harrison (paul.harrison@manchester.ac.uk) * @since 21 Oct 2022 */ class LifecycleTestModelTest extends AutoRoundTripTest { - private ATest atest; - private ATest2 atest2; - - /** - * @throws java.lang.Exception - */ - @BeforeAll - static void setUpBeforeClass() throws Exception { - } - - /** - * @throws java.lang.Exception - */ - @BeforeEach - void setUp() throws Exception { - - } - - /** - * @throws java.lang.Exception - */ - @AfterEach - void tearDown() throws Exception { - } - - - - /** - * {@inheritDoc} - * overrides @see org.ivoa.vodml.validation.AutoRoundTripTest#createModel() - */ - @Override - public LifecycleTestModel createModel() { - final ReferredTo referredTo = new ReferredTo(3); - List contained = Arrays.asList(new Contained("firstcontained"),new Contained("secondContained")); - List refcont = Arrays.asList(new ReferredLifeCycle("rc1"), new ReferredLifeCycle("rc2")); - atest = ATest.createATest(a -> { - a.ref1 = referredTo; - a.contained = contained; - a.refandcontained = refcont; - - }); - atest2 = new ATest2(atest,referredTo, refcont.get(0)); - - LifecycleTestModel model = new LifecycleTestModel(); - model.addContent(atest); - model.addContent(atest2); - model.processReferences(); - assertTrue(atest.refandcontained.get(1).getId() != 0, "id setting did not work"); - return model; - } - - /** - * {@inheritDoc} - * overrides @see org.ivoa.vodml.validation.AutoRoundTripTest#testModel(org.ivoa.vodml.VodmlModel) - */ - @Override - public void testModel(LifecycleTestModel m) { - - List ratest = m.getContent(ATest.class); - ratest.get(0).getRefandcontained().get(0).setTest3("changed"); - List ratest2 = m.getContent(ATest2.class); - m.processReferences(); - System.out.println("ref and contained val ="+ratest2.get(0).getRefcont().getTest3()); - - - - } - + private ATest atest; + private ATest2 atest2; + + /** + * @throws java.lang.Exception + */ + @BeforeAll + static void setUpBeforeClass() throws Exception {} + + /** + * @throws java.lang.Exception + */ + @BeforeEach + void setUp() throws Exception {} + + /** + * @throws java.lang.Exception + */ + @AfterEach + void tearDown() throws Exception {} + + /** + * {@inheritDoc} + * overrides @see org.ivoa.vodml.validation.AutoRoundTripTest#createModel() + */ + @Override + public LifecycleTestModel createModel() { + final ReferredTo referredTo = new ReferredTo(3); + List contained = + Arrays.asList(new Contained("firstcontained"), new Contained("secondContained")); + List refcont = + Arrays.asList(new ReferredLifeCycle("rc1"), new ReferredLifeCycle("rc2")); + atest = + ATest.createATest( + a -> { + a.ref1 = referredTo; + a.contained = contained; + a.refandcontained = refcont; + }); + atest2 = new ATest2(atest, referredTo, refcont.get(0)); + + LifecycleTestModel model = new LifecycleTestModel(); + model.addContent(atest); + model.addContent(atest2); + model.processReferences(); + assertTrue(atest.refandcontained.get(1).getId() != 0, "id setting did not work"); + return model; + } + + /** + * {@inheritDoc} + * overrides @see org.ivoa.vodml.validation.AutoRoundTripTest#testModel(org.ivoa.vodml.VodmlModel) + */ + @Override + public void testModel(LifecycleTestModel m) { + + List ratest = m.getContent(ATest.class); + ratest.get(0).getRefandcontained().get(0).setTest3("changed"); + List ratest2 = m.getContent(ATest2.class); + m.processReferences(); + System.out.println("ref and contained val =" + ratest2.get(0).getRefcont().getTest3()); + } } - - diff --git a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/notstccoords/CoordsModelTest.java b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/notstccoords/CoordsModelTest.java index 8806d865..47536bf1 100644 --- a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/notstccoords/CoordsModelTest.java +++ b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/notstccoords/CoordsModelTest.java @@ -1,11 +1,11 @@ /* - * Created on 5 Nov 2021 + * Created on 5 Nov 2021 * Copyright 2021 Paul Harrison (paul.harrison@manchester.ac.uk) * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License in file LICENSE - */ + */ package org.ivoa.dm.notstccoords; @@ -16,66 +16,69 @@ /** * An example test for the "not coords" model. * note that this test runs JSON and XML serialisation test as well as validating the model instance. - * @author Paul Harrison (paul.harrison@manchester.ac.uk) + * @author Paul Harrison (paul.harrison@manchester.ac.uk) * @since 5 Nov 2021 */ class CoordsModelTest extends AutoRoundTripWithValidationTest { - /** logger for this class */ - private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory - .getLogger(CoordsModelTest.class); - - @Override - public CoordsModel createModel() { - //see https://github.com/mcdittmar/ivoa-dm-examples/blob/master/assets/examples/coords/current/instances/astrocoordsys.jovial for jovial version of this test. - Unit deg = new Unit("deg"); - SpaceSys ICRS_SYS = new SpaceSys().withFrame( - SpaceFrame.createSpaceFrame( f-> { - f.refPosition = new StdRefLocation("TOPOCENTRE"); - f.spaceRefFrame="ICRS"; - f.planetaryEphem="DE432"; - } - )); + /** logger for this class */ + private static final org.slf4j.Logger logger = + org.slf4j.LoggerFactory.getLogger(CoordsModelTest.class); - TimeSys TIMESYS_TT = new TimeSys().withFrame( - TimeFrame.createTimeFrame( f -> { - f.refPosition = new StdRefLocation("TOPOCENTRE"); - f.timescale = "TT"; - f.refDirection = new CustomRefLocation() - .withEpoch("J2014.25") - .withPosition( - LonLatPoint.createLonLatPoint(p-> { - p.lon = new RealQuantity(6.752477,deg); - p.lat = new RealQuantity(-16.716116,deg); - p.dist = new RealQuantity(8.6, new Unit("ly")); - p.coordSys = ICRS_SYS; - } - ) - ); - }) - ); - GenericSys SPECSYS = new GenericSys().withFrame( - GenericFrame.createGenericFrame(f -> { - f.refPosition = new StdRefLocation("TOPOCENTRE"); - f.planetaryEphem = "DE432"; - } - ) - ); + @Override + public CoordsModel createModel() { + // see + // https://github.com/mcdittmar/ivoa-dm-examples/blob/master/assets/examples/coords/current/instances/astrocoordsys.jovial for jovial version of this test. + Unit deg = new Unit("deg"); + SpaceSys ICRS_SYS = + new SpaceSys() + .withFrame( + SpaceFrame.createSpaceFrame( + f -> { + f.refPosition = new StdRefLocation("TOPOCENTRE"); + f.spaceRefFrame = "ICRS"; + f.planetaryEphem = "DE432"; + })); + TimeSys TIMESYS_TT = + new TimeSys() + .withFrame( + TimeFrame.createTimeFrame( + f -> { + f.refPosition = new StdRefLocation("TOPOCENTRE"); + f.timescale = "TT"; + f.refDirection = + new CustomRefLocation() + .withEpoch("J2014.25") + .withPosition( + LonLatPoint.createLonLatPoint( + p -> { + p.lon = new RealQuantity(6.752477, deg); + p.lat = new RealQuantity(-16.716116, deg); + p.dist = new RealQuantity(8.6, new Unit("ly")); + p.coordSys = ICRS_SYS; + })); + })); + GenericSys SPECSYS = + new GenericSys() + .withFrame( + GenericFrame.createGenericFrame( + f -> { + f.refPosition = new StdRefLocation("TOPOCENTRE"); + f.planetaryEphem = "DE432"; + })); - CoordsModel modelInstance = new CoordsModel(); + CoordsModel modelInstance = new CoordsModel(); - modelInstance.addReference(TIMESYS_TT); - modelInstance.addReference(SPECSYS); - modelInstance.addReference(ICRS_SYS); - modelInstance.processReferences(); - return modelInstance; - } + modelInstance.addReference(TIMESYS_TT); + modelInstance.addReference(SPECSYS); + modelInstance.addReference(ICRS_SYS); + modelInstance.processReferences(); + return modelInstance; + } - @Override - public void testModel(CoordsModel coordsModel) { - //TODO actually make some specialist tests on returned model instance. - } + @Override + public void testModel(CoordsModel coordsModel) { + // TODO actually make some specialist tests on returned model instance. + } } - - diff --git a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/sample/catalog/BaseSourceCatalogueTest.java b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/sample/catalog/BaseSourceCatalogueTest.java index 29e07a92..5194a0fd 100644 --- a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/sample/catalog/BaseSourceCatalogueTest.java +++ b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/sample/catalog/BaseSourceCatalogueTest.java @@ -1,11 +1,11 @@ /* - * Created on 10 Nov 2022 + * Created on 10 Nov 2022 * Copyright 2022 Paul Harrison (paul.harrison@manchester.ac.uk) * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License in file LICENSE - */ + */ package org.ivoa.dm.sample.catalog; @@ -19,7 +19,6 @@ import java.util.Arrays; import java.util.Date; import java.util.List; - import org.ivoa.dm.filter.PhotometricSystem; import org.ivoa.dm.filter.PhotometryFilter; import org.ivoa.dm.ivoa.RealQuantity; @@ -29,99 +28,116 @@ /** * . - * @author Paul Harrison (paul.harrison@manchester.ac.uk) + * @author Paul Harrison (paul.harrison@manchester.ac.uk) * @since 10 Nov 2022 */ public abstract class BaseSourceCatalogueTest extends AbstractTest { - protected SourceCatalogue sc; - protected PhotometricSystem ps; + protected SourceCatalogue sc; + protected PhotometricSystem ps; - @org.junit.jupiter.api.BeforeEach - void setUp() { - final Unit jansky = new Unit("Jy"); - final Unit degree = new Unit("degree"); - final Unit GHz= new Unit("GHz"); - final SkyCoordinateFrame frame = new SkyCoordinateFrame().withName("J2000").withEquinox("J2000.0").withDocumentURI("http://coord.net"); - - final AlignedEllipse ellipseError = new AlignedEllipse(.2, .1); - SDSSSource sdss = new SDSSSource().withPositionError(ellipseError);// UNUSED, but just checking position error subsetting. - sdss.setPositionError(ellipseError); - AlignedEllipse theError = sdss.getPositionError(); - - final List filters = List.of( - createPhotometryFilter(fl -> { - fl.bandName ="C-Band"; - fl.spectralLocation = new RealQuantity(5.0,GHz); - fl.dataValidityFrom = new Date(); - fl.dataValidityTo = new Date(); - fl.description = "radio band"; - fl.name = fl.bandName; - }), - createPhotometryFilter(fl -> { - fl.bandName ="L-Band"; - fl.spectralLocation = new RealQuantity(1.5,GHz); - fl.dataValidityFrom = new Date(); - fl.dataValidityTo = new Date(); - fl.description = "radio band"; - fl.name = fl.bandName; - }) - ); - - - ps = new PhotometricSystem("test photometric system", 1, filters); - sc = createSourceCatalogue(c -> { - c.name = "testCat"; - c.entry = Arrays.asList(createSDSSSource(s -> { - s.name = "testSource"; - s.classification = SourceClassification.AGN; - s.position = createSkyCoordinate(co -> { - co.frame = frame; - co.latitude = new RealQuantity(52.5, degree ); - co.longitude = new RealQuantity(2.5, degree ); - }); - s.positionError = ellipseError;//note subsetting forces compile need AlignedEllipse - - s.luminosity = Arrays.asList( - createLuminosityMeasurement(l ->{ - l.description = "lummeas"; - l.type = LuminosityType.FLUX; - l.value = new RealQuantity(2.5, jansky ); - l.error = new RealQuantity(.25, jansky ); - l.filter = filters.get(0); - - }) - ,createLuminosityMeasurement(l ->{ - l.description = "lummeas2"; - l.filter = filters.get(1); - l.type = LuminosityType.FLUX; - l.value = new RealQuantity(3.5, jansky ); - l.error = new RealQuantity(.25, jansky );//TODO should be allowed to be null - - }) - - ); - })); - } - ); - - - } + @org.junit.jupiter.api.BeforeEach + void setUp() { + final Unit jansky = new Unit("Jy"); + final Unit degree = new Unit("degree"); + final Unit GHz = new Unit("GHz"); + final SkyCoordinateFrame frame = + new SkyCoordinateFrame() + .withName("J2000") + .withEquinox("J2000.0") + .withDocumentURI("http://coord.net"); - protected void checkModel(List lin) { - assertEquals(1, lin.size()); - SourceCatalogue scin = lin.get(0); - System.out.println(lin.get(0).getName()); - SDSSSource src = (SDSSSource) scin.getEntry().get(0); - AlignedEllipse perr = src.getPositionError(); - assertEquals(0.2, perr.longError); - assertTrue(!src.getLuminosity().get(0).getFilter().getBandName().equals( - src.getLuminosity().get(1).getFilter().getBandName()),"failure to distinguish references"); - SkyCoordinateFrame fr = src.getPosition().getFrame(); - assertNotNull(fr); - assertEquals("J2000", fr.getName()); - } + final AlignedEllipse ellipseError = new AlignedEllipse(.2, .1); + SDSSSource sdss = + new SDSSSource() + .withPositionError(ellipseError); // UNUSED, but just checking position error + // subsetting. + sdss.setPositionError(ellipseError); + AlignedEllipse theError = sdss.getPositionError(); -} + final List filters = + List.of( + createPhotometryFilter( + fl -> { + fl.bandName = "C-Band"; + fl.spectralLocation = new RealQuantity(5.0, GHz); + fl.dataValidityFrom = new Date(); + fl.dataValidityTo = new Date(); + fl.description = "radio band"; + fl.name = fl.bandName; + }), + createPhotometryFilter( + fl -> { + fl.bandName = "L-Band"; + fl.spectralLocation = new RealQuantity(1.5, GHz); + fl.dataValidityFrom = new Date(); + fl.dataValidityTo = new Date(); + fl.description = "radio band"; + fl.name = fl.bandName; + })); + + ps = new PhotometricSystem("test photometric system", 1, filters); + sc = + createSourceCatalogue( + c -> { + c.name = "testCat"; + c.entry = + Arrays.asList( + createSDSSSource( + s -> { + s.name = "testSource"; + s.classification = SourceClassification.AGN; + s.position = + createSkyCoordinate( + co -> { + co.frame = frame; + co.latitude = new RealQuantity(52.5, degree); + co.longitude = new RealQuantity(2.5, degree); + }); + s.positionError = ellipseError; // note subsetting + // forces compile need + // AlignedEllipse + s.luminosity = + Arrays.asList( + createLuminosityMeasurement( + l -> { + l.description = "lummeas"; + l.type = LuminosityType.FLUX; + l.value = new RealQuantity(2.5, jansky); + l.error = new RealQuantity(.25, jansky); + l.filter = filters.get(0); + }), + createLuminosityMeasurement( + l -> { + l.description = "lummeas2"; + l.filter = filters.get(1); + l.type = LuminosityType.FLUX; + l.value = new RealQuantity(3.5, jansky); + l.error = + new RealQuantity( + .25, jansky); // TODO should be allowed to be null + })); + })); + }); + } + protected void checkModel(List lin) { + assertEquals(1, lin.size()); + SourceCatalogue scin = lin.get(0); + System.out.println(lin.get(0).getName()); + SDSSSource src = (SDSSSource) scin.getEntry().get(0); + AlignedEllipse perr = src.getPositionError(); + assertEquals(0.2, perr.longError); + assertTrue( + !src.getLuminosity() + .get(0) + .getFilter() + .getBandName() + .equals(src.getLuminosity().get(1).getFilter().getBandName()), + "failure to distinguish references"); + SkyCoordinateFrame fr = src.getPosition().getFrame(); + assertNotNull(fr); + assertEquals("J2000", fr.getName()); + } +} diff --git a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/sample/catalog/SourceCatalogueTest.java b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/sample/catalog/SourceCatalogueTest.java index 71afd408..a06b44e5 100644 --- a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/sample/catalog/SourceCatalogueTest.java +++ b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/sample/catalog/SourceCatalogueTest.java @@ -1,190 +1,278 @@ package org.ivoa.dm.sample.catalog; -import static org.junit.jupiter.api.Assertions.*; - -import java.io.IOException; -import java.util.List; +import static org.junit.jupiter.api.Assertions.*; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.networknt.schema.*; +import com.networknt.schema.output.OutputUnit; import jakarta.xml.bind.JAXBContext; import jakarta.xml.bind.JAXBException; +import jakarta.xml.bind.Marshaller; +import java.io.File; +import java.io.FileWriter; +import java.io.IOException; +import java.io.InputStream; +import java.io.StringWriter; +import java.util.List; import javax.xml.parsers.ParserConfigurationException; import javax.xml.transform.TransformerException; - -import com.fasterxml.jackson.core.JsonProcessingException; - import org.ivoa.dm.sample.SampleModel; import org.ivoa.dm.sample.catalog.inner.SourceCatalogue; -import org.ivoa.vodml.VodmlModel; +import org.ivoa.vodml.validation.ModelValidator; +import org.ivoa.vodml.validation.ModelValidator.ValidationResult; /* * Created on 20/08/2021 by Paul Harrison (paul.harrison@manchester.ac.uk). */ class SourceCatalogueTest extends BaseSourceCatalogueTest { - /** logger for this class */ - private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory - .getLogger(SourceCatalogueTest.class); - @org.junit.jupiter.api.Test - void sourceCatJaxBTest() throws JAXBException, ParserConfigurationException, TransformerException, IOException { - - logger.debug("starting JAXB test"); - - JAXBContext jc = SampleModel.contextFactory(); - - SampleModel model = new SampleModel(); - model.addContent(sc); - model.addContent(ps); - model.processReferences(); - - SampleModel modelin = modelRoundTripXMLwithTest(model); - checkModel(modelin.getContent(SourceCatalogue.class)); - System.out.println("generating schema"); - SampleModel.writeXMLSchema(); + /** logger for this class */ + private static final org.slf4j.Logger logger = + org.slf4j.LoggerFactory.getLogger(SourceCatalogueTest.class); + + @org.junit.jupiter.api.Test + void sourceCatJaxBTest() + throws JAXBException, ParserConfigurationException, TransformerException, IOException { + + logger.debug("starting JAXB test"); + + JAXBContext jc = SampleModel.contextFactory(); + + SampleModel model = new SampleModel(); + model.addContent(ps); + model.addContent(sc); + + model.processReferences(); + + SampleModel modelin = modelRoundTripXMLwithTest(model); + checkModel(modelin.getContent(SourceCatalogue.class)); + System.out.println("generating schema"); + SampleModel.writeXMLSchema(); + } + + @org.junit.jupiter.api.Test + void sourceCatJPATest() { + jakarta.persistence.EntityManager em = setupH2Db(SampleModel.pu_name()); + em.getTransaction().begin(); + sc.persistRefs(em); + em.persist(sc); // TODO need to test whether Photometric system is saved.... + em.getTransaction().commit(); + Long id = sc.getId(); + + // flush any existing entities + em.clear(); + em.getEntityManagerFactory().getCache().evictAll(); + + // now read back + em.getTransaction().begin(); + List cats = + em.createNamedQuery("SourceCatalogue.findById", SourceCatalogue.class) + .setParameter("id", id) + .getResultList(); + checkModel(cats); + + // now try to add into a new model + SampleModel model = new SampleModel(); + for (SourceCatalogue c : cats) { + c.forceLoad(); // force any lazy loading to happen + model.addContent(c); + } + + em.getTransaction().commit(); + dumpDbData(em, "test_dump.sql"); + } + + @org.junit.jupiter.api.Test + void sourceCatJSONTest() throws JsonProcessingException { + SampleModel model = new SampleModel(); + model.addContent(sc); + model.processReferences(); + SampleModel modelin = modelRoundTripJSONwithTest(model); + checkModel(modelin.getContent(SourceCatalogue.class)); + } + + @org.junit.jupiter.api.Test + void sourceCatDeleteTest() throws JsonProcessingException { + SampleModel model = new SampleModel(); + model.addContent(sc); + model.processReferences(); + model.deleteContent(sc); // + SampleModel modelin = + modelRoundTripJSONwithTest(model); // FIXME need to test that the refenences are gone + } + + @org.junit.jupiter.api.Test + void sourceCatJPACloneTest() throws JsonProcessingException { + SampleModel model = new SampleModel(); + jakarta.persistence.EntityManager em = setupH2Db(SampleModel.pu_name()); + em.getTransaction().begin(); + sc.persistRefs(em); + em.persist(sc); + em.getTransaction().commit(); + model.addContent(sc); + + em.getTransaction().begin(); + sc.jpaClone(em); + sc.setName("cloned catalogue"); + sc.getEntry().get(0).setName("cloned source"); + em.merge(sc); + em.getTransaction().commit(); + model.addContent(sc); + // note that sc gets updated by the clone - so would appear twice in the following + // SampleModel modelin = roundTripJSON(model.management()); + + List cats = + em.createQuery("select s from SourceCatalogue s", SourceCatalogue.class).getResultList(); + model = new SampleModel(); + for (SourceCatalogue s : cats) { + model.addContent(s); } - @org.junit.jupiter.api.Test - void sourceCatJPATest() { - jakarta.persistence.EntityManager em = setupH2Db(SampleModel.pu_name()); - em.getTransaction().begin(); - sc.persistRefs(em); - em.persist(sc); // TODO need to test whether Photometric system is saved.... - em.getTransaction().commit(); - Long id = sc.getId(); + SampleModel modelin = modelRoundTripJSONwithTest(model); + assertNotNull(modelin); + long ncat = (long) em.createQuery("select count(o) from SourceCatalogue o").getSingleResult(); + assertEquals(2, ncat, "number of catalogues"); + long nsrc = (long) em.createQuery("select count(o) from SDSSSource o").getSingleResult(); + assertEquals(2, nsrc, "number of sources"); + } - //flush any existing entities - em.clear(); - em.getEntityManagerFactory().getCache().evictAll(); + @org.junit.jupiter.api.Test + void sourceCatCopyTest() throws JsonProcessingException { + SourceCatalogue newsc = new SourceCatalogue(sc); + assertNotNull(newsc); + assertNotNull(newsc.getEntry().get(0).position); + sc.setName("sillytest"); + assertEquals("testCat", newsc.getName()); + SampleModel model = new SampleModel(); + model.addContent(sc); + model.addContent(newsc); + model.processReferences(); + SampleModel modelin = modelRoundTripJSONwithTest(model); + assertNotNull(modelin); + } - // now read back - em.getTransaction().begin(); - List cats = em.createNamedQuery("SourceCatalogue.findById", SourceCatalogue.class) - .setParameter("id", id).getResultList(); - checkModel(cats); + @org.junit.jupiter.api.Test + void listManipulationTest() { + jakarta.persistence.EntityManager em = setupH2Db(SampleModel.pu_name()); + em.getTransaction().begin(); + sc.persistRefs(em); + em.persist(sc); + em.getTransaction().commit(); + List ls = sc.getEntry(); + List lms = ls.get(0).getLuminosity(); + // copy constructor creates an object not + LuminosityMeasurement lumnew = new LuminosityMeasurement(lms.get(0)); + lumnew.setDescription("this is new"); + // merge in changes to the existing db + lms.get(0).updateUsing(lumnew); + em.getTransaction().begin(); + em.merge(lms.get(0)); + em.getTransaction().commit(); + em.clear(); + // read back from db + List cats = + em.createQuery("select s from SourceCatalogue s", SourceCatalogue.class).getResultList(); + assertEquals( + "this is new", cats.get(0).getEntry().get(0).getLuminosity().get(0).getDescription()); - // now try to add into a new model - SampleModel model = new SampleModel(); - for(SourceCatalogue c :cats) { - c.forceLoad();//force any lazy loading to happen - model.addContent(c); - } + lumnew.setDescription("another way to update"); + lumnew._id = + lms.get(0)._id; // NB this setting of IDs directly not available in end DM API - only + // possible because of shared package + ls.get(0).replaceInLuminosity(lumnew); - em.getTransaction().commit(); - dumpDbData(em, "test_dump.sql"); - + em.getTransaction().begin(); + em.merge(lms.get(0)); + em.getTransaction().commit(); + em.clear(); + List cats2 = + em.createQuery("select s from SourceCatalogue s", SourceCatalogue.class).getResultList(); + assertEquals( + "another way to update", + cats2.get(0).getEntry().get(0).getLuminosity().get(0).getDescription()); + } + + @org.junit.jupiter.api.Test + void externalXMLSchemaTest() throws JAXBException, IOException { + + SampleModel model = new SampleModel(); + model.addContent(sc); + model.addContent(ps); + model.processReferences(); + + String topschema = model.descriptor().schemaMap().get(model.descriptor().xmlNamespace()); + System.out.println(topschema); + assertNotNull(topschema); + InputStream schemastream = this.getClass().getResourceAsStream("/" + topschema); + assertNotNull(schemastream); + + ModelValidator validator = new ModelValidator(model); + Marshaller jaxbMarshaller = model.management().contextFactory().createMarshaller(); + jaxbMarshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); + File fout = File.createTempFile("samplemod", ".tmp"); + FileWriter fw = new FileWriter(fout); + jaxbMarshaller.marshal(model, fw); + ValidationResult result = validator.validate(fout); + if (!result.isOk) { + System.err.println("File " + fout.getAbsolutePath()); + result.printValidationErrors(System.err); } + assertTrue(result.isOk, "validation with external schema"); + } + + @org.junit.jupiter.api.Test + void externalJSONSchemaTest() throws JAXBException, IOException { - @org.junit.jupiter.api.Test - void sourceCatJSONTest() throws JsonProcessingException { - SampleModel model = new SampleModel(); - model.addContent(sc); - model.processReferences(); - SampleModel modelin = modelRoundTripJSONwithTest(model); - checkModel(modelin.getContent(SourceCatalogue.class)); - - } - - @org.junit.jupiter.api.Test - void sourceCatDeleteTest() throws JsonProcessingException { - SampleModel model = new SampleModel(); - model.addContent(sc); - model.processReferences(); - model.deleteContent(sc); // - SampleModel modelin = modelRoundTripJSONwithTest(model); // FIXME need to test that the refenences are gone - - } - - @org.junit.jupiter.api.Test - void sourceCatJPACloneTest() throws JsonProcessingException { - SampleModel model = new SampleModel(); - jakarta.persistence.EntityManager em = setupH2Db(SampleModel.pu_name()); - em.getTransaction().begin(); - sc.persistRefs(em); - em.persist(sc); - em.getTransaction().commit(); - model.addContent(sc); - - em.getTransaction().begin(); - sc.jpaClone(em); - sc.setName("cloned catalogue"); - sc.getEntry().get(0).setName("cloned source"); - em.merge(sc); - em.getTransaction().commit(); - model.addContent(sc); - // note that sc gets updated by the clone - so would appear twice in the following -// SampleModel modelin = roundTripJSON(model.management()); - - List cats = em.createQuery("select s from SourceCatalogue s", SourceCatalogue.class).getResultList(); - model = new SampleModel(); - for (SourceCatalogue s : cats) { - model.addContent(s); - } - - SampleModel modelin = modelRoundTripJSONwithTest(model); - assertNotNull(modelin); - long ncat = (long) em.createQuery("select count(o) from SourceCatalogue o").getSingleResult(); - assertEquals(2, ncat,"number of catalogues"); - long nsrc = (long) em.createQuery("select count(o) from SDSSSource o").getSingleResult(); - assertEquals(2, nsrc,"number of sources"); - - } - - @org.junit.jupiter.api.Test - void sourceCatCopyTest() throws JsonProcessingException - { - SourceCatalogue newsc = new SourceCatalogue(sc); - assertNotNull(newsc); - assertNotNull(newsc.getEntry().get(0).position); - sc.setName("sillytest"); - assertEquals("testCat", newsc.getName()); - SampleModel model = new SampleModel(); - model.addContent(sc); - model.addContent(newsc); - model.processReferences(); - SampleModel modelin = modelRoundTripJSONwithTest(model); - assertNotNull(modelin); - } - - @org.junit.jupiter.api.Test - void listManipulationTest() - { - jakarta.persistence.EntityManager em = setupH2Db(SampleModel.pu_name()); - em.getTransaction().begin(); - sc.persistRefs(em); - em.persist(sc); - em.getTransaction().commit(); - List ls = sc.getEntry(); - List lms = ls.get(0).getLuminosity(); - //copy constructor creates an object not - LuminosityMeasurement lumnew = new LuminosityMeasurement(lms.get(0)); - lumnew.setDescription("this is new"); - // merge in changes to the existing db - lms.get(0).updateUsing(lumnew); - em.getTransaction().begin(); - em.merge(lms.get(0)); - em.getTransaction().commit(); - em.clear(); - - // read back from db - List cats = em.createQuery("select s from SourceCatalogue s", SourceCatalogue.class).getResultList(); - assertEquals("this is new", cats.get(0).getEntry().get(0).getLuminosity().get(0).getDescription()); - - lumnew.setDescription("another way to update"); - lumnew._id = lms.get(0)._id; //NB this setting of IDs directly not available in end DM API - only possible because of shared package - - ls.get(0).replaceInLuminosity(lumnew); - - em.getTransaction().begin(); - em.merge(lms.get(0)); - em.getTransaction().commit(); - em.clear(); - - List cats2 = em.createQuery("select s from SourceCatalogue s", SourceCatalogue.class).getResultList(); - assertEquals("another way to update", cats2.get(0).getEntry().get(0).getLuminosity().get(0).getDescription()); - - - } + SampleModel model = new SampleModel(); + model.addContent( + ps); // IMPL N.B add the photometric system first, as it has sc used it in contained + // references - it does work the other way round, but looks strange.... + model.addContent(sc); + model.processReferences(); + + ObjectMapper mapper = model.management().jsonMapper(); + StringWriter fw = new StringWriter(); + mapper.writerWithDefaultPrettyPrinter().writeValue(fw, model); + System.out.println(fw.toString()); + JsonSchemaFactory jsonSchemaFactory = + JsonSchemaFactory.getInstance( + SpecVersion.VersionFlag.V202012, + builder -> + // This creates a mapping from $id which starts with + // https://www.example.org/ to the retrieval URI classpath:schema/ + builder.schemaMappers( + schemaMappers -> + schemaMappers.mapPrefix("https://ivoa.net/dm/", "classpath:/"))); + SchemaValidatorsConfig config = new SchemaValidatorsConfig(); + // By default JSON Path is used for reporting the instance location and evaluation path + config.setPathType(PathType.JSON_POINTER); + + // Due to the mapping the schema will be retrieved from the classpath at + // classpath:schema/example-main.json. + // If the schema data does not specify an $id the absolute IRI of the schema location will + // be used as the $id. + JsonSchema schema = + jsonSchemaFactory.getSchema( + SchemaLocation.of("https://ivoa.net/dm/Sample.vo-dml.json"), config); + OutputUnit outputUnit = + schema.validate( + fw.toString(), + InputFormat.JSON, + OutputFormat.HIERARCHICAL, + executionContext -> { + // By default since Draft 2019-09 the format keyword only generates + // annotations and not assertions + executionContext.getExecutionConfig().setFormatAssertionsEnabled(true); + }); + + if (!outputUnit.isValid()) { + System.err.println(outputUnit.toString()); + } + assertTrue(outputUnit.isValid()); + } } diff --git a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/serializationsample/SerializationExampleTest.java b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/serializationsample/SerializationExampleTest.java index d733ac71..4a529ef5 100644 --- a/tools/gradletooling/sample/src/test/java/org/ivoa/dm/serializationsample/SerializationExampleTest.java +++ b/tools/gradletooling/sample/src/test/java/org/ivoa/dm/serializationsample/SerializationExampleTest.java @@ -1,35 +1,33 @@ package org.ivoa.dm.serializationsample; -import org.junit.jupiter.api.Test; - import java.util.List; +import org.junit.jupiter.api.Test; /* * Created on 16/05/2023 by Paul Harrison (paul.harrison@manchester.ac.uk). */ -public class SerializationExampleTest extends org.ivoa.vodml.testing.AutoRoundTripTest{ - - @Override - public MyModelModel createModel() { - MyModelModel retval = new MyModelModel(); - Refa refa = new Refa("a value"); - Refb refb = new Refb("a name", "another val"); +public class SerializationExampleTest + extends org.ivoa.vodml.testing.AutoRoundTripTest { - List cc = List.of(new Dcont("a D","dval"), new Econt("an E", "eval")); + @Override + public MyModelModel createModel() { + MyModelModel retval = new MyModelModel(); + Refa refa = new Refa("a value"); + Refb refb = new Refb("a name", "another val"); - SomeContent c = new SomeContent("a z val", cc, refa, refb); - retval.addContent(c); + List cc = List.of(new Dcont("a D", "dval"), new Econt("an E", "eval")); - return retval; - } + SomeContent c = new SomeContent("a z val", cc, refa, refb); + retval.addContent(c); - @Override - public void testModel(org.ivoa.dm.serializationsample.MyModelModel myModelModel) { + return retval; + } - } + @Override + public void testModel(org.ivoa.dm.serializationsample.MyModelModel myModelModel) {} - @Test - public void testStandaloneList(){ - List cc = List.of(new Dcont("a D","dval"), new Econt("an E", "eval")); - } + @Test + public void testStandaloneList() { + List cc = List.of(new Dcont("a D", "dval"), new Econt("an E", "eval")); + } } diff --git a/tools/xslt/binding_setup.xsl b/tools/xslt/binding_setup.xsl index 4af86438..9b98d4ef 100644 --- a/tools/xslt/binding_setup.xsl +++ b/tools/xslt/binding_setup.xsl @@ -12,10 +12,8 @@ - setting mapping - @@ -24,13 +22,11 @@ - setting models - opening file diff --git a/tools/xslt/common-binding.xsl b/tools/xslt/common-binding.xsl index 8537b4f2..f619c405 100644 --- a/tools/xslt/common-binding.xsl +++ b/tools/xslt/common-binding.xsl @@ -236,14 +236,32 @@ + + + + + + + + + + + + + + + + + - - - - + + + + + @@ -275,6 +293,9 @@ + + + @@ -295,6 +316,9 @@ + + + unknown language @@ -302,6 +326,88 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -389,17 +495,30 @@ + + + + - - - - - + + + - - - + + + + + + + + + + + + + + diff --git a/tools/xslt/common-structure-functions.xsl b/tools/xslt/common-structure-functions.xsl index 179d7c12..17911525 100644 --- a/tools/xslt/common-structure-functions.xsl +++ b/tools/xslt/common-structure-functions.xsl @@ -145,7 +145,7 @@ note - only define functions in here as it is included in the schematron rules - + @@ -165,6 +165,60 @@ note - only define functions in here as it is included in the schematron rules + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + models in scope= + + + + + model references type= contained= ok= + + + + ** refs + + + + + + + + + + + + diff --git a/tools/xslt/create-catalogues.xsl b/tools/xslt/create-catalogues.xsl new file mode 100644 index 00000000..e012b45a --- /dev/null +++ b/tools/xslt/create-catalogues.xsl @@ -0,0 +1,40 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/tools/xslt/jaxb.xsl b/tools/xslt/jaxb.xsl index d53260db..1a4d787c 100644 --- a/tools/xslt/jaxb.xsl +++ b/tools/xslt/jaxb.xsl @@ -29,10 +29,13 @@ + @jakarta.xml.bind.annotation.XmlAccessorType( jakarta.xml.bind.annotation.XmlAccessType.NONE ) - @jakarta.xml.bind.annotation.XmlType( name = "") - + @jakarta.xml.bind.annotation.XmlType( name = "" + + + ) @jakarta.xml.bind.annotation.XmlSeeAlso({ }) @@ -72,7 +75,7 @@ - @jakarta.xml.bind.annotation.XmlType( name = "") + @jakarta.xml.bind.annotation.XmlType( name = "") - - - - - - @jakarta.xml.bind.annotation.XmlElement( name = "", required = , type = .class) - - @jakarta.xml.bind.annotation.XmlID - - + + + + + + + + @jakarta.xml.bind.annotation.XmlAttribute(name = "", required =) + + + @jakarta.xml.bind.annotation.XmlElement( name = "", required =, type = .class) + + + + + @jakarta.xml.bind.annotation.XmlID + + @@ -165,32 +178,11 @@ Writing to Overall Model file - - - - - models in scope= - - - - - model references type= contained= ok= - - - - ** refs - - - - - - - - + + - + filtered refs= @@ -243,7 +235,7 @@ @XmlType public static class References { - @XmlElement + @XmlElement(name="") @JsonProperty("") @@ -263,11 +255,12 @@ ).collect( Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)); - + @XmlElements(value = { - @XmlElement(name="", - type = .class) + + @XmlElement(name="", + type = .class) , }) @@ -401,24 +394,15 @@ @Override public Map<String, String> schemaMap() { final Map<String,String> schemaMap = new HashMap<>(); - - - - schemaMap.put("",""); - - - - schemaMap.put("",""); - - - + + schemaMap.put(); return schemaMap; } @Override public String xmlNamespace() { - return ""; + return ""; } diff --git a/tools/xslt/jpa.xsl b/tools/xslt/jpa.xsl index f580fe24..84b39bf9 100644 --- a/tools/xslt/jpa.xsl +++ b/tools/xslt/jpa.xsl @@ -168,7 +168,10 @@ @jakarta.persistence.Basic( optional = ) - @jakarta.persistence.Column( name = "", nullable = ) + @jakarta.persistence.Column( name = "", nullable = + + , length= + ) @jakarta.persistence.Embedded @@ -419,7 +422,6 @@ - Opening file diff --git a/tools/xslt/vo-dml-v1.0.sch.xml b/tools/xslt/vo-dml-v1.0.sch.xml index f4144c1a..45a4d68b 100644 --- a/tools/xslt/vo-dml-v1.0.sch.xml +++ b/tools/xslt/vo-dml-v1.0.sch.xml @@ -152,7 +152,7 @@ type of of reference is not an object type but a '' - + Reference used in is already use in unrelated composition which has lifecycle implications (i.e. the reference could disappear unless code is aware of relationship) diff --git a/tools/xslt/vo-dml2java.xsl b/tools/xslt/vo-dml2java.xsl index f2520eda..4a4edf27 100644 --- a/tools/xslt/vo-dml2java.xsl +++ b/tools/xslt/vo-dml2java.xsl @@ -636,11 +636,11 @@ * getter for XMLID */ - @jakarta.xml.bind.annotation.XmlAttribute(name = "id" ) + @jakarta.xml.bind.annotation.XmlAttribute(name = "_id" ) @jakarta.xml.bind.annotation.XmlID @Override public String getXmlId(){ - return org.ivoa.vodml.jaxb.XmlIdManagement.createXMLId(_id); + return org.ivoa.vodml.jaxb.XmlIdManagement.createXMLId(_id, this.getClass()); } @Override public void setXmlId (String id) @@ -757,6 +757,7 @@ package ; * Return the string representation of this enum constant (value) * @return string representation of this enum constant (value) */ + @com.fasterxml.jackson.annotation.JsonValue public final String value() { return this.value; } @@ -880,16 +881,24 @@ package ; - - /** + + /** * : Attribute : multiplicity * */ - + - + + + @jakarta.persistence.Transient + + + + + + protected ; @@ -900,7 +909,7 @@ package ; protected ; - + @@ -912,26 +921,26 @@ package ; - /** + /* * . Attribute : subsetted - * + * IMPL - done with getter and setter. */ - - @org.ivoa.vodml.annotation.VoDml(id="", role=org.ivoa.vodml.annotation.VodmlRole.attribute) - - - - true - - - protected ; - - + + + + + + + + true + + @jakarta.persistence.Access(jakarta.persistence.AccessType.PROPERTY) + @@ -1335,7 +1344,7 @@ package ; - + @jakarta.xml.bind.annotation.XmlEnumValue("") ("") , diff --git a/tools/xslt/vo-dml2jsonschema.xsl b/tools/xslt/vo-dml2jsonschema.xsl index 0742b3a5..b91630b8 100644 --- a/tools/xslt/vo-dml2jsonschema.xsl +++ b/tools/xslt/vo-dml2jsonschema.xsl @@ -1,8 +1,14 @@ @@ -25,163 +32,193 @@ FIXME - this is still not a complete representation of the JSON produced - - - - - + - http://www.w3.org/2001/XMLSchema - vodml-base - - - - - - Generating JSON - considering models + Generating JSON - considering models - - Model = - - - - - - - - - - - - - - - - - - - Opening file - + { - "$schema": "https://json-schema.org/draft/2020-12/schema", - - - - - - - - - + "$schema": "https://json-schema.org/draft/2020-12/schema" + + ,"title" : "" + , + ,"type": "object" + ,"properties": { + "": { + "type": "object" + ,"properties" : + { + + + } + + } + + } + + ,"$defs" : { + "$comment" : "placeholder to make commas easier!" + + + + + - + } } - - - - + + + + + "refs" : { + "type" : "object" + ,"properties" : { + "$comment" : "placeholder to make commas easier!" + + ,"" : { + "type": "array" + ,"items" : { + + } + } + + } + + } + + + + + + , "content" : { + "type" : "array" + ,"items" : { + "anyOf" : [ + + ,{} + + ] + } + + } + + + + "description" : "" + + + "" + + + ,"$id": "" + - - - - + + + + - - - - + + + ,"additionalProperties": false + + , : { + + "allOf" : [ - - - - - - { - "type": "object", - - - } - &cr;&cr; - - - - - - - - - - - - - + {}, + { - - - - - - - - - - { - "type": "object", - - - + "type": "object" + , + ,"properties" : { + "$comment" : "placeholder to make commas easier!" + + ,"@type" : { "type": "string"} + + + + + + ,"_id" : { "type": "number"} + + } + + + } ] + + } - &cr;&cr; - + &cr;&cr; + + , : { + + "allOf" : [ - - - - - - + {}, + { + + "type": "object" + , + ,"properties" : { + "$comment" : "placeholder to make commas easier!" - - - + + } + + + } ] + + + } + &cr;&cr; + + + + + + ,"required": [ + + + + + + + + + ] - - - - - - + + , : { - "enum": {} - - + + ,"enum": [] } &cr;&cr; @@ -189,26 +226,24 @@ FIXME - this is still not a complete representation of the JSON produced - - - - - - - - + + + , : { + "type": "object" + ,"properties" : { + "value" : "string" + } + } + - - - - - - - "" : {"type" : } + , "" : { + + , + } @@ -232,189 +267,31 @@ FIXME - this is still not a complete representation of the JSON produced - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Model4vodml-ref: No model supplied for - - - - - !!!!!!! ERROR No prefix found in Model4vodml-ref for - - - - - - - - - - - - - - - - - - - - - - - - - - - - - XSDType: No model supplied for - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - !!!!!!! ERROR No prefix found in findmaping for - - - - - - - - - - - - - - - - - - - - - - - - . - - - - - + ,"" : { + "type":"array" + ,"items": { + + + "anyOf": [ + + ] + + + + + + } + } - - - - - - . - - - - + + , "" : { + "oneOf" : [ + {}, + {} + ] + , + } diff --git a/tools/xslt/vo-dml2xsd.xsl b/tools/xslt/vo-dml2xsd.xsl index a4689271..f540d35c 100644 --- a/tools/xslt/vo-dml2xsd.xsl +++ b/tools/xslt/vo-dml2xsd.xsl @@ -75,7 +75,7 @@ being able to choose a more specific sub-type. - Generating XSD - considering models + Generating XSD - considering models diff --git a/tools/xslt/vo-dml2xsdNew.xsl b/tools/xslt/vo-dml2xsdNew.xsl index eaea72db..e776d6f2 100644 --- a/tools/xslt/vo-dml2xsdNew.xsl +++ b/tools/xslt/vo-dml2xsdNew.xsl @@ -5,6 +5,8 @@ t The intention is that this wll eventually be a better documented, but equivalent version to the java generated schema note that this schema is substantially different from the era when this code was stored in volute +* references are treated differently +* there are --> @@ -18,10 +20,9 @@ note that this schema is substantially different from the era when this code was xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:vo-dml="http://www.ivoa.net/xml/VODML/v1" xmlns:vf="http://www.ivoa.net/xml/VODML/functions" - xmlns:vfs="http://www.ivoa.net/xml/VODML/schemaFunctions" xmlns:bnd="http://www.ivoa.net/xml/vodml-binding/v0.9.1" - xmlns:vodml-base="http://www.ivoa.net/xml/vo-dml/xsd/base/v0.1" - exclude-result-prefixes="bnd vf vfs" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + exclude-result-prefixes="bnd vf vo-dml" > @@ -29,13 +30,9 @@ note that this schema is substantially different from the era when this code was - - - - @@ -45,7 +42,6 @@ note that this schema is substantially different from the era when this code was - Generating XSD - considering models @@ -53,22 +49,20 @@ note that this schema is substantially different from the era when this code was - - - - + - + + Generating XSD - considering models + + - Writing to XSD file - - - Generated by gradle vodml tools - + Generated by gradle vo-dml tools + @@ -77,13 +71,39 @@ note that this schema is substantially different from the era when this code was - - - - - + + + + + + + + + + + + + 0 + unbounded + + + + + + + + + + + + + + + + + @@ -92,15 +112,14 @@ note that this schema is substantially different from the era when this code was - - import model - - + + + @@ -108,10 +127,10 @@ note that this schema is substantially different from the era when this code was - + - + @@ -149,21 +168,26 @@ note that this schema is substantially different from the era when this code was - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + &cr;&cr; @@ -178,10 +202,11 @@ note that this schema is substantially different from the era when this code was - + + @@ -204,7 +229,7 @@ note that this schema is substantially different from the era when this code was - + @@ -224,9 +249,10 @@ note that this schema is substantially different from the era when this code was - + + @@ -271,20 +297,54 @@ note that this schema is substantially different from the era when this code was - - - + + - + + + + + + + + + + + + optional + required + - - + + + + + + + + + + + + + + + + + + + + + + + + @@ -311,20 +371,21 @@ note that this schema is substantially different from the era when this code was - + + this is a reference + - - - + xsd:IDREF + @@ -344,11 +405,11 @@ note that this schema is substantially different from the era when this code was - + - + @@ -383,39 +444,8 @@ note that this schema is substantially different from the era when this code was - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + diff --git a/xsd/vo-dml-binding.xsd b/xsd/vo-dml-binding.xsd index a2d3c640..6b8a0fe7 100644 --- a/xsd/vo-dml-binding.xsd +++ b/xsd/vo-dml-binding.xsd @@ -12,50 +12,61 @@ targetNamespace="http://www.ivoa.net/xml/vodml-binding/v0.9.1" attributeFormDefa - - - the name of the model - i.e. what appears in vo-dml:model/name for the particular model - - - - the file name for the model - - - - - - - the xml namespace that is to be the target namespace for the generated model code - - - - - the xml namespace that is to be the target namespace for the generated model code - - - - - - the inheritance strategy in generating tables - follows JPA strategies - - - - - - - - - - - The size of the discrimator column - - - - - - - the list of type mappings for the model. - + + + + the name of the model - i.e. what appears in vo-dml:model/name for the particular model + + + + + + the file name for the model + + + + + + + + the xml namespace that is to be the target namespace for the generated model code + + + + + + + the xml namespace that is to be the target namespace for the generated model code + + + + + + + + + the inheritance strategy in generating tables - follows JPA strategies + + + + + + + + + + + The size of the discrimator column + + + + + + + the list of type mappings for the model. + + + @@ -96,18 +107,19 @@ targetNamespace="http://www.ivoa.net/xml/vodml-binding/v0.9.1" attributeFormDefa In general it must be used to map primitive types to various languages, but it can also be used for dataTypes. - - - - - - - Maps to XSD predefined type such as xsd:string etc. - SHould NOT have 'xsd' or 'xs' prefix, this must be supplied by generator. - - - - + + + + + + + Maps to XSD predefined type such as xsd:string etc. SHould NOT have 'xsd' or 'xs' prefix, this must be + supplied by generator. + + + + + @@ -174,7 +186,44 @@ In general it must be used to map primitive types to various languages, but it c The list of mapped models - The tooling allows for each model to have its own binding file or for multiple models to have binding customization in a single file + + + + An eclectic mixture of possible mapping customizations. + + + + + + + is an attribute in the XML serialization - the default is for it to be represented as an element if this emply type is not present. + + + + + + + + The length of the model element - arguably this really should be some sort of constraint in the model itsellf + - it has been included here as an intermediate measure whilst a constraint system is better specified. + + + + + + + + + + + + + whether the type is a JSON built-in type - +IMPL - this could be put into the code - especially as the number of JSON built-in types is small + + + + \ No newline at end of file