diff --git a/CHANGELOG.md b/CHANGELOG.md index baa8652a..1580b039 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,9 +5,16 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). -## Unreleased +## [2.4.0] - 2022-05-30 + +### Changed * Use of external [FnO](https://fno.io/) function handling component - +* Use `docker cp` for copying files between Docker containers. +* Updated Flink from verion 1.14.0 to 1.14.4 + +### Fixed +* Parameter for FnO docs (internal [issue #150](https://gitlab.ilabt.imec.be/rml/proc/rml-streamer/-/issues/150)) + ## [2.3.0] - 2022-04-26 ### Added @@ -181,3 +188,4 @@ can be set with the program argument `--baseIRI`. [2.2.1]: https://github.com/RMLio/RMLStreamer/compare/v2.2.0...v2.2.1 [2.2.2]: https://github.com/RMLio/RMLStreamer/compare/v2.2.1...v2.2.2 [2.3.0]: https://github.com/RMLio/RMLStreamer/compare/v2.2.2...v2.3.0 +[2.4.0]: https://github.com/RMLio/RMLStreamer/compare/v2.3.0...v2.4.0 diff --git a/README.md b/README.md index fb65e907..a55ad700 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ If you want to develop, read [these instructions](documentation/README_DEVELOPME RMLStreamer runs its jobs on Flink clusters. More information on how to install Flink and getting started can be found [here](https://ci.apache.org/projects/flink/flink-docs-release-1.14/try-flink/local_installation.html). At least a local cluster must be running in order to start executing RML Mappings with RMLStreamer. -Please note that this version works with Flink 1.14.0 with Scala 2.11 support, which can be downloaded [here](https://archive.apache.org/dist/flink/flink-1.14.0/flink-1.14.0-bin-scala_2.11.tgz). +Please note that this version works with Flink 1.14.4 with Scala 2.11 support, which can be downloaded [here](https://archive.apache.org/dist/flink/flink-1.14.4/flink-1.14.4-bin-scala_2.11.tgz). ### Building RMLStreamer @@ -83,6 +83,8 @@ $FLINK_BIN run toKafka --broker-list --top ``` Usage: RMLStreamer [toFile|toKafka|toTCPSocket|noOutput] [options] + -f, --function-descriptions ,... + An optional list of paths to function description files (in RDF using FnO). A path can be a file location or a URL. -j, --job-name The name to assign to the job on the Flink cluster. Put some semantics in here ;) -i, --base-iri diff --git a/configuration_example.properties b/configuration_example.properties deleted file mode 100644 index ecb63a05..00000000 --- a/configuration_example.properties +++ /dev/null @@ -1 +0,0 @@ -flinkBin=/home/pheyvaer/Applications/flink-1.3.2/bin/flink diff --git a/docker/README.md b/docker/README.md index af9916d4..4fabd22b 100644 --- a/docker/README.md +++ b/docker/README.md @@ -157,12 +157,10 @@ on `/var/lib/docker/volumes/docker_data/_data`, as shown by the `$ docker volume Copy `scenario-1` subfolder to our docker data volume, that can be read by RMLStreamer: ``` -$ [sudo] cp -r scenario-1 /var/lib/docker/volumes/docker_data/_data/ -$ [sudo] chmod -R 777 /var/lib/docker/volumes/docker_data/_data +$ docker cp scenario-1/mapping.rml.ttl docker_taskmanager_1:/mnt/data/scenario-1/mapping.rml.ttl +$ docker cp scenario-1/input.json docker_taskmanager_1:/mnt/data/scenario-1/input.json ``` -(TODO: is there a more user-friendly way to put data on docker volumes?) - ### 2. Start RMLStreamer Go back to your browser, and fill in the following `Program Arguments`: @@ -178,5 +176,11 @@ If all goes well, you will see that the job has finished, after a few seconds: ![Job done](images/scenario-1-job-done.png) -The result is written to `/var/lib/docker/volumes/docker_data/_data/scenario-1/output.nt` +The result is written to `/mnt/data/scenario-1/output.nt` inside the Docker container. and should contain the same triples as `scenario-1/output.nq`. + +You can get the generated output from the Docker container by copying it back to the host: + +``` +$ docker cp docker_taskmanager_1:/mnt/data/scenario-1/output.nt . +``` diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index e583d479..1d64d036 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -2,7 +2,7 @@ version: '3' services: jobmanager: - image: flink:1.14.0-scala_2.11-java11 + image: flink:1.14.4-scala_2.11-java11 expose: - "6123" ports: @@ -14,7 +14,7 @@ services: - data:/mnt/data taskmanager: - image: flink:1.14.0-scala_2.11-java11 + image: flink:1.14.4-scala_2.11-java11 expose: - "6121" - "6122" diff --git a/documentation/README_Functions.md b/documentation/README_Functions.md index 3b5d3177..3db0f51f 100644 --- a/documentation/README_Functions.md +++ b/documentation/README_Functions.md @@ -1,54 +1,56 @@ # README: Functions +## Built-in functions -When deploying and running jobs on Flink, make sure -- to either place the external jars (e.g. `IDLabFunctions.jar` and `GrelFunctions.jar`) in Flink's `lib` directory, -or package the RMLStreamer along with those jars by placing them in `src/main/resources`. -- to place the following files in the directory where the `flink run ...` command is issued. -These files can be obtained from `src/main/resources`: - - `functions_grel.ttl` - - `functions_grel.ttl` - - `grel_java_mapping.ttl` - - `idlab_java_mapping.ttl` ---- - -## Example: RML Streamer + Flink -Flink's `lib` directory should contain the jar-files with the custom functions. In this example, these are marked with `*` -``` -flink-1.14.0-scala_2.11 - └── lib - ├── GrelFunctions.jar * - ├── IDLabFunctions.jar * - ├── flink-dist_2.11-1.14.0.jar - ├── flink-table-blink_2.11-1.14.0.jar - └── flink-table_2.11-1.14.0.jar -``` -When running the RML Streamer on Flink, the directory should look like -``` -. -├── RMLStreamer-2.2.0.jar -├── functions_grel.ttl -├── functions_idlab.ttl -├── grel_java_mapping.ttl -├── idlab_java_mapping.ttl -├── mapping.ttl -└── input_data.csv -``` -Note that the function descriptions and function mappings are present. +Two function libraries are supported out of the box in RMLStreamer: +* GREL functions () +* IDLab functions () + +No extra configuration or parameters need to be provided to use those. + +Check out [src/test/resources/fno-testcases](../src/test/resources/fno-testcases) for example RML Mappings. + +## Using other function libraries + +RMLStreamer can also execute functions from other libraries. +To use functions, three things have to be provided: + +* A JAR file containing the code to execute functions; +* Function descriptions: a document describing the functions in the library semantically using [FnO](https://fno.io/). +* Implementation mappings: a document mapping the functions from the function descriptions + to the implementation in the JAR file, also using FnO. + +Often the function descriptions an the implementation mappings are bundled in one document. + +How to do this is explained as an example in the [Function Agent](https://github.com/FnOio/function-agent-java#example) +library, which is used by RMLStreamer to handle functions. -The command for running the RML Streamer on Flink should look like +A minimal example is provided in the tests: [src/test/resources/sandbox/function_related/external_jar/](../src/test/resources/sandbox/function_related/external_jar) + +If there are a JAR file and FnO descriptions, RMLStreamer can be invoked with the `-f` or `--function-descriptions` +parameter, like so: + +```shell +$ FLINK_BIN run [Flink options] -c io.rml.framework.Main \ + toFile \ + --output-path /tmp/helloworld.nt \ + --mapping-file //src/test/resources/sandbox/function_related/external_jar/mapping.ttl \ + --function-descriptions //src/test/resources/sandbox/function_related/external_jar/simple-test-function-fno.ttl ``` -~/flink/flink-1.14.0-scala_2.11/bin/flink run -c io.rml.framework.Main RMLStreamer-2.2.0.jar toFile --output-path $(pwd)'/out.ttl' -m mapping.ttl -``` - + +Notes +- The paths to funtion descriptions or JAR files can also be URLs; +- One can pass multiple function description locations to the `--function-descriptions` parameter separated by a space. + ## Test Cases ### FnO Tests -The official FnO testcases that are working can be found at `test/resources/fno-testcases`. +The official FnO testcases that are working can be found at [src/test/resources/fno-testcases](../src/test/resources/fno-testcases). ### SandboxTests > These tests are still experimental. -The resources can be found at `test/resources/sandbox/function_related` and are executed from `io.rml.framework.SandboxTests`. +The resources can be found at [src/test/resources/sandbox/function_related](../src/test/resources/sandbox/function_related) +and are executed from `io.rml.framework.SandboxTests`. Every test output is compared to the RMLMapper's output (e.g. `output.ttl`) for that test. A test passes when its output is equal to the RMLMapper's output. Tests marked as `pending` should be considered as not working.
@@ -79,6 +81,8 @@ The sandbox testcases are - `equal`: uses function `idlab-fn:equal` +- `external_jar`: uses the `HelloWorld` function from an external JAR file. + - `notEqual`: uses function `idlab-fn:notEqual` - `using_trueCondition_and_contains` @@ -89,101 +93,7 @@ The sandbox testcases are - `idlab-fn:trueCondition` - `idlab-fn:equal` - - - -## Tutorial: Using a function from a local JAR - -The following FnO-testcases use the function `toUpperCaseURL` -- RMLFNOTC0004-CSV -- RMLFNOTC0005-CSV -- RMLFNOTC0006-CSV - -The following steps show how to integrate the `toUpperCaseURL` function in the RML Streamer. - -### Step 1: creating the JAR -This step is based on the best-practice example [`grel-functions-java`](https://github.com/FnOio/grel-functions-java). -- Create package `io.fno.idlab` and within that package, create the class `IDLabFunctions` -- For these testcases, we need a function that returns the given URL in uppercase. -- Make sure to set the Maven compiler to a version compatible with the RMLStreamer's version. - -The following listing serves a minimalistic example that shows a possible implementation of the `toUpperCaseURL`-function. -```Java -package io.fno.idlab; - -public class IDLabFunctions { - public static String toUpperCaseURL(String s) { - if(!s.startsWith("http")) - return "http://" + s.toUpperCase(); - return s.toUpperCase(); - } -} -``` - -Use Maven to build a JAR-file, and move this JAR-file to the RMLStreamer’s `main/resources`. - -### Step 2: defining the FnO descriptions -An FnO description represents the abstract definition of a function.
-The aforementioned testcases require a function that returns a valid URL in uppercase. - Its description is shown in the following listing, and can be found in `functions_idlab.ttl`. - -```Turtle -idlab-fn:toUpperCaseURL - a fno:Function ; - fno:name "toUppercaseURL" ; - rdfs:label "toUppercaseURL" ; - dcterms:description "Returns an uppercase, valid url." ; - fno:solves grel:prob_ucase ; - fno:expects ( idlab-fn:_str ) ; - fno:returns ( idlab-fn:_stringOut ) . -``` - -### Step 3: map FnO descriptions to the corresponding implementations -In the previous step, the abstract functions were created. -The current step will define the link between abstract function descriptions and the corresponding implementation. - This is illustrated by the following listing, extracted from `idlab_java_mapping.ttl`. -```Turtle -grelm:IDLabFunctions - a fnoi:JavaClass ; - doap:download-page "IDLabFunctions.jar" ; - fnoi:class-name "io.fno.idlab.IDLabFunctions" . - - -grelm:uppercaseURLMapping - a fno:Mapping ; - fno:function idlab-fn:toUpperCaseURL; - fno:implementation grelm:IDLabFunctions ; - fno:parameterMapping [ ] ; - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "toUpperCaseURL" ] ; -. - -``` -This mapping instructs the RML Streamer to look for a method called `toUpperCaseURL` within the `io.fno.idlab.IDLabFunctions`-class of the `IDLabFunctions.jar`. Make sure the JAR-file is located in `main/resources`. - -## How the `FunctionLoader` works - -The function descriptions and mappings mentioned in the previous steps will be used by the `FunctionLoader`. - -First, a `FunctionLoader` has to be aware of the available functions. -Therefore, it can be instantiated providing file paths to the function description files. -When no such file paths are provided, the default function descriptions are used (i.e. `functions_grel.ttl`). - -Secondly, function URIs are mapped to the corresponding implementations by parsing the function mappings - (e.g. `resources/grel_java_mapping.ttl` and `resources/idlab_java_mapping.ttl`). - During this step, every function URI is mapped to a `FunctionMetaData`-object which contains the necessary metadata such as: the *download-page* of the library, the *class-name* of the function, the *method-name*, *input parameters* and *output parameters*. - - -## How the `FunctionLoader` is used - -Initially, the `FunctionLoader` is used to read and parse function descriptions and mappings. -Afterwards, when running FNOT-testcases, the `FunctionLoader`-instance is used by `io.rml.framework.engine.statement.FunctionMapGeneratorAssembler` -to load and bind every function as specified in the testcase's `mapping.ttl`. - - - # Remarks -- When the RMLStreamer is unable to find a function description or function mapping, bind method parameters to values, it will be logged as an error to the console +- When the RMLStreamer is unable to find a function description or function mapping, bind method parameters to values, it will be logged as a warning to the console and the function will not be applied. \ No newline at end of file diff --git a/documentation/README_Netty_Snapshot.md b/documentation/README_Netty_Snapshot.md deleted file mode 100644 index ac371745..00000000 --- a/documentation/README_Netty_Snapshot.md +++ /dev/null @@ -1,31 +0,0 @@ -# README: Using JitPack -(kept for reference only, not applicable anymore for RMLStreamer) - -### Using JitPack to get unpublished Maven Dependencies from a GitHub/GitLab repository - -If you want to use a Maven dependency that is not yet published on Maven Central or you want a specific version, [JitPack](https://jitpack.io/) can be used to get a specific snapshot from the GitHub/GitLab repository. Using JitPack is very simple, you simly go to the GitHub/GitLab repository and copy its link and paste on the JitPack site. Select the version and build system and get the config generated by JitPack. At the moment, this project uses JitPack for Bahir-Flink. - -### Step by step by step example - -In this example the maven dependency [Bahir-Flink](https://github.com/apache/bahir-flink) is built via JitPack. - -1. Copy the url of the repository you want to use. - - ![Step 1: copy the repo link](img/copy_repo_link.png) - -2. Paste the repository url and look up the repo on JitPack. - - ![Step 2: look up repo on JitPack](img/paste_repo_link.png) - -3. Select the build, commit, ... you need. - - ![Step 3: get desired version](img/get_build.png) - -4. Copy the `pom.xml` patches and paste them in the `pom.xml` of your project. - - ![Step 4: copy patches](img/copy_patch.png) - -When using a repository that contains different subprojets, you can also select one of those subprojets, instead of the entire project. In this example the `flink-netty-connector` subproject is selected from the Bahir-Flink repository. - -![Select a subproject](img/select_subproject.png) - diff --git a/pom.xml b/pom.xml index 393f93fd..7b0f3ac9 100644 --- a/pom.xml +++ b/pom.xml @@ -28,7 +28,7 @@ SOFTWARE. io.rml RMLStreamer - 2.3.0 + 2.4.0 jar RMLStreamer @@ -36,7 +36,7 @@ SOFTWARE. UTF-8 - 1.14.0 + 1.14.4 1.7.32 2.17.0 4.3.1 @@ -55,6 +55,14 @@ SOFTWARE. + + repo.maven.apache.org + https://repo.maven.apache.org/maven2/ + + + jitpack.io + https://jitpack.io + bintray https://jcenter.bintray.com @@ -376,6 +384,28 @@ SOFTWARE. 4.1.4 + + com.github.FnOio + function-agent-java + v0.1.0 + + + org.apache.jena + * + + + + + com.github.fnoio + grel-functions-java + v0.7.3 + + + com.github.fnoio + idlab-functions-java + v0.1.0 + + diff --git a/src/main/java/io/rml/framework/flink/util/FunctionsFlinkUtil.java b/src/main/java/io/rml/framework/flink/util/FunctionsFlinkUtil.java new file mode 100644 index 00000000..bf679b08 --- /dev/null +++ b/src/main/java/io/rml/framework/flink/util/FunctionsFlinkUtil.java @@ -0,0 +1,183 @@ +package io.rml.framework.flink.util; + +import be.ugent.idlab.knows.functions.agent.Agent; +import be.ugent.idlab.knows.functions.agent.AgentFactory; +import be.ugent.idlab.knows.functions.agent.dataType.DataTypeConverterProvider; +import be.ugent.idlab.knows.functions.agent.functionModelProvider.FunctionModelProvider; +import be.ugent.idlab.knows.functions.agent.functionModelProvider.fno.FnOFunctionModelProvider; +import be.ugent.idlab.knows.functions.agent.functionModelProvider.fno.exception.FnOException; +import be.ugent.idlab.knows.misc.FileFinder; +import org.apache.flink.api.common.cache.DistributedCache; +import org.apache.flink.api.java.ExecutionEnvironment; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.net.URL; +import java.nio.file.Files; +import java.nio.file.StandardCopyOption; +import java.util.*; + +/** + * MIT License + *

+ * Copyright (C) 2017 - 2022 RDF Mapping Language (RML) + *

+ * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + *

+ * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + *

+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + * THE SOFTWARE. + **/ +public class FunctionsFlinkUtil { + final private static Set functionDescriptionLocations = new HashSet<>(); + final private static Set implementationLocations = new HashSet<>(); + + final private static Logger logger = LoggerFactory.getLogger(FunctionsFlinkUtil.class); + + public static void putFunctionFilesInFlinkCache(ExecutionEnvironment env, StreamExecutionEnvironment senv, String... functionDescriptionLocations) throws FnOException { + //if (!FunctionsFlinkUtil.functionDescriptionLocations.isEmpty()) { + // logger.warn("Flink distributed cache already initialised. Not adding any functions!"); + // return; + //} + + // initialise a DataTypeConverterProvider + logger.debug("Initialising DataTypeConverterProvider..."); + final DataTypeConverterProvider dataTypeConverterProvider = new DataTypeConverterProvider(); + logger.debug("DataTypeConverterProvider initialised!"); + + // if no function description files are given, add the "defaut" ones: IDLab functions & GREL functions. + if (functionDescriptionLocations.length == 0) { + logger.debug("No function description locations given, so adding IDLab and GREL functions"); + FunctionsFlinkUtil.functionDescriptionLocations.add("functions_grel.ttl"); + FunctionsFlinkUtil.functionDescriptionLocations.add("grel_java_mapping.ttl"); + FunctionsFlinkUtil.functionDescriptionLocations.add("fno/functions_idlab.ttl"); + FunctionsFlinkUtil.functionDescriptionLocations.add("fno/functions_idlab_classes_java_mapping.ttl"); + } else { + logger.debug("Adding given function description locations."); + Collections.addAll(FunctionsFlinkUtil.functionDescriptionLocations, functionDescriptionLocations); + } + + // put function description files in cache + for (String functionDescriptionLocation : FunctionsFlinkUtil.functionDescriptionLocations) { + // try to locate the actual file, e.g. when it is on the class path... + String fileLocationPath = getRealFilePath(functionDescriptionLocation); + env.registerCachedFile(fileLocationPath, functionDescriptionLocation); + senv.registerCachedFile(fileLocationPath, functionDescriptionLocation); + } + + // parse all FnO documents and get the external jar locations, if any. + logger.debug("Initialising FunctionModelProvider..."); + FunctionModelProvider functionModelProvider = new FnOFunctionModelProvider(dataTypeConverterProvider, functionDescriptionLocations); + logger.debug("FunctionModelProvider initialised!"); + functionModelProvider.getFunctions().values().stream() + .map(function -> function.getFunctionMapping().getImplementation().getLocation()) + .filter(location -> !(location).isEmpty()) + .sorted() + .distinct() + .forEach(location -> { + try { + String realLocation = getRealFilePath(location); + implementationLocations.add(location); + env.registerCachedFile(realLocation, location); + senv.registerCachedFile(realLocation, location); + } catch (Throwable t) { + logger.warn("Could not locate '{}'. Certain functions might not execute.", location); + } + }); + + } + + /** + * Tries to instantiate a function Agent from function description locations and implemenation locations known to this + * singleton. If the locations are mapped in the given distributed cache, those will be used to initialise the function + * Agent. + * @param distributedCache The given Flink distributed cache. If the locations are known to the cache, these are used + * to initialise the function Agent + * @return A function agent ready to be used in a RichMapFunction. + * @throws FnOException When something goes wrong + */ + public static Agent initFunctionAgentForProcessor(final DistributedCache distributedCache) throws FnOException { + // map functionDescriptionLocations to cached locations + String[] mappedFunctionDescriptionLocations = new String[functionDescriptionLocations.size()]; + int i = 0; + for (String functionDescriptionLocation : functionDescriptionLocations) { + mappedFunctionDescriptionLocations[i++] = getMappedLocationFromCache(functionDescriptionLocation, distributedCache); + } + + // map implementation locations + Map implementationLocationsMap = new HashMap<>(); + for (String implementationLocation : implementationLocations) { + implementationLocationsMap.put(implementationLocation, getMappedLocationFromCache(implementationLocation, distributedCache)); + } + + return AgentFactory.createFromFnO(implementationLocationsMap, mappedFunctionDescriptionLocations); + } + + /** + * This method tries to find the real location of a given file path on the file system. + * If the file is not located directly on the system (e.g. in a Jar file or a web URI), then it is copied to + * a temporary file then *that* location is returned, so that Flink can opy it into its cache. + * @param path The original location path + * @return The location of the contents of the given path. This can be to a temporary file. + */ + private static String getRealFilePath(final String path) { + logger.debug("Finding real location of '{}'...", path); + URL fileURI = FileFinder.findFile(path); + if (fileURI == null) { + String msg = "Could not find location '" + path + "'."; + logger.warn(msg); + throw new RuntimeException(msg); + } + if (fileURI.getProtocol().equals("file")) { + String realPath = fileURI.getPath(); + logger.debug("Found a real file at '{}'", realPath); + return realPath; + } else { + // write contents of file to temp file and return that location + logger.debug("Path '{}' not a real file on the file system. Copying it to a temporary file.", fileURI); + try (InputStream in = fileURI.openStream()) { + File tempFile = File.createTempFile("rmlStreamer", "tmp"); + logger.debug("Created temporary file '{}' to put '{}'", tempFile.getCanonicalPath(), path); + Files.copy(in, tempFile.toPath(), StandardCopyOption.REPLACE_EXISTING); + tempFile.deleteOnExit(); + return tempFile.getCanonicalPath(); + } catch (IOException e) { + logger.warn("Could not read content of '{}'. This might lead to missing functions.", path); + throw new RuntimeException(e); + } + } + } + + /** + * Returns the location as known in the distributed cache, or the given location if not found in the distributed cache + * @param location The location to search for in the distributed cache. + * @param distributedCache a Flink distributed cache instance + * @return The mapped location or the given location if not present in cache. + */ + private static String getMappedLocationFromCache(final String location, final DistributedCache distributedCache) { + String mappedLocation; + try { + mappedLocation = distributedCache.getFile(location).getAbsolutePath(); + } catch (IllegalArgumentException e) { + logger.warn("File '{}' not found in file mapping cache. Using this location.", location); + mappedLocation = location; + } + return mappedLocation; + } +} diff --git a/src/main/resources/GrelFunctions.jar b/src/main/resources/GrelFunctions.jar deleted file mode 100644 index ef7c392b..00000000 Binary files a/src/main/resources/GrelFunctions.jar and /dev/null differ diff --git a/src/main/resources/IDLabFunctions.jar b/src/main/resources/IDLabFunctions.jar deleted file mode 100644 index 290ceb15..00000000 Binary files a/src/main/resources/IDLabFunctions.jar and /dev/null differ diff --git a/src/main/resources/functions_grel.ttl b/src/main/resources/functions_grel.ttl index 23dedb56..67ac0a15 100644 --- a/src/main/resources/functions_grel.ttl +++ b/src/main/resources/functions_grel.ttl @@ -27,7 +27,7 @@ grel:dataset dcterms:title "GREL Fno Descriptions"@en ; dcterms:description "The FnO descriptions of the GREL functions" ; dcterms:contributor ; - dcterms:modified "2020-01-20"^^xsd:date ; + dcterms:modified "2021-08-04"^^xsd:date ; cc:license ; dcterms:issued "2016-12-15"^^xsd:date ; dcterms:rights "Copyright © Ghent University – imec – IDLab"@en ; @@ -53,6 +53,18 @@ grel:prob_array_join fno:name "The array join problem"^^xsd:string ; dcterms:description "Joining an array into a string."^^xsd:string . +#GET +# https://docs.openrefine.org/manual/grelfunctions#geta-n-from-n-to-optional + +grel:array_get + a fno:Function ; + fno:name "get" ; + dcterms:description "get" ; + fno:expects ( grel:param_a grel:p_int_i_from grel:p_int_i_opt_to ) ; + fno:returns ( grel:output_array ) . + +#JOIN + grel:array_join a fno:Function ; fno:name "join" ; @@ -84,6 +96,61 @@ grel:stringOut fno:predicate grel:stringOutput ; fno:type xsd:string . +#LENGTH + +grel:array_length + a fno:Function ; + fno:name "length" ; + dcterms:description "length" ; + fno:expects ( grel:param_a ) ; + fno:returns ( grel:output_number ) . + +#SLICE +# https://docs.openrefine.org/manual/grelfunctions#slicea-n-from-n-to-optional + +grel:array_slice + a fno:Function ; + fno:name "slice" ; + dcterms:description "slice" ; + fno:expects ( grel:param_a grel:p_int_i_from grel:p_int_i_opt_to ) ; + fno:returns ( grel:output_number ) . + +#REVERSE + +grel:array_reverse + a fno:Function ; + fno:name "reverse" ; + dcterms:description "reverse" ; + fno:expects ( grel:param_a ) ; + fno:returns ( grel:output_array ) . + +#SORT + +grel:array_sort + a fno:Function ; + fno:name "sort" ; + dcterms:description "sort" ; + fno:expects ( grel:param_a ) ; + fno:returns ( grel:output_array ) . + +#SUM + +grel:array_sum + a fno:Function ; + fno:name "sum" ; + dcterms:description "sum" ; + fno:expects ( grel:param_a ) ; + fno:returns ( grel:output_number ) . + +#UNIQUES + +grel:array_uniques + a fno:Function ; + fno:name "uniques" ; + dcterms:description "uniques" ; + fno:expects ( grel:param_a ) ; + fno:returns ( grel:output_array ) . + grel:prob_bool a fno:Problem ; fno:name "The bool problem"^^xsd:string ; @@ -195,16 +262,122 @@ grel:param_false rdfs:label "eFalse" ; fno:predicate grel:any_false ; fno:type xsd:any ; - fno:required "true"^^xsd:boolean . - + fno:required "false"^^xsd:boolean . grel:output_any - a fno:Parameter ; - #fno:name "" ; - #rdfs:label "" ; - fno:predicate grel:any_output ; - fno:type rdf:object; - fno:required "true"^^xsd:boolean . + a fno:Output ; + fno:name "any" ; + rdfs:label "any" ; + fno:predicate grel:o_any ; + fno:type xsd:any . + +#grel:controls_forEachIndex +# a fno:Function ; +# fno:name "forEachIndex" ; +# dcterms:description "forEachIndex" ; +# fno:expects ( grel:param_a grel:uri_index grel:uri_value grel:param_any_e ) ; +# fno:returns ( grel:output_array ) . + +#grel:uri_value +# a fno:Parameter ; +# fno:name "v" ; +# rdfs:label "v" ; +# fno:predicate grel:param_value ; +# fno:type xsd:string ; +# fno:required "true"^^xsd:boolean . + +#grel:uri_index +# a fno:Parameter ; +# fno:name "i" ; +# rdfs:label "i" ; +# fno:predicate grel:p_int_index ; +# fno:type xsd:string ; +# fno:required "true"^^xsd:boolean . + +#grel:controls_forNonBlank +# a fno:Function ; +# fno:name "forNonBlank" ; +# dcterms:description "forNonBlank" ; +# fno:expects ( grel:param_any_e grel:uri_value grel:param_false grel:param_true ) ; +# fno:returns ( grel:output_any ) . + +#grel:controls_forRange +# a fno:Function ; +# fno:name "forRange" ; +# dcterms:description "forRange" ; +# fno:expects ( grel:p_int_i_from grel:inti_i_to grel:param_int_i_step grel:uri_value grel:param_any_e ) ; +# fno:returns ( grel:output_array ) . + +#grel:param_int_i_step +# a fno:Parameter ; +# fno:name "step" ; +# rdfs:label "step" ; +# fno:predicate grel:string_step ; +# fno:type xsd:int ; +# fno:required "true"^^xsd:boolean . + +#grel:controls_filter +# a fno:Function ; +# fno:name "filter" ; +# dcterms:description "filter" ; +# fno:expects ( grel:param_a grel:uri_value grel:param_any_e ) ; +# fno:returns ( grel:output_array ) . + +#grel:controls_forEach +# a fno:Function ; +# fno:name "forEach" ; +# dcterms:description "forEach" ; +# fno:expects ( grel:param_a grel:uri_value grel:param_any_e ) ; +# fno:returns ( grel:output_array ) . + +#grel:controls_with +# a fno:Function ; +# fno:name "with" ; +# dcterms:description "with" ; +# fno:expects ( grel:param_any_e grel:uri_value grel:param_any_e ) ; +# fno:returns ( grel:output_any ) . + +#grel:controls_isBlank +# a fno:Function ; +# fno:name "isBlank" ; +# dcterms:description "isBlank" ; +# fno:expects ( grel:param_any_e ) ; +# fno:returns ( grel:output_bool ) . + +#grel:controls_isError +# a fno:Function ; +# fno:name "isError" ; +# dcterms:description "isError" ; +# fno:expects ( grel:param_any_e ) ; +# fno:returns ( grel:output_bool ) . + +#grel:controls_isNonBlank +# a fno:Function ; +# fno:name "isNonBlank" ; +# dcterms:description "isNonBlank" ; +# fno:expects ( grel:param_any_e ) ; +# fno:returns ( grel:output_bool ) . + +#grel:controls_isNotNull +# a fno:Function ; +# fno:name "isNotNull" ; +# dcterms:description "isNotNull" ; +# fno:expects ( grel:param_any_e ) ; +# fno:returns ( grel:output_bool ) . + +#grel:controls_isNull +# a fno:Function ; +# fno:name "isNull" ; +# dcterms:description "isNull" ; +# fno:expects ( grel:param_any_e ) ; +# fno:returns ( grel:output_bool ) . + +#grel:controls_isNumeric +# a fno:Function ; +# fno:name "isNumeric" ; +# dcterms:description "isNumeric" ; +# fno:expects ( grel:param_any_e ) ; +# fno:returns ( grel:output_bool ) . grel:prob_string a fno:Problem ; @@ -239,9 +412,16 @@ grel:output_number fno:name "number" ; rdfs:label "number" ; fno:predicate grel:o_number ; - fno:type xsd:number . + fno:type xsd:integer . + +#TOSTRING -#TODO toStringValue +grel:string_toString + a fno:Function ; + fno:name "toString" ; + dcterms:description "toString" ; + fno:expects ( grel:param_any_e ) ; + fno:returns ( grel:stringOut ) . #STARTSWITH @@ -296,12 +476,16 @@ grel:string_contains fno:expects ( grel:valueParam grel:param_string_sub ) ; fno:returns ( grel:output_bool ) . -#TODO indexOf +grel:prob_indexOf + a fno:Problem ; + fno:name "The indexOf problem"^^xsd:string ; + dcterms:description "The index of the first occurence within a string."^^xsd:string . grel:string_indexOf a fno:Function ; fno:name "indexOf" ; dcterms:description "indexOf" ; + fno:solves grel:prob_indexOf ; fno:expects ( grel:valueParam grel:param_string_sub ) ; fno:returns ( grel:output_number ) . @@ -344,7 +528,7 @@ grel:toUpperCase fno:expects ( grel:valueParam ) ; fno:returns ( grel:stringOut ) . -#TODO toTitleCase +#TOTITLECASE grel:string_toTitlecase a fno:Function ; @@ -366,14 +550,14 @@ grel:string_trim #TODO strip sameAs trim -#grel:valueParamtrip +#grel:string_strip # a fno:Function ; # fno:name "strip" ; # dcterms:description "strip" ; # fno:expects ( grel:valueParam ) ; # fno:returns ( grel:stringOut ) . -#TODO chomp +#CHOMP grel:string_chomp a fno:Function ; @@ -410,12 +594,12 @@ grel:p_int_i_opt_to fno:name "to" ; rdfs:label "to" ; fno:predicate grel:param_int_i_opt_to ; - fno:type xsd:string ; + fno:type xsd:int ; fno:required "false"^^xsd:boolean . # TODO slice sameAs substring -#grel:valueParamlice +#grel:string_slice # a fno:Function ; # fno:name "slice" ; # dcterms:description "slice" ; @@ -431,16 +615,22 @@ grel:p_int_i_opt_to # fno:expects ( grel:valueParam grel:p_int_i_from grel:p_int_i_opt_to ) ; # fno:returns ( grel:stringOut ) . -# TODO lastIndexOf +#LASTINDEXOF -#grel:string_lastIndexOf -# a fno:Function ; -# fno:name "lastIndexOf" ; -# dcterms:description "lastIndexOf" ; -# fno:expects ( grel:valueParam grel:param_string_sub ) ; -# fno:returns ( grel:output_number ) . +grel:prob_lastIndexOf + a fno:Problem ; + fno:name "The lastIndexOf problem"^^xsd:string ; + dcterms:description "Retrieving the last index of an occurrence in a string."^^xsd:string . + +grel:string_lastIndexOf + a fno:Function ; + fno:name "lastIndexOf" ; + dcterms:description "lastIndexOf" ; + fno:solves grel:prob_lastIndexOf ; + fno:expects ( grel:valueParam grel:param_string_sub ) ; + fno:returns ( grel:output_number ) . -# TODO replace +#REPLACE grel:prob_replace a fno:Problem ; @@ -448,13 +638,13 @@ grel:prob_replace dcterms:description "Replacing each non-overlapping occurrence of a regular expression pattern with a replacement string. Regular expession matching may involve modifier flags."^^xsd:string . -#grel:string_replace -# a fno:Function ; -# fno:name "replace" ; -# dcterms:description "replace" ; -# fno:solves grel:prob_replace ; -# fno:expects ( grel:valueParam grel:param_find grel:param_replace ) ; -# fno:returns ( grel:stringOut ) . +grel:string_replace + a fno:Function ; + fno:name "replace" ; + dcterms:description "replace" ; + fno:solves grel:prob_replace ; + fno:expects ( grel:valueParam grel:param_find grel:param_replace ) ; + fno:returns ( grel:stringOut ) . grel:param_find a fno:Parameter ; @@ -472,16 +662,16 @@ grel:param_replace fno:type xsd:string ; fno:required "true"^^xsd:boolean . -#TODO replaceChars +#REPLACECHARS -#grel:string_replaceChars -# a fno:Function ; -# fno:name "replaceChars" ; -# dcterms:description "replaceChars" ; -# fno:expects ( grel:valueParam grel:param_find grel:param_replace ) ; -# fno:returns ( grel:stringOut ) . +grel:string_replaceChars + a fno:Function ; + fno:name "replaceChars" ; + dcterms:description "replaceChars" ; + fno:expects ( grel:valueParam grel:param_find grel:param_replace ) ; + fno:returns ( grel:stringOut ) . -#TODO match +#MATCH grel:prob_regex a fno:Problem ; @@ -489,12 +679,12 @@ grel:prob_regex dcterms:description "Returning true if a string (?arg1) matches the regular expression supplied as a pattern (?arg2) as influenced by the value of flags (?arg3), otherwise returns false."^^xsd:string . -#grel:string_match -# a fno:Function ; -# fno:name "match" ; -# dcterms:description "match" ; -# fno:expects ( grel:valueParam grel:param_regex ) ; -# fno:returns ( grel:output_array ) . +grel:string_match + a fno:Function ; + fno:name "match" ; + dcterms:description "match" ; + fno:expects ( grel:valueParam grel:param_regex ) ; + fno:returns ( grel:output_array ) . grel:output_array a fno:Output ; @@ -503,7 +693,6 @@ grel:output_array fno:predicate grel:o_array ; fno:type rdf:List . - grel:param_regex a fno:Parameter ; fno:name "p" ; @@ -521,14 +710,14 @@ grel:param_regex # fno:expects ( grel:valueParam grel:param_regex ) ; # fno:returns ( grel:output_array ) . -#TODO toNumber +#TONUMBER -#grel:string_toNumber -# a fno:Function ; -# fno:name "toNumber" ; -# dcterms:description "toNumber" ; -# fno:expects ( grel:param_any_e ) ; -# fno:returns ( grel:output_number ) . +grel:string_toNumber + a fno:Function ; + fno:name "toNumber" ; + dcterms:description "toNumber" ; + fno:expects ( grel:param_any_e ) ; + fno:returns ( grel:output_number ) . grel:param_any_e a fno:Parameter ; @@ -547,14 +736,15 @@ grel:string_split fno:expects ( grel:valueParam grel:param_string_sep ) ; fno:returns ( grel:output_array ) . -#TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#splitbylengthsstring-s-number-n1-number-n2- +#SPLITBYLENGTHS +# https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#splitbylengthsstring-s-number-n1-number-n2- -#grel:string_splitByLengths -# a fno:Function ; -# fno:name "splitByLengths" ; -# dcterms:description "splitByLengths" ; -# fno:expects ( grel:valueParam grel:param_int_i grel:param_int_i2 grel:param_int_rep_i ) ; -# fno:returns ( grel:output_array ) . +grel:string_splitByLengths + a fno:Function ; + fno:name "splitByLengths" ; + dcterms:description "splitByLengths" ; + fno:expects ( grel:valueParam grel:param_int_i grel:param_int_i2 grel:param_int_rep_i ) ; + fno:returns ( grel:output_array ) . grel:param_int_i a fno:Parameter ; @@ -580,14 +770,15 @@ grel:param_int_rep_i fno:type xsd:string ; fno:required "true"^^xsd:boolean . -#TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#smartsplitstring-s-optional-string-sep +#SMARTSPLIT +# https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#smartsplitstring-s-optional-string-sep -#grel:valueParammartSplit -# a fno:Function ; -# fno:name "smartSplit" ; -# dcterms:description "smartSplit" ; -# fno:expects ( grel:valueParam grel:param_opt_sep ) ; -# fno:returns ( grel:output_array ) . +grel:string_smartSplit + a fno:Function ; + fno:name "smartSplit" ; + dcterms:description "smartSplit" ; + fno:expects ( grel:valueParam grel:param_opt_sep ) ; + fno:returns ( grel:output_array ) . grel:param_opt_sep a fno:Parameter ; @@ -597,23 +788,25 @@ grel:param_opt_sep fno:type xsd:string ; fno:required "true"^^xsd:boolean . -#TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#splitbychartypes +#SPLITBYCHARTYPE +# https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#splitbychartypes -#grel:valueParamplitByCharType -# a fno:Function ; -# fno:name "splitByCharType" ; -# dcterms:description "splitByCharType" ; -# fno:expects ( grel:valueParam ) ; -# fno:returns ( grel:output_array ) . +grel:string_splitByCharType + a fno:Function ; + fno:name "splitByCharType" ; + dcterms:description "splitByCharType" ; + fno:expects ( grel:valueParam ) ; + fno:returns ( grel:output_array ) . -#TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#partitionstring-s-string-or-regex-frag-optional-boolean-omitfragment +#PARTITION +# https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#partitionstring-s-string-or-regex-frag-optional-boolean-omitfragment -#grel:string_partition -# a fno:Function ; -# fno:name "partition" ; -# dcterms:description "partition" ; -# fno:expects ( grel:valueParam grel:param_fragment grel:param_bool_opt_b ) ; -# fno:returns ( grel:output_array ) . +grel:string_partition + a fno:Function ; + fno:name "partition" ; + dcterms:description "partition" ; + fno:expects ( grel:valueParam grel:param_fragment grel:param_bool_opt_b ) ; + fno:returns ( grel:output_array ) . grel:param_fragment a fno:Parameter ; @@ -631,23 +824,25 @@ grel:param_bool_opt_b fno:type xsd:boolean ; fno:required "false"^^xsd:boolean . -#TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#rpartitionstring-s-string-or-regex-frag-optional-boolean-omitfragment +#RPARTITION +# https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#rpartitionstring-s-string-or-regex-frag-optional-boolean-omitfragment -#grel:string_rpartition -# a fno:Function ; -# fno:name "rpartition" ; -# dcterms:description "rpartition" ; -# fno:expects ( grel:valueParam grel:param_fragment grel:param_bool_opt_b ) ; -# fno:returns ( grel:output_array ) . +grel:string_rpartition + a fno:Function ; + fno:name "rpartition" ; + dcterms:description "rpartition" ; + fno:expects ( grel:valueParam grel:param_fragment grel:param_bool_opt_b ) ; + fno:returns ( grel:output_array ) . -#TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#diffo1-o2-optional-string-timeunit +#DIFF +# https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#diffo1-o2-optional-string-timeunit -#grel:string_diff -# a fno:Function ; -# fno:name "diff" ; -# dcterms:description "diff" ; -# fno:expects ( grel:valueParam grel:valueParam2 ) ; -# fno:returns ( grel:stringOut ) . +grel:string_diff + a fno:Function ; + fno:name "diff" ; + dcterms:description "diff" ; + fno:expects ( grel:valueParam grel:valueParam2 ) ; + fno:returns ( grel:stringOut ) . grel:valueParam2 a fno:Parameter ; @@ -681,16 +876,15 @@ grel:modeParam fno:predicate grel:modeParameter ; fno:required "true"^^xsd:boolean . -#TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#unescapestring-s-string-mode - -#grel:string_unescape -# a fno:Function ; -# fno:name "unescape" ; -# dcterms:description "unescape" ; -# fno:expects ( grel:valueParam grel:modeParam ) ; -# fno:returns ( grel:stringOut ) . +#UNESCAPE +# https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#unescapestring-s-string-mode -#TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#md5string-s +grel:string_unescape + a fno:Function ; + fno:name "unescape" ; + dcterms:description "unescape" ; + fno:expects ( grel:valueParam grel:modeParam ) ; + fno:returns ( grel:stringOut ) . grel:prob_crypto a fno:Problem ; @@ -702,60 +896,60 @@ grel:prob_md5 fno:name "The md5 problem"^^xsd:string ; dcterms:description "Calculating the md5 checksum."^^xsd:string . -#grel:string_md5 -# a fno:Function ; -# fno:name "md5" ; -# dcterms:description "md5" ; -# fno:solves grel:prob_md5 ; -# fno:expects ( grel:valueParam ) ; -# fno:returns ( grel:stringOut ) . - -#TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#sha1string-s +grel:string_md5 + a fno:Function ; + fno:name "md5" ; + dcterms:description "md5" ; + fno:solves grel:prob_md5 ; + fno:expects ( grel:valueParam ) ; + fno:returns ( grel:stringOut ) . grel:prob_sha1 a fno:Problem ; fno:name "The sha1 problem"^^xsd:string ; dcterms:description "Calculating the SHA1 checksum."^^xsd:string . -#grel:string_sha1 -# a fno:Function ; -# fno:name "sha1" ; -# dcterms:description "sha1" ; -# fno:solves grel:prob_sha1 ; -# fno:expects ( grel:valueParam ) ; -# fno:returns ( grel:stringOut ) . +grel:string_sha1 + a fno:Function ; + fno:name "sha1" ; + dcterms:description "sha1" ; + fno:solves grel:prob_sha1 ; + fno:expects ( grel:valueParam ) ; + fno:returns ( grel:stringOut ) . -#TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#phoneticstring-s-string-encoding +#PHONETIC +# https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#phoneticstring-s-string-encoding -#grel:string_phonetic -# a fno:Function ; -# fno:name "phonetic" ; -# dcterms:description "phonetic" ; -# fno:expects ( grel:valueParam grel:valueParamtring_encoding ) ; -# fno:returns ( grel:stringOut ) . +grel:string_phonetic + a fno:Function ; + fno:name "phonetic" ; + dcterms:description "phonetic" ; + fno:expects ( grel:valueParam grel:param_string_encoding ) ; + fno:returns ( grel:stringOut ) . -grel:valueParamtring_encoding +grel:param_string_encoding a fno:Parameter ; fno:name "encoding" ; rdfs:label "encoding" ; - fno:predicate grel:p_encoding ; + fno:predicate grel:p_string_encoding ; fno:type xsd:string ; fno:required "true"^^xsd:boolean . -#TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#reinterpretstring-s-string-encoder +#REINTERPRET +# https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#reinterpretstring-s-string-encoder -#grel:string_reinterpret -# a fno:Function ; -# fno:name "reinterpret" ; -# dcterms:description "reinterpret" ; -# fno:expects ( grel:valueParam grel:valueParamtring_encoder ) ; -# fno:returns ( grel:stringOut ) . +grel:string_reinterpret + a fno:Function ; + fno:name "reinterpret" ; + dcterms:description "reinterpret" ; + fno:expects ( grel:valueParam grel:param_string_encoder ) ; + fno:returns ( grel:stringOut ) . -grel:valueParamtring_encoder +grel:param_string_encoder a fno:Parameter ; fno:name "encoder" ; rdfs:label "encoder" ; - fno:predicate grel:p_encoder ; + fno:predicate grel:p_string_encoder ; fno:type xsd:string ; fno:required "true"^^xsd:boolean . @@ -786,14 +980,15 @@ grel:valueParamtring_encoder # fno:expects ( grel:valueParam grel:param_int_i ) ; # fno:returns ( grel:output_array ) . -#TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#unicodestring-s +#UNICODE +# https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#unicodestring-s -#grel:string_unicode -# a fno:Function ; -# fno:name "unicode" ; -# dcterms:description "unicode" ; -# fno:expects ( grel:valueParam ) ; -# fno:returns ( grel:output_array ) . +grel:string_unicode + a fno:Function ; + fno:name "unicode" ; + dcterms:description "unicode" ; + fno:expects ( grel:valueParam ) ; + fno:returns ( grel:output_array ) . #TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#unicodetypestring-s @@ -815,20 +1010,20 @@ grel:valueParamtring_encoder #TODO https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions#mqlkeyunquotestring-key -grel:string_mqlKeyUnquote - a fno:Function ; - fno:name "mqlKeyUnquote" ; - dcterms:description "mqlKeyUnquote" ; - fno:expects ( grel:valueParamtring_key ) ; - fno:returns ( grel:stringOut ) . - -grel:valueParamtring_key - a fno:Parameter ; - fno:name "key" ; - rdfs:label "key" ; - fno:predicate grel:p_key ; - fno:type xsd:string ; - fno:required "true"^^xsd:boolean . +# grel:string_mqlKeyUnquote +# a fno:Function ; +# fno:name "mqlKeyUnquote" ; +# dcterms:description "mqlKeyUnquote" ; +# fno:expects ( grel:valueParamtring_key ) ; +# fno:returns ( grel:stringOut ) . + +# grel:valueParamtring_key +# a fno:Parameter ; +# fno:name "key" ; +# rdfs:label "key" ; +# fno:predicate grel:p_key ; +# fno:type xsd:string ; +# fno:required "true"^^xsd:boolean . grel:prob_date a fno:Problem ; @@ -870,42 +1065,135 @@ grel:prob_year fno:name "The year problem"^^xsd:string ; dcterms:description "Extracting the year from a date/time literal."^^xsd:string . -#grel:date_datePart -# a fno:Function ; -# fno:name "datePart" ; -# dcterms:description "datePart" ; -# fno:expects ( grel:date_d grel:string_unit ) ; -# fno:returns ( grel:output_any ) . - -#grel:date_d -# a fno:Parameter ; -# fno:name "d" ; -# rdfs:label "d" ; -# fno:predicate grel:param_d ; -# fno:type xsd:date ; -# fno:required "true"^^xsd:boolean . - -#grel:string_unit -# a fno:Parameter ; -# fno:name "unit" ; -# rdfs:label "unit" ; -# fno:predicate grel:param_unit ; -# fno:type xsd:string ; -# fno:required "true"^^xsd:boolean . - grel:prob_now a fno:Problem ; fno:name "The now problem"^^xsd:string ; dcterms:description "Getting the current date and time."^^xsd:string . -#grel:date_now +#NOW + +grel:date_now + a fno:Function ; + fno:name "now" ; + dcterms:description "now" ; + fno:expects ( ) ; + fno:returns ( grel:output_datetime ) . + +grel:output_datetime + a fno:Output ; + fno:name "datetime" ; + rdfs:label "datetime" ; + fno:predicate grel:o_datetime ; + fno:type xsd:dateTime . + +#TODATE + +grel:date_toDate + a fno:Function ; + fno:name "toDate" ; + dcterms:description "toDate" ; + fno:expects ( grel:valueParam grel:param_string_pattern ) ; + fno:returns ( grel:output_date ) . + +grel:param_string_pattern + a fno:Parameter ; + fno:name "pattern" ; + rdfs:label "pattern" ; + fno:predicate grel:p_string_pattern ; + fno:type xsd:string ; + fno:required "true"^^xsd:boolean . + +grel:output_date + a fno:Output ; + fno:name "date" ; + rdfs:label "date" ; + fno:predicate grel:o_date ; + fno:type xsd:date . + +#TOSTRING +#TODO grel:date_toString sameAs grel:date_toDate + +# grel:date_toString # a fno:Function ; -# fno:name "now" ; -# dcterms:description "now" ; -# fno:expects ( ) ; -# fno:returns ( grel:output_datetime ) . +# fno:name "toString" ; +# dcterms:description "toString" ; +# fno:expects ( grel:valueParam grel:param_string_pattern ) ; +# fno:returns ( grel:output_date ) . -#TODO output_datetime +#DIFF + +grel:date_diff + a fno:Function ; + fno:name "diff" ; + dcterms:description "diff" ; + fno:expects ( grel:param_datetime_d grel:param_datetime_d2 grel:param_string_timeunit ) ; + fno:returns ( grel:stringOut ) . + +grel:param_datetime_d + a fno:Parameter ; + fno:name "d" ; + rdfs:label "d" ; + fno:predicate grel:p_datetime_d ; + fno:type xsd:dateTime ; + fno:required "true"^^xsd:boolean . + +grel:param_datetime_d2 + a fno:Parameter ; + fno:name "d2" ; + rdfs:label "d2" ; + fno:predicate grel:p_datetime_d2 ; + fno:type xsd:dateTime ; + fno:required "true"^^xsd:boolean . + +grel:param_string_timeunit + a fno:Parameter ; + fno:name "timeUnit" ; + rdfs:label "timeUnit" ; + fno:predicate grel:p_string_timeunit ; + fno:type xsd:string ; + fno:required "true"^^xsd:boolean . + +#INC + +grel:date_inc + a fno:Function ; + fno:name "inc" ; + dcterms:description "inc" ; + fno:expects ( grel:param_date_d grel:param_dec_n grel:param_string_unit ) ; + fno:returns ( grel:output_date ) . + +grel:param_date_d + a fno:Parameter ; + fno:name "d" ; + rdfs:label "d" ; + fno:predicate grel:p_date_d ; + fno:type xsd:date ; + fno:required "true"^^xsd:boolean . + +grel:param_dec_n + a fno:Parameter ; + fno:name "d" ; + rdfs:label "d" ; + fno:predicate grel:p_dec_n ; + fno:type xsd:decimal ; + fno:required "true"^^xsd:boolean . + +grel:param_string_unit + a fno:Parameter ; + fno:name "unit" ; + rdfs:label "unit" ; + fno:predicate grel:p_string_unit ; + fno:type xsd:string ; + fno:required "true"^^xsd:boolean . + +#DATEPART + +grel:date_datePart + a fno:Function ; + fno:name "datePart" ; + dcterms:description "datePart" ; + fno:expects ( grel:param_date_d grel:param_string_unit ) ; + fno:returns ( grel:output_any ) . grel:prob_mathematical a fno:Problem ; @@ -918,58 +1206,34 @@ grel:prob_floor dcterms:description "Returning the largest (closest to positive infinity) number with no fractional part that is not greater than the given value."^^xsd:string . -#grel:math_floor -# a fno:Function ; -# fno:name "floor" ; -# dcterms:description "floor" ; -# fno:solves grel:prob_floor ; -# fno:expects ( grel:dec_n ) ; -# fno:returns ( grel:output_number ) . +#FLOOR -#grel:dec_n -# a fno:Parameter ; -# fno:name "d" ; -# rdfs:label "d" ; -# fno:predicate grel:param_n ; -# fno:type xsd:decimal ; -# fno:required "true"^^xsd:boolean . +grel:math_floor + a fno:Function ; + fno:name "floor" ; + dcterms:description "floor" ; + fno:solves grel:prob_floor ; + fno:expects ( grel:param_dec_n ) ; + fno:returns ( grel:output_number ) . -# +# # skos:broader grel:math_floor . -grel:prob_max - a fno:Problem ; - fno:name "The max problem"^^xsd:string ; - dcterms:description "Returning the largest value of the input."^^xsd:string . - -#grel:math_max -# a fno:Function ; -# fno:name "max" ; -# dcterms:description "max" ; -# fno:solves grel:prob_max ; -# fno:expects ( grel:dec_n grel:dec_n2 ) ; -# fno:returns ( grel:output_number ) . - -#grel:dec_n2 -# a fno:Parameter ; -# fno:name "d2" ; -# rdfs:label "d2" ; -# fno:predicate grel:param_n2 ; -# fno:type xsd:decimal ; -# fno:required "true"^^xsd:boolean . +grel:prob_ceil + a fno:Problem ; + fno:name "The ceil problem"^^xsd:string ; + dcterms:description + "Returning the smallest (closest to negative infinity) number with no fractional part that is not less than the value of arg."^^xsd:string . -grel:prob_min - a fno:Problem ; - fno:name "The min problem"^^xsd:string ; - dcterms:description "Returning the smallest value of the input."^^xsd:string . +#CEIL -#grel:math_min -# a fno:Function ; -# fno:name "min" ; -# dcterms:description "min" ; -# fno:solves grel:prob_min ; -# fno:expects ( grel:dec_n grel:dec_n2 ) ; -# fno:returns ( grel:output_number ) . +grel:math_ceil + a fno:Function ; + fno:name "ceil" ; + dcterms:description "ceil" ; + fno:solves grel:prob_ceil ; + fno:expects ( grel:param_dec_n ) ; + fno:returns ( grel:output_number ) . grel:prob_round a fno:Problem ; @@ -977,192 +1241,165 @@ grel:prob_round dcterms:description "Returning the number with no fractional part that is closest to the argument. If there are two such numbers, then the one that is closest to positive infinity is returned."^^xsd:string . -#grel:math_round -# a fno:Function ; -# fno:name "round" ; -# dcterms:description "round" ; -# fno:solves grel:prob_round ; -# fno:expects ( grel:dec_n ) ; -# fno:returns ( grel:output_number ) . +#ROUND + +grel:math_round + a fno:Function ; + fno:name "round" ; + dcterms:description "round" ; + fno:solves grel:prob_round ; + fno:expects ( grel:param_dec_n ) ; + fno:returns ( grel:output_number ) . # # skos:broader grel:math_round . -grel:prob_ceil - a fno:Problem ; - fno:name "The ceil problem"^^xsd:string ; - dcterms:description - "Returning the smallest (closest to negative infinity) number with no fractional part that is not less than the value of arg."^^xsd:string . - -#grel:math_ceil -# a fno:Function ; -# fno:name "ceil" ; -# dcterms:description "ceil" ; -# fno:solves grel:prob_ceil ; -# fno:expects ( grel:dec_n ) ; -# fno:returns ( grel:output_number ) . - -grel:prob_coalesce +grel:prob_min a fno:Problem ; - fno:name "The coalesce problem"^^xsd:string ; - dcterms:description "Returning the first bound argument."^^xsd:string . - -#grel:other_coalesce -# a fno:Function ; -# fno:name "coalesce" ; -# dcterms:description "coalesce" ; -# fno:solves grel:prob_coalesce ; -# fno:expects ( grel:any_rep_e ) ; -# fno:returns ( grel:output_any ) . - -# HERE - -#grel:param_b2 -# a fno:Parameter ; -# fno:name "b2" ; -# rdfs:label "b2" ; -# fno:predicate grel:bool_b2 ; -# fno:type xsd:boolean ; -# fno:required "true"^^xsd:boolean . - -#grel:bool_rep_b -# a fno:Parameter ; -# fno:name "b3" ; -# rdfs:label "b3" ; -# fno:predicate grel:param_rep_b ; -# fno:type xsd:string ; -# fno:required "true"^^xsd:boolean . - -#grel:date_d2 -# a fno:Parameter ; -# fno:name "d2" ; -# rdfs:label "d2" ; -# fno:predicate grel:param_d2 ; -# fno:type xsd:date ; -# fno:required "true"^^xsd:boolean . - -#grel:dec_n_exp -# a fno:Parameter ; -# fno:name "e" ; -# rdfs:label "e" ; -# fno:predicate grel:p_any_exp ; -# fno:type xsd:decimal ; -# fno:required "true"^^xsd:boolean . - -#grel:string_field -# a fno:Parameter ; -# fno:name "field" ; -# rdfs:label "field" ; -# fno:predicate grel:param_field ; -# fno:type xsd:string ; -# fno:required "true"^^xsd:boolean . - -#grel:string_format -# a fno:Parameter ; -# fno:name "format" ; -# rdfs:label "format" ; -# fno:predicate grel:param_format ; -# fno:type xsd:string ; -# fno:required "true"^^xsd:boolean . - -#grel:string_rep_format -# a fno:Parameter ; -# fno:name "format1" ; -# rdfs:label "format1" ; -# fno:predicate grel:param_rep_format ; -# fno:type xsd:string ; -# fno:required "true"^^xsd:boolean . + fno:name "The min problem"^^xsd:string ; + dcterms:description "Returning the smallest value of the input."^^xsd:string . -#grel:uri_index -# a fno:Parameter ; -# fno:name "i" ; -# rdfs:label "i" ; -# fno:predicate grel:p_int_index ; -# fno:type xsd:string ; -# fno:required "true"^^xsd:boolean . +#MIN -#grel:any_rep_e -# a fno:Parameter ; -# fno:name "o1" ; -# rdfs:label "o1" ; -# fno:predicate grel:param_rep_e ; -# fno:type xsd:any ; -# fno:required "true"^^xsd:boolean . +grel:math_min + a fno:Function ; + fno:name "min" ; + dcterms:description "min" ; + fno:solves grel:prob_min ; + fno:expects ( grel:param_dec_n grel:param_dec_n2 ) ; + fno:returns ( grel:output_number ) . -#grel:bool_opt_omit -# a fno:Parameter ; -# fno:name "omitFragment" ; -# rdfs:label "omitFragment" ; -# fno:predicate grel:param_omit ; -# fno:type xsd:string ; -# fno:required "false"^^xsd:boolean . +grel:param_dec_n2 + a fno:Parameter ; + fno:name "d2" ; + rdfs:label "d2" ; + fno:predicate grel:param_n2 ; + fno:type xsd:decimal ; + fno:required "true"^^xsd:boolean . -#grel:param_int_i_step -# a fno:Parameter ; -# fno:name "step" ; -# rdfs:label "step" ; -# fno:predicate grel:string_step ; -# fno:type xsd:int ; -# fno:required "true"^^xsd:boolean . +grel:prob_max + a fno:Problem ; + fno:name "The max problem"^^xsd:string ; + dcterms:description "Returning the largest value of the input."^^xsd:string . -#grel:string_time_unit -# a fno:Parameter ; -# fno:name "timeUnit" ; -# rdfs:label "timeUnit" ; -# fno:predicate grel:param_time_unit ; -# fno:type xsd:string ; -# fno:required "true"^^xsd:boolean . +#MAX -#grel:param_int_i_to -# a fno:Parameter ; -# fno:name "to" ; -# rdfs:label "to" ; -# fno:predicate grel:param_to ; -# fno:type xsd:int ; -# fno:required "true"^^xsd:boolean . +grel:math_max + a fno:Function ; + fno:name "max" ; + dcterms:description "max" ; + fno:solves grel:prob_max ; + fno:expects ( grel:param_dec_n grel:param_dec_n2 ) ; + fno:returns ( grel:output_number ) . -#grel:uri_value -# a fno:Parameter ; -# fno:name "v" ; -# rdfs:label "v" ; -# fno:predicate grel:param_value ; -# fno:type xsd:string ; -# fno:required "true"^^xsd:boolean . +#MOD -#grel:controls_forEachIndex -# a fno:Function ; -# fno:name "forEachIndex" ; -# dcterms:description "forEachIndex" ; -# fno:expects ( grel:param_a grel:uri_index grel:uri_value grel:param_any_e ) ; -# fno:returns ( grel:output_array ) . +grel:math_mod + a fno:Function ; + fno:name "mod" ; + dcterms:description "mod" ; + fno:expects ( grel:param_int_i grel:param_int_i2 ) ; + fno:returns ( grel:output_number ) . -#grel:controls_forNonBlank -# a fno:Function ; -# fno:name "forNonBlank" ; -# dcterms:description "forNonBlank" ; -# fno:expects ( grel:param_any_e grel:uri_value grel:param_false grel:param_true ) ; -# fno:returns ( grel:output_any ) . +grel:prob_coalesce + a fno:Problem ; + fno:name "The coalesce problem"^^xsd:string ; + dcterms:description "Returning the first bound argument."^^xsd:string . -#grel:date_toDate -# a fno:Function ; -# fno:name "toDate" ; -# dcterms:description "toDate" ; -# fno:expects ( grel:param_any_e grel:string_rep_format ) ; -# fno:returns ( grel:output_date ) . +#LN + +grel:math_ln + a fno:Function ; + fno:name "ln" ; + dcterms:description "ln" ; + fno:expects ( grel:param_dec_n ) ; + fno:returns ( grel:output_number ) . + +#LOG + +grel:math_log + a fno:Function ; + fno:name "log" ; + dcterms:description "log" ; + fno:expects ( grel:param_dec_n ) ; + fno:returns ( grel:output_number ) . + +#EXP + +grel:math_exp + a fno:Function ; + fno:name "exp" ; + dcterms:description "exp" ; + fno:expects ( grel:param_dec_n ) ; + fno:returns ( grel:output_number ) . + +#POW + +grel:math_pow + a fno:Function ; + fno:name "pow" ; + dcterms:description "pow" ; + fno:expects ( grel:param_dec_n grel:param_dec_n_exp ) ; + fno:returns ( grel:output_number ) . + +grel:param_dec_n_exp + a fno:Parameter ; + fno:name "e" ; + rdfs:label "e" ; + fno:predicate grel:p_any_exp ; + fno:type xsd:decimal ; + fno:required "true"^^xsd:boolean . -#grel:array_sum +#grel:math_sum # a fno:Function ; # fno:name "sum" ; # dcterms:description "sum" ; -# fno:expects ( ) ; +# fno:expects ( grel:param_a ) ; # fno:returns ( grel:output_number ) . -#grel:controls_forRange -# a fno:Function ; -# fno:name "forRange" ; -# dcterms:description "forRange" ; -# fno:expects ( grel:p_int_i_from grel:inti_i_to grel:param_int_i_step grel:uri_value grel:param_any_e ) ; -# fno:returns ( grel:output_array ) . +#TYPE + +grel:other_type + a fno:Function ; + fno:name "type" ; + dcterms:description "type" ; + fno:expects ( grel:param_any_e ) ; + fno:returns ( grel:stringOut ) . + +#HASFIELD + +grel:other_hasField + a fno:Function ; + fno:name "hasField" ; + dcterms:description "hasField" ; + fno:expects ( grel:param_any_e grel:param_string_field ) ; + fno:returns ( grel:output_bool ) . + +grel:param_string_field + a fno:Parameter ; + fno:name "field" ; + rdfs:label "field" ; + fno:predicate grel:p_string_field ; + fno:type xsd:string ; + fno:required "true"^^xsd:boolean . + +#COALESCE + +grel:other_coalesce + a fno:Function ; + fno:name "coalesce" ; + dcterms:description "coalesce" ; + fno:solves grel:prob_coalesce ; + fno:expects ( grel:param_any_rep_e ) ; + fno:returns ( grel:output_any ) . + +grel:param_any_rep_e + a fno:Parameter ; + fno:name "objects" ; + rdfs:label "objects" ; + fno:predicate grel:p_any_rep_e ; + fno:type xsd:any ; + fno:required "true"^^xsd:boolean . #grel:other_facetCount # a fno:Function ; @@ -1178,98 +1415,13 @@ grel:prob_coalesce # fno:expects ( ) ; # fno:returns ( grel:output_array ) . -#grel:controls_filter -# a fno:Function ; -# fno:name "filter" ; -# dcterms:description "filter" ; -# fno:expects ( grel:param_a grel:uri_value grel:param_any_e ) ; -# fno:returns ( grel:output_array ) . - -#grel:controls_forEach -# a fno:Function ; -# fno:name "forEach" ; -# dcterms:description "forEach" ; -# fno:expects ( grel:param_a grel:uri_value grel:param_any_e ) ; -# fno:returns ( grel:output_array ) . - -#grel:controls_with -# a fno:Function ; -# fno:name "with" ; -# dcterms:description "with" ; -# fno:expects ( grel:param_any_e grel:uri_value grel:param_any_e ) ; -# fno:returns ( grel:output_any ) . - -#grel:date_diff -# a fno:Function ; -# fno:name "diff" ; -# dcterms:description "diff" ; -# fno:expects ( grel:date_d grel:date_d2 grel:string_time_unit ) ; -# fno:returns ( grel:output_number ) . - -#grel:array_get -# a fno:Function ; -# fno:name "get" ; -# dcterms:description "get" ; -# fno:expects ( grel:param_a grel:p_int_i_from grel:p_int_i_opt_to ) ; -# fno:returns ( grel:output_array ) . - -#grel:array_slice -# a fno:Function ; -# fno:name "slice" ; -# dcterms:description "slice" ; -# fno:expects ( grel:param_a grel:p_int_i_from grel:p_int_i_opt_to ) ; -# fno:returns ( grel:output_number ) . - - -#grel:date_inc -# a fno:Function ; -# fno:name "inc" ; -# dcterms:description "inc" ; -# fno:expects ( grel:date_d grel:dec_n grel:string_unit ) ; -# fno:returns ( grel:output_date ) . - -#grel:date_toDate -# a fno:Function ; -# fno:name "toDate" ; -# dcterms:description "toDate" ; -# fno:expects ( ) ; -# fno:returns ( grel:output_date ) . - -#grel:math_mod -# a fno:Function ; -# fno:name "mod" ; -# dcterms:description "mod" ; -# fno:expects ( grel:param_int_i grel:param_int_i2 ) ; -# fno:returns ( grel:output_number ) . - -#grel:math_pow -# a fno:Function ; -# fno:name "pow" ; -# dcterms:description "pow" ; -# fno:expects ( grel:dec_n grel:dec_n_exp ) ; -# fno:returns ( grel:output_number ) . - #grel:other_get # a fno:Function ; # fno:name "get" ; # dcterms:description "get" ; -# fno:expects ( grel:param_any_e grel:string_field ) ; +# fno:expects ( grel:param_any_e grel:param_string_field ) ; # fno:returns ( grel:output_any ) . -#grel:date_toString -# a fno:Function ; -# fno:name "toString" ; -# dcterms:description "toString" ; -# fno:expects ( grel:date_d grel:string_format ) ; -# fno:returns ( grel:stringOut ) . - -#grel:other_hasField -# a fno:Function ; -# fno:name "hasField" ; -# dcterms:description "hasField" ; -# fno:expects ( grel:param_any_e grel:string_field ) ; -# fno:returns ( grel:output_bool ) . - #grel:other_htmlAttr # a fno:Function ; # fno:name "htmlAttr" ; @@ -1284,104 +1436,6 @@ grel:prob_coalesce # fno:expects ( ) ; # fno:returns ( grel:output_element ) . -#grel:array_length -# a fno:Function ; -# fno:name "length" ; -# dcterms:description "length" ; -# fno:expects ( grel:param_a ) ; -# fno:returns ( grel:output_number ) . - -#grel:array_reverse -# a fno:Function ; -# fno:name "reverse" ; -# dcterms:description "reverse" ; -# fno:expects ( grel:param_a ) ; -# fno:returns ( grel:output_array ) . - -#grel:math_sum -# a fno:Function ; -# fno:name "sum" ; -# dcterms:description "sum" ; -# fno:expects ( grel:param_a ) ; -# fno:returns ( grel:output_number ) . - -#grel:array_uniques -# a fno:Function ; -# fno:name "uniques" ; -# dcterms:description "uniques" ; -# fno:expects ( grel:param_a ) ; -# fno:returns ( grel:output_array ) . - -#grel:array_sort -# a fno:Function ; -# fno:name "sort" ; -# dcterms:description "sort" ; -# fno:expects ( grel:param_a ) ; -# fno:returns ( grel:output_array ) . - -#grel:math_exp -# a fno:Function ; -# fno:name "exp" ; -# dcterms:description "exp" ; -# fno:expects ( grel:dec_n ) ; -# fno:returns ( grel:output_number ) . - -#grel:math_ln -# a fno:Function ; -# fno:name "ln" ; -# dcterms:description "ln" ; -# fno:expects ( grel:dec_n ) ; -# fno:returns ( grel:output_number ) . - -#grel:math_log -# a fno:Function ; -# fno:name "log" ; -# dcterms:description "log" ; -# fno:expects ( grel:dec_n ) ; -# fno:returns ( grel:output_number ) . - -#grel:controls_isBlank -# a fno:Function ; -# fno:name "isBlank" ; -# dcterms:description "isBlank" ; -# fno:expects ( grel:param_any_e ) ; -# fno:returns ( grel:output_bool ) . - -#grel:controls_isError -# a fno:Function ; -# fno:name "isError" ; -# dcterms:description "isError" ; -# fno:expects ( grel:param_any_e ) ; -# fno:returns ( grel:output_bool ) . - -#grel:controls_isNonBlank -# a fno:Function ; -# fno:name "isNonBlank" ; -# dcterms:description "isNonBlank" ; -# fno:expects ( grel:param_any_e ) ; -# fno:returns ( grel:output_bool ) . - -#grel:controls_isNotNull -# a fno:Function ; -# fno:name "isNotNull" ; -# dcterms:description "isNotNull" ; -# fno:expects ( grel:param_any_e ) ; -# fno:returns ( grel:output_bool ) . - -#grel:controls_isNull -# a fno:Function ; -# fno:name "isNull" ; -# dcterms:description "isNull" ; -# fno:expects ( grel:param_any_e ) ; -# fno:returns ( grel:output_bool ) . - -#grel:controls_isNumeric -# a fno:Function ; -# fno:name "isNumeric" ; -# dcterms:description "isNumeric" ; -# fno:expects ( grel:param_any_e ) ; -# fno:returns ( grel:output_bool ) . - #grel:other_htmlText # a fno:Function ; # fno:name "htmlText" ; @@ -1410,13 +1464,6 @@ grel:prob_coalesce # fno:expects ( ) ; # fno:returns ( grel:stringOut ) . -#grel:other_type -# a fno:Function ; -# fno:name "type" ; -# dcterms:description "type" ; -# fno:expects ( grel:param_any_e ) ; -# fno:returns ( grel:stringOut ) . - #grel:other_parseHtml # a fno:Function ; # fno:name "parseHtml" ; @@ -1438,10 +1485,44 @@ grel:prob_coalesce # fno:expects ( grel:param_any_e ) ; # fno:returns ( grel:stringOut ) . -#grel:string_toString -# a fno:Function ; -# fno:name "toString" ; -# dcterms:description "toString" ; -# fno:expects ( grel:param_any_e ) ; -# fno:returns ( grel:stringOut ) . +# HERE +#grel:param_b2 +# a fno:Parameter ; +# fno:name "b2" ; +# rdfs:label "b2" ; +# fno:predicate grel:bool_b2 ; +# fno:type xsd:boolean ; +# fno:required "true"^^xsd:boolean . + +#grel:bool_rep_b +# a fno:Parameter ; +# fno:name "b3" ; +# rdfs:label "b3" ; +# fno:predicate grel:param_rep_b ; +# fno:type xsd:string ; +# fno:required "true"^^xsd:boolean . + +#grel:string_format +# a fno:Parameter ; +# fno:name "format" ; +# rdfs:label "format" ; +# fno:predicate grel:param_format ; +# fno:type xsd:string ; +# fno:required "true"^^xsd:boolean . + +#grel:bool_opt_omit +# a fno:Parameter ; +# fno:name "omitFragment" ; +# rdfs:label "omitFragment" ; +# fno:predicate grel:param_omit ; +# fno:type xsd:string ; +# fno:required "false"^^xsd:boolean . + +#grel:param_int_i_to +# a fno:Parameter ; +# fno:name "to" ; +# rdfs:label "to" ; +# fno:predicate grel:param_to ; +# fno:type xsd:int ; +# fno:required "true"^^xsd:boolean . diff --git a/src/main/resources/functions_idlab.ttl b/src/main/resources/functions_idlab.ttl deleted file mode 100644 index c28d299a..00000000 --- a/src/main/resources/functions_idlab.ttl +++ /dev/null @@ -1,210 +0,0 @@ -@prefix cc: . -@prefix dcterms: . -@prefix fno: . -@prefix foaf: . -@prefix grel: . -@prefix rdf: . -@prefix rdfs: . -@prefix void: . -@prefix xsd: . -@prefix idlab-fn: . -grel:void - a void:DatasetDescription ; - dcterms:title "A VoID Description of the GREL Fno Descriptions Dataset" ; - dcterms:creator ; - foaf:primaryTopic grel:dataset ; -. - - - rdf:type foaf:Person ; - foaf:mbox "mailto:Ben.DeMeester@UGent.be" ; - foaf:name "Ben De Meester"@en . - -grel:dataset - a void:Dataset ; - foaf:homepage ; - foaf:homepage ; - dcterms:title "FnO descriptions for IDLab functions"@en ; - dcterms:description "The FnO descriptions of the IDLab functions" ; - dcterms:contributor ; - dcterms:modified "2020-06-16"^^xsd:date ; - cc:license ; - dcterms:issued "2016-12-15"^^xsd:date ; - dcterms:rights "Copyright © Ghent University – imec – IDLab"@en ; - dcterms:subject ; - dcterms:subject ; - void:feature ; - void:exampleResource grel:boolean_and ; - void:vocabulary . - -# Contents: -# - Parameters (per java) param_{var} with predicate p_{type}_{var} -# - Outputs (per java) output_{type} with predicate o_{type} -# - Problems (per java) prob_{var} -# - Functions (per java) - - - -grel:prob_ucase - a fno:Problem ; - fno:name "The ucase problem"^^xsd:string ; - dcterms:description "Converting a string to upper case characters."^^xsd:string . - - - -idlab-fn:toUpperCaseURL - a fno:Function ; - fno:name "toUppercaseURL" ; - rdfs:label "toUppercaseURL" ; - dcterms:description "Returns an uppercase, valid url." ; - fno:solves grel:prob_ucase ; - fno:expects ( idlab-fn:_str ) ; - fno:returns ( idlab-fn:_stringOut ) . - - -idlab-fn:trueCondition - a fno:Function ; - fno:name "True condition" ; - rdfs:label "True condition" ; - dcterms:description "Returns the value if the given boolean is true, else null" ; - fno:expects ( idlab-fn:_strBoolean idlab-fn:_str ) ; - fno:returns ( idlab-fn:_stringOut ) . - -idlab-fn:equal - a fno:Function ; - fno:name "equal" ; - rdfs:label "equal" ; - dcterms:description "Returns true if strings are equal." ; - fno:expects ( grel:valueParam grel:valueParam2 ) ; - fno:returns ( grel:stringOut ) . - -idlab-fn:notEqual - a fno:Function ; - fno:name "not equal" ; - rdfs:label "not equal" ; - dcterms:description "Returns true if strings are not equal." ; - fno:expects ( grel:valueParam grel:valueParam2 ) ; - fno:returns ( grel:stringOut ) . - -idlab-fn:stringContainsOtherString - a fno:Function ; - fno:name "stringContainsOtherString" ; - rdfs:label "stringContainsOtherString" ; - dcterms:description "Returns true if a string is found in a delimited string." ; - fno:expects ( idlab-fn:_str idlab-fn:_otherStr idlab-fn:_delimiter ) ; - fno:returns ( idlab-fn:_stringOut ) . - - - -idlab-fn:random - a fno:Function ; - fno:name "random" ; - rdfs:label "random" ; - dcterms:description "Returns a random uuid." ; - fno:expects ( ) ; - fno:returns ( idlab-fn:_stringOut ) . - -idlab-fn:_path - a fno:Parameter ; - fno:name "input path" ; - rdfs:label "input path" ; - fno:type xsd:string ; - fno:predicate idlab-fn:path . - -idlab-fn:_str - a fno:Parameter ; - fno:name "input string" ; - rdfs:label "input string" ; - fno:type xsd:string ; - fno:predicate idlab-fn:str . - -idlab-fn:_strBoolean - a fno:Parameter ; - fno:name "boolean encoded as string" ; - rdfs:label "boolean encoded as string" ; - fno:type xsd:string ; - fno:predicate idlab-fn:strBoolean . - -idlab-fn:_endpoint - a fno:Parameter ; - fno:name "endpoint url string" ; - rdfs:label "endpoint url string" ; - fno:type xsd:string ; - fno:predicate idlab-fn:endpoint . - -idlab-fn:_otherStr - a fno:Parameter ; - fno:name "other input string" ; - rdfs:label "other input string" ; - fno:type xsd:string ; - fno:predicate idlab-fn:otherStr . - -idlab-fn:_expectedStr - a fno:Parameter ; - fno:name "expected input string" ; - rdfs:label "expected input string" ; - fno:type xsd:string ; - fno:predicate idlab-fn:expectedStr . - -idlab-fn:_result - a fno:Parameter ; - fno:name "result string" ; - rdfs:label "result string" ; - fno:type xsd:string ; - fno:predicate idlab-fn:result . - -idlab-fn:_list - a fno:Parameter ; - fno:name "list" ; - rdfs:label "list" ; - fno:type rdf:List ; - fno:predicate idlab-fn:list . - -idlab-fn:_listOut - a fno:Parameter ; - fno:name "output list" ; - rdfs:label "output list" ; - fno:type rdf:List ; - fno:predicate idlab-fn:listOut . - -idlab-fn:_delimiter - a fno:Parameter ; - fno:name "delimiter" ; - rdfs:label "delimiter" ; - fno:type xsd:string ; - fno:predicate idlab-fn:delimiter . - -idlab-fn:_test - a fno:Parameter ; - fno:name "test" ; - rdfs:label "test" ; - fno:type xsd:double ; - fno:predicate idlab-fn:p_test . - -idlab-fn:_from - a fno:Parameter ; - fno:name "from" ; - rdfs:label "from" ; - fno:type xsd:double ; - fno:predicate idlab-fn:p_from . - -idlab-fn:_to - a fno:Parameter ; - fno:name "to" ; - rdfs:label "to" ; - fno:type xsd:double ; - fno:predicate idlab-fn:p_to . - -idlab-fn:_stringOut - a fno:Output ; - fno:name "output string" ; - rdfs:label "output string" ; - fno:type xsd:string ; - fno:predicate idlab-fn:stringOut . - -idlab-fn:_boolOut - a fno:Output ; - fno:name "output boolean" ; - rdfs:label "output boolean" ; - fno:type xsd:boolean ; - fno:predicate idlab-fn:o_boolOut . diff --git a/src/main/resources/grel_java_mapping.ttl b/src/main/resources/grel_java_mapping.ttl deleted file mode 100644 index ba3c6341..00000000 --- a/src/main/resources/grel_java_mapping.ttl +++ /dev/null @@ -1,322 +0,0 @@ -@prefix cc: . -@prefix dcterms: . -@prefix doap: . -@prefix fno: . -@prefix fnoi: . -@prefix fnom: . -@prefix foaf: . -@prefix grel: . -@prefix grelm: . -@prefix rdf: . -@prefix rdfs: . -@prefix void: . -@prefix xsd: . - -grelm:void - a void:DatasetDescription ; - dcterms:title "A VoID Description of the GREL Fno JAVA Mapping Descriptions Dataset" ; - dcterms:creator ; - foaf:primaryTopic grelm:dataset ; -. - - - rdf:type foaf:Person ; - foaf:mbox "mailto:Ben.DeMeester@UGent.be" ; - foaf:name "Ben De Meester"@en . - -grelm:dataset - a void:Dataset ; - foaf:homepage ; - foaf:homepage ; - dcterms:title "GREL Fno JAVA Mapping Descriptions"@en ; - dcterms:description "The FnO JAVA mapping descriptions of the GREL functions" ; - dcterms:contributor ; - dcterms:modified "2020-01-21"^^xsd:date ; - cc:license ; - dcterms:issued "2020-01-21"^^xsd:date ; - dcterms:rights "Copyright © Ghent University – imec – IDLab"@en ; - dcterms:subject ; - dcterms:subject ; - void:feature ; - void:exampleResource grelm:boolean_and ; - void:vocabulary , , - . - -# Contents: -# - Parameters (per java) param_{var} with predicate p_{type}_{var} -# - Outputs (per java) output_{type} with predicate o_{type} -# - Problems (per java) prob_{var} -# - Functions (per java) - -grelm:javaArray - a fnoi:JavaClass ; - doap:download-page "GrelFunctions.jar" ; - fnoi:class-name "io.fno.grel.ArrayFunctions" . - -#CONCAT - -grelm:getMapping - a fno:Mapping ; - fno:function grel:array_get ; - fno:implementation grelm:javaArray ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "get" ] ; -. - -#CONCAT - -grelm:joinMapping - a fno:Mapping ; - fno:function grel:array_join ; - fno:implementation grelm:javaArray ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "join" ] ; -. - -grelm:javaBoolean - a fnoi:JavaClass ; - doap:download-page "GrelFunctions.jar" ; - fnoi:class-name "io.fno.grel.BooleanFunctions" . - -#AND - -grelm:andMapping - a fno:Mapping ; - fno:function grel:boolean_and ; - fno:implementation grelm:javaBoolean ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "and" ] ; -. - -#OR - -grelm:orMapping - a fno:Mapping ; - fno:function grel:boolean_or ; - fno:implementation grelm:javaBoolean ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "or" ] ; -. - -#NOT - -grelm:notMapping - a fno:Mapping ; - fno:function grel:boolean_not ; - fno:implementation grelm:javaBoolean ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "not" ] ; -. - -#XOR - -grelm:xorMapping - a fno:Mapping ; - fno:function grel:boolean_xor ; - fno:implementation grelm:javaBoolean ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "xor" ] ; -. - -grelm:javaControls - a fnoi:JavaClass ; - doap:download-page "GrelFunctions.jar" ; - fnoi:class-name "io.fno.grel.ControlsFunctions" . - -#IFTHENELSE - -grelm:ifthenelseMapping - a fno:Mapping ; - fno:function grel:controls_if ; - fno:implementation grelm:javaControls ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "ifThenElse" ] ; -. - -grelm:javaString - a fnoi:JavaClass ; - doap:download-page "GrelFunctions.jar" ; - fnoi:class-name "io.fno.grel.StringFunctions" . - -#STRING_LENGTH - -grelm:lengthMapping - a fno:Mapping ; - fno:function grel:string_length ; - fno:implementation grelm:javaString ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "length" ] ; -. - -#STARTSWITH - -grelm:startsWithMapping - a fno:Mapping ; - fno:function grel:string_startsWith ; - fno:implementation grelm:javaString ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "startsWith" ] ; -. - -#ENDSWITH - -grelm:endsWithMapping - a fno:Mapping ; - fno:function grel:string_endsWith ; - fno:implementation grelm:javaString ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "endsWith" ] ; -. - -#CONTAINS - -grelm:containsMapping - a fno:Mapping ; - fno:function grel:string_contains ; - fno:implementation grelm:javaString ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "contains" ] ; -. - -#TOLOWERCASE - -grelm:lowercaseMapping - a fno:Mapping ; - fno:function grel:toLowerCase ; - fno:implementation grelm:javaString ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "toLowercase" ] ; -. - -#TOUPPERCASE - -grelm:uppercaseMapping - a fno:Mapping ; - fno:function grel:toUpperCase ; - fno:implementation grelm:javaString ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "toUppercase" ] ; -. - -#TRIM - -grelm:trimMapping - a fno:Mapping ; - fno:function grel:string_trim ; - fno:implementation grelm:javaString ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "trim" ] ; -. - -#CHOMP - -grelm:trimMapping - a fno:Mapping ; - fno:function grel:string_chomp ; - fno:implementation grelm:javaString ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "chomp" ] ; -. - -#SUBSTRING - -grelm:substringMapping - a fno:Mapping ; - fno:function grel:string_substring ; - fno:implementation grelm:javaString ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "substring" ] ; -. - -#SPLIT - -grelm:splitMapping - a fno:Mapping ; - fno:function grel:string_split ; - fno:implementation grelm:javaString ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "split" ] ; -. - -#ESCAPE - -grelm:escapeMapping - a fno:Mapping ; - fno:function grel:escape ; - fno:implementation grelm:javaString ; -# TODO if not clear - fno:parameterMapping [ ] ; -# TODO if not clear - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "escape" ] ; -. - diff --git a/src/main/resources/idlab_java_mapping.ttl b/src/main/resources/idlab_java_mapping.ttl deleted file mode 100644 index ceb6343e..00000000 --- a/src/main/resources/idlab_java_mapping.ttl +++ /dev/null @@ -1,125 +0,0 @@ -@prefix cc: . -@prefix dcterms: . -@prefix doap: . -@prefix fno: . -@prefix fnoi: . -@prefix fnom: . -@prefix foaf: . -@prefix grel: . -@prefix grelm: . -@prefix rdf: . -@prefix rdfs: . -@prefix void: . -@prefix xsd: . -@prefix idlab-fn: . -grelm:void - a void:DatasetDescription ; - dcterms:title "A VoID Description of the IDLab Fno Java Mapping Descriptions Dataset" ; - dcterms:creator ; - foaf:primaryTopic grelm:dataset ; -. - - - rdf:type foaf:Person ; - foaf:mbox "mailto:Ben.DeMeester@UGent.be" ; - foaf:name "Ben De Meester"@en . - -grelm:dataset - a void:Dataset ; - foaf:homepage ; - foaf:homepage ; - dcterms:title "IDLab Fno Java Mapping Descriptions"@en ; - dcterms:description "The FnO Java mapping descriptions of the IDLab functions" ; - dcterms:contributor ; - dcterms:modified "2020-06-16"^^xsd:date ; - cc:license ; - dcterms:issued "2020-06-16"^^xsd:date ; - dcterms:rights "Copyright © Ghent University – imec – IDLab"@en ; - dcterms:subject ; - dcterms:subject ; - void:feature ; - void:exampleResource grelm:boolean_and ; - void:vocabulary , , - . - -# Contents: -# - Parameters (per java) param_{var} with predicate p_{type}_{var} -# - Outputs (per java) output_{type} with predicate o_{type} -# - Problems (per java) prob_{var} -# - Functions (per java) - -################################ -#### IMPLEMENTATIONS -################################ -grelm:IDLabFunctions - a fnoi:JavaClass ; - doap:download-page "IDLabFunctions.jar" ; - fnoi:class-name "io.fno.idlab.IDLabFunctions" . - -################################ -#### MAPPINGS -################################ - -#UPPERCASERURL -grelm:uppercaseURLMapping - a fno:Mapping ; - fno:function idlab-fn:toUpperCaseURL; - fno:implementation grelm:IDLabFunctions ; - fno:parameterMapping [ ] ; - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "toUpperCaseURL" ] ; -. - -grelm:trueConditionMapping - a fno:Mapping ; - fno:function idlab-fn:trueCondition; - fno:implementation grelm:IDLabFunctions ; - fno:parameterMapping [ ] ; - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "trueCondition" ] ; -. - - -grelm:equalMapping - a fno:Mapping ; - fno:function idlab-fn:equal; - fno:implementation grelm:IDLabFunctions ; - fno:parameterMapping [ ] ; - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "equal" ] ; -. - - -grelm:notEqualMapping - a fno:Mapping ; - fno:function idlab-fn:notEqual; - fno:implementation grelm:IDLabFunctions ; - fno:parameterMapping [ ] ; - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "notEqual" ] ; -. - -grelm:stringContainsOtherStringMapping - a fno:Mapping ; - fno:function idlab-fn:stringContainsOtherString; - fno:implementation grelm:IDLabFunctions ; - fno:parameterMapping [ ] ; - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "stringContainsOtherString" ] ; -. - -grelm:randomMapping - a fno:Mapping ; - fno:function idlab-fn:random; - fno:implementation grelm:IDLabFunctions ; - fno:parameterMapping [ ] ; - fno:returnMapping [ ] ; - fno:methodMapping [ a fnom:StringMethodMapping ; - fnom:method-name "random" ] ; -. - diff --git a/src/main/scala/io/rml/framework/Main.scala b/src/main/scala/io/rml/framework/Main.scala index 3f719d37..08d248aa 100644 --- a/src/main/scala/io/rml/framework/Main.scala +++ b/src/main/scala/io/rml/framework/Main.scala @@ -26,7 +26,7 @@ package io.rml.framework **/ -import io.rml.framework.api.{FnOEnvironment, RMLEnvironment} +import io.rml.framework.api.RMLEnvironment import io.rml.framework.core.extractors.{JoinConfigMapCache, NodeCache} import io.rml.framework.core.internal.Logging import io.rml.framework.core.item.{EmptyItem, Item, JoinedItem} @@ -37,9 +37,9 @@ import io.rml.framework.engine._ import io.rml.framework.engine.composers.StreamJoinComposer import io.rml.framework.engine.statement.StatementEngine import io.rml.framework.flink.connector.kafka.{RMLPartitioner, UniversalKafkaConnectorFactory} -import io.rml.framework.flink.function.{FnOEnvironmentLoader, FnOEnvironmentStreamLoader, RichItemIdentityFunction, RichStreamItemIdentityFunction} import io.rml.framework.flink.sink.{RichMQTTSink, TargetSinkFactory} import io.rml.framework.flink.source.{FileDataSet, Source} +import io.rml.framework.flink.util.FunctionsFlinkUtil import org.apache.flink.api.common.eventtime.{SerializableTimestampAssigner, WatermarkStrategy} import org.apache.flink.api.common.serialization.{SimpleStringEncoder, SimpleStringSchema} import org.apache.flink.api.scala._ @@ -94,17 +94,12 @@ object Main extends Logging { // Triple maps are also organized in categories (does it contain streams, does it contain joins, ... ) val formattedMapping = Util.readMappingFile(config.mappingFilePath) - - // Default function config - // TODO: support adding variable function related files using CLI arguments - FnOEnvironment.loadDefaultConfiguration() - FnOEnvironment.intializeFunctionLoader() - - // set up execution environments, Flink needs these to know how to operate (local, cluster mode, ...) implicit val env = ExecutionEnvironment.getExecutionEnvironment implicit val senv = StreamExecutionEnvironment.getExecutionEnvironment + // TODO: check FunctionUtils.scala + FunctionsFlinkUtil.putFunctionFilesInFlinkCache(env.getJavaEnv, senv.getJavaEnv, config.functionDescriptionLocations.get: _*) if (config.checkpointInterval.isDefined) { senv.enableCheckpointing(config.checkpointInterval.get, CheckpointingMode.AT_LEAST_ONCE); // This is what Kafka supports ATM, see https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/connectors/guarantees.html @@ -269,13 +264,11 @@ object Main extends Logging { ).nonEmpty } else true // if there are no join conditions all items can pass // filter out all empty items (some iterators can emit empty items) - }).filter(iterItems => { - iterItems.nonEmpty - }) - - + }) + .filter(iterItems => { + iterItems.nonEmpty + }) - val parentTriplesMap = NodeCache.getTriplesMap(tm.parentTriplesMap).get; val parentDataStreamTimestamped = Source(parentTriplesMap.logicalSource).asInstanceOf[io.rml.framework.flink.source.Stream] .stream @@ -361,17 +354,6 @@ object Main extends Logging { } }) - val preProcessingFunction = - if (FnOEnvironment.getFunctionLoader.isDefined) { - val functionLoaderOption = FnOEnvironment.getFunctionLoader - val jarSources = functionLoaderOption.get.getSources - val classNames = functionLoaderOption.get.getClassNames - new FnOEnvironmentStreamLoader(jarSources, classNames) - } else { - logInfo("FunctionLoader in RMLEnvironment is NOT DEFINED") - new RichStreamItemIdentityFunction() - } - // Create sinks for every logical target val logicalTargetId2Sinks = TargetSinkFactory.createStreamSinksFromLogicalTargetCache() @@ -384,7 +366,6 @@ object Main extends Logging { val dataStream = source.stream // this will generate a stream of items // process every item by a processor with a loaded engine - .map(preProcessingFunction) .map(new StdStreamProcessor(engine)) .name("Execute mapping statements on items") @@ -610,17 +591,6 @@ object Main extends Logging { } }) - val preProcessingFunction = - if (FnOEnvironment.getFunctionLoader.isDefined) { - val functionLoaderOption = FnOEnvironment.getFunctionLoader - val jarSources = functionLoaderOption.get.getSources - val classNames = functionLoaderOption.get.getClassNames - new FnOEnvironmentLoader(jarSources, classNames) - } else { - logInfo("FunctionLoader in RMLEnvironment is NOT DEFINED") - new RichItemIdentityFunction() - } - // This is the collection of all data streams that are created by the current mapping val processedDataSets: immutable.Iterable[DataSet[String]] = sourceEngineMap.map(entry => { @@ -630,7 +600,6 @@ object Main extends Logging { source.dataset // this will generate a dataset of items // process every item by a processor with a loaded engine - .map(preProcessingFunction) .map(new StdStaticProcessor(engine)) .name("Execute mapping statements on items") .map(outputStringToLogicalTargetIDs => { @@ -672,17 +641,16 @@ object Main extends Logging { // filter out all items that do not contain the childs join condition .filter(item => { - if (tm.joinCondition.isDefined) { + if (tm.joinCondition.isDefined) { val child_attributes = tm.joinCondition.get.child - child_attributes.flatMap( lit => item.refer(lit.value)).nonEmpty } else true // if there are no join conditions all items can pass - - // filter out all empty items (some iterators can emit empty items) - }).filter(item => { - !item.isInstanceOf[EmptyItem] - }) + }) + // filter out all empty items (some iterators can emit empty items) + .filter(item => { + !item.isInstanceOf[EmptyItem] + }) val parentTriplesMap = NodeCache.getTriplesMap(tm.parentTriplesMap).get; diff --git a/src/main/scala/io/rml/framework/api/FnOEnvironment.scala b/src/main/scala/io/rml/framework/api/FnOEnvironment.scala deleted file mode 100644 index 26409ac7..00000000 --- a/src/main/scala/io/rml/framework/api/FnOEnvironment.scala +++ /dev/null @@ -1,94 +0,0 @@ -package io.rml.framework.api - -import io.rml.framework.core.function.FunctionLoader -import io.rml.framework.core.internal.Logging -import io.rml.framework.core.util.Util - -import scala.collection.mutable -import scala.reflect.io.Path - -object FnOEnvironment extends Logging{ - private val functionDescriptionFilePaths : mutable.MutableList[Path] = mutable.MutableList() - private val functionMappingFilePaths : mutable.MutableList[Path] = mutable.MutableList() - private var functionLoader : Option[FunctionLoader] = None - val loadedClassesMap : scala.collection.mutable.Map[String, Class[_]] = scala.collection.mutable.Map() - - def getFunctionDescriptionFilePaths() = { - this.functionDescriptionFilePaths.toList - } - def addFunctionDescriptionFilePath(path : Path) : Unit = { - this.functionDescriptionFilePaths += path - } - - def getFunctionMappingFilePaths() = { - this.functionMappingFilePaths.toList - } - def addFunctionMappingFilePaths(path : Path): Unit = { - this.functionMappingFilePaths += path - } - - def intializeFunctionLoader() = { - this.functionLoader = FunctionLoader.apply(getFunctionDescriptionFilePaths(), getFunctionMappingFilePaths()) - this.functionLoader - } - def getFunctionLoader: Option[FunctionLoader] = { - this.functionLoader - } - - /** - * Default FnO Configuration. - * Function descriptions: - * - functions_grel.ttl - * - functions_idlab.ttl - * Function mappings - * - grel_java_mapping.ttl - * - idlab_java_mapping.ttl - * - * RMLStreamer will look for these files in directory where the RMLStreamer is executed from. - * Note: make sure to add jars with custom functions to Flink's `/lib` directory. - */ - def loadDefaultConfiguration() = { - - val defaultFunctionDescriptionFilePaths = List( - "./functions_grel.ttl", - "./functions_idlab.ttl" - ) - - val defaultFunctionMappingFilePaths = List( - "./grel_java_mapping.ttl", - "./idlab_java_mapping.ttl" - ) - - // adding default function description file paths to the RMLEnvironment - defaultFunctionDescriptionFilePaths.foreach(strPath=> { - try - { - val file = Util.getFile(strPath); - if (file.exists()) { - val p = Path.string2path(file.getAbsolutePath) - FnOEnvironment.addFunctionDescriptionFilePath(p) - } else { - logWarning(s"Can't add function description file to RMLEnvironment ( $strPath ). This will result in errors when using functions!"); - } - } - catch { - case e : Exception => logWarning(s"Can't add function description file to RMLEnvironment ( $strPath ). This will result in errors when using functions! Exception: ${e.getMessage}") - } - }) - - // adding default function description file paths to the RMLEnvironment - defaultFunctionMappingFilePaths.foreach(strPath=> { - try { - val file = Util.getFile(strPath); - if (file.exists()) { - val p = Path.string2path(file.getAbsolutePath) - FnOEnvironment.addFunctionMappingFilePaths(p) - } else { - logWarning(s"Can't add function mapping file to RMLEnvironment ( $strPath ). This will result in errors when using functions!"); - } - }catch { - case e : Exception => logWarning(s"Can't add function mapping file to RMLEnvironment ( $strPath ). This will result in errors when using functions! Exception: ${e.getMessage}") - } - }) - } -} diff --git a/src/main/scala/io/rml/framework/core/function/FunctionLoader.scala b/src/main/scala/io/rml/framework/core/function/FunctionLoader.scala deleted file mode 100644 index affa8df8..00000000 --- a/src/main/scala/io/rml/framework/core/function/FunctionLoader.scala +++ /dev/null @@ -1,140 +0,0 @@ -package io.rml.framework.core.function - -import io.rml.framework.api.{FnOEnvironment, RMLEnvironment} -import io.rml.framework.core.function.model.{Function, FunctionMetaData, Parameter} -import io.rml.framework.core.function.std.StdFunctionLoader -import io.rml.framework.core.internal.Logging -import io.rml.framework.core.model.Uri -import io.rml.framework.core.model.rdf.{RDFGraph, RDFNode} -import io.rml.framework.core.util.Turtle - -import java.io.{File, IOException} -import scala.collection.mutable.{Map => MutableMap} -import scala.reflect.io.Path - -abstract class FunctionLoader extends Logging { - /** - * Map Function Uri to FunctionMetaData object - */ - protected val functionMap: MutableMap[Uri, FunctionMetaData] = MutableMap() - - def getSources: List[String] = { - functionMap.values.map(_.source).toList.distinct - } - def getClassNames : List[String] = { - functionMap.values.map(_.className).toList.distinct - } - /** - * Creates and returns a [[Function]] for [[FunctionMetaData]] given function [[Uri]], if the function uri is present. - * - * @param uri: function uri - * @return function (if successful) - */ - def createFunction(uri: Uri): Option[Function] = { - logDebug(s"createFunction: ${uri.value}") - - val optFunctionMetaData = functionMap.get(uri) - - if (optFunctionMetaData.isDefined) { - val functionMetaData = optFunctionMetaData.get - logDebug(s"Creating function: $uri, ${functionMetaData.toString}" ) - Some(Function(functionMetaData.identifier, functionMetaData)) - } else { - // when the function uri is not present in the function map, complain. - val availableFunctionURIs = functionMap.keys.map(u=>u.toString) - throw new IOException(s"The function with URI ${uri.value} can not be found.\n" + - s"The available function URIs are: " + availableFunctionURIs) - } - } - - /** - * The given `functionMappingFile` should be a Turtle-file containing the function mappings. These mappings will be parsed and - * the FunctionLoader's functionMap is updated accordingly. - * - * @param functionMappingFile - * @return - */ - def parseFunctionMapping(functionMappingFile: File): FunctionLoader = { - val graph = RDFGraph.fromFile(functionMappingFile, RMLEnvironment.getGeneratorBaseIRI(),Turtle) - parseFunctionMapping(graph) - this - } - - /** - * The given `graph` should contain the function mappings. These mappings will be parsed and - * the FunctionLoader's functionMap is updated accordingly. - * - * @param graph [[RDFGraph]] representing a function mapping - * @return [[FunctionLoader]] - */ - def parseFunctionMapping(graph: RDFGraph): FunctionLoader - - /** - * Parse [[Parameter]] from the given [[RDFNode]] which represents the parameter - * - * @param rdfNode [[RDFNode]] containing the mapping information of the parameter - * @param pos [[Int]] integer position of the parameter - * @return - */ - def parseParameter(rdfNode: RDFNode, pos: Int): Parameter - -} - - -object FunctionLoader extends Logging{ - private var singletonFunctionLoader : Option[FunctionLoader] = None - - /** - * Private helper method for reading in the function descriptions as an RDFGraph. - * @param filePath - * @return RDFGraph containing the function descriptions. - */ - private def readFunctionDescriptionsFromFile(filePath : String): RDFGraph = { - logDebug(s"FunctionLoader is reading function descriptions from : ${filePath}") - RDFGraph.fromFile(new File(filePath), RMLEnvironment.getGeneratorBaseIRI(), Turtle) - } - - /** - * Construction of the (singleton) FunctionLoader instance. - * @return FunctionLoader - */ - def apply(functionDescriptionPaths : List[Path] = FnOEnvironment.getFunctionDescriptionFilePaths(), - functionMappingPaths : List[Path] = FnOEnvironment.getFunctionMappingFilePaths() - ): Option[FunctionLoader] = { - if(singletonFunctionLoader.isEmpty) { - - // The functionDescriptionsGraph is populated by iterating over the filepaths of the function description files. - // These function description files are obtained from the RMLEnvironment. - val fdit =functionDescriptionPaths.map(_.path).iterator - // construct the initial functionDescriptionTriplesGraph using the first functiondescription filepath - val functionDescriptionsGraph : Option[RDFGraph] = - if(fdit.hasNext) - Some(readFunctionDescriptionsFromFile(fdit.next())) - else - None - - // If more function description filepaths are specified, they will be read in. - // The resulting triples will be added to the functionDescriptionsGraph - while (fdit.hasNext) { - val fdescGraph = readFunctionDescriptionsFromFile(fdit.next()) - functionDescriptionsGraph.get.addTriples(fdescGraph.listTriples) - } - - // construct functionLoader - if(functionDescriptionsGraph.isDefined) { - singletonFunctionLoader = Some(StdFunctionLoader(functionDescriptionsGraph.get)) - // now parse the mappings - functionMappingPaths.foreach(fmp => { - singletonFunctionLoader.get.parseFunctionMapping(new File(fmp.path)) - }) - - } else { - logWarning("No function graph found. Continuing without loading functions.") - None - } - } - singletonFunctionLoader - } -} - - diff --git a/src/main/scala/io/rml/framework/core/function/FunctionUtils.scala b/src/main/scala/io/rml/framework/core/function/FunctionUtils.scala index 26b59e91..20ed53a9 100644 --- a/src/main/scala/io/rml/framework/core/function/FunctionUtils.scala +++ b/src/main/scala/io/rml/framework/core/function/FunctionUtils.scala @@ -1,51 +1,15 @@ package io.rml.framework.core.function -import io.rml.framework.api.RMLEnvironment import io.rml.framework.core.function.model.Parameter import io.rml.framework.core.internal.Logging import io.rml.framework.core.model.{Entity, Literal, Uri} import io.rml.framework.core.vocabulary.{RDFVoc, XsdVoc} import io.rml.framework.shared.RMLException -import java.io.File -import java.net.MalformedURLException import java.time.Instant -import scala.reflect.internal.util.ScalaClassLoader.URLClassLoader object FunctionUtils extends Logging { - - /** * - * TODO: IMPLEMENT THIS THE SCALA WAY SINCE TYPE ERASURE WILL CAUSE PROBLEM WITH REFLECTIONS IN SCALA - * - */ - /** - * See [[http://stackoverflow.com/questions/3216780/problem-reloading-a-jar-using-urlclassloader?lq=1]] - * for problems relating to unclosed dirty jar file handle if there ever was a need to reload the jar file - * at runtime. - * - * See [[http://stackoverflow.com/questions/8867766/forName()scala-dynamic-object-class-loading]] - * for loading in scala object using reflection - * - * See [[https://medium.com/@giposse/scala-reflection-d835832ed13a]] for a better scala reflection tutorial? - * - * @param jarFile - * @param className - * @return - */ - def loadClassFromJar(jarFile: File, className: String): Class[_] = { - logDebug(s"Loading $className from jar file $jarFile") - try { - val classloader = new URLClassLoader(List(jarFile.toURI.toURL), RMLEnvironment.getClass.getClassLoader) - logDebug(s"Class loader ${classloader.getParent}") - Class.forName(className, true, classloader) - } catch { - case e@(_: MalformedURLException | _: ClassNotFoundException) => - logError(e.getMessage) - throw e - } - } - def typeCastDataType(output: Entity, dataType: Option[Uri]): Option[Entity] = { if(dataType.isDefined){ val typeClass = getTypeClass(dataType.get) diff --git a/src/main/scala/io/rml/framework/core/function/model/DynamicFunction.scala b/src/main/scala/io/rml/framework/core/function/model/DynamicFunction.scala index ec8d3f32..e69de29b 100644 --- a/src/main/scala/io/rml/framework/core/function/model/DynamicFunction.scala +++ b/src/main/scala/io/rml/framework/core/function/model/DynamicFunction.scala @@ -1,156 +0,0 @@ -package io.rml.framework.core.function.model - -import io.rml.framework.api.FnOEnvironment -import io.rml.framework.core.function.{FunctionUtils, ReflectionUtils} -import io.rml.framework.core.model.rdf.SerializableRDFQuad -import io.rml.framework.core.model.{Entity, Literal, Uri} - -import java.io.{File, IOException, ObjectInputStream, ObjectOutputStream} -import java.lang.reflect.Method - -import java.io.{File, IOException, ObjectInputStream, ObjectOutputStream} -import java.lang.reflect.Method - - -/** - * A dynamic function is loaded from an external jar at-runtime. - * - * @param identifier [[String]] used to identify this DynamicFunction - * @param metaData contains information required for loading, initializing the function - */ -case class DynamicFunction(identifier: String, metaData: FunctionMetaData) extends Function { - - @transient - private var optMethod: Option[Method] = None - - @throws(classOf[IOException]) - private def writeObject(out: ObjectOutputStream): Unit = { - out.defaultWriteObject() - } - - /** - * There are two ways in which the method can be loaded: - * 1. From the FnOEnvironment. This means that the class was by Flink's UserCodeClassLoader was able to retrieve - * the class from an external jar (located in FLINK_HOME/lib) - * 2. From a jar that is bundled with the RMLStreamer. - * @return - */ - private def loadMethod() : Option[Method] = { - val optClass = FnOEnvironment.loadedClassesMap.get(metaData.className) - if (optClass.isDefined) { - logDebug(s"The class ${metaData.className} was found in the external jars. Will try to load the function ${metaData.methodName}") - optMethod = ReflectionUtils.searchByMethodNameAndParameterTypes(optClass.get, metaData.methodName, metaData.inputParam.map(_.paramType): _*) - } else { - logInfo(s"Could not find ${metaData.className} for this function in the FnOEnvironment" + - s" (i.e. the class could not be found in an external jar)." + - s"If the specified source (${metaData.source}) is included in the RML Streamer's resources, " + - s"and the class (${metaData.className}) can be found, " + - s"then ${metaData.methodName} will be loaded from there.") - val jarFile = getClass.getClassLoader.getResource(metaData.source).getFile - val classOfMethod = FunctionUtils.loadClassFromJar(new File(jarFile), metaData.className) - optMethod = ReflectionUtils.searchByMethodNameAndParameterTypes(classOfMethod, metaData.methodName, metaData.inputParam.map(_.paramType): _*) - } - optMethod - } - - override def execute(paramTriples: List[SerializableRDFQuad]): Option[Iterable[Entity]] = { - // if a group (key: uri) results in a list with 1 element, extract that single element - // otherwise, when a group has a list with more than 1 element, keep it as a list - val argResourcesGroupedByUri = paramTriples.groupBy(_.predicate).map { - pair => { - pair._2.length match { - case 0 => pair._1.uri -> None - case 1 => pair._1.uri -> pair._2.head - case _ => pair._1.uri -> pair._2.toList - } - } - } - - val argObjectsGroupedByUri = argResourcesGroupedByUri.map { - pair => { - if (pair._2.isInstanceOf[Iterable[SerializableRDFQuad]]) { - pair._1 -> pair._2.asInstanceOf[Iterable[SerializableRDFQuad]].map(x => x.`object`.value.value) - } else { - pair._1 -> pair._2.asInstanceOf[SerializableRDFQuad].`object`.value.value - } - - } - } - - val orderedArgValues = metaData - .inputParam - .sortBy(_.position).flatMap(ip => { - - if (argObjectsGroupedByUri.contains(ip.paramUri)) - argObjectsGroupedByUri.get(ip.paramUri) - else - Some(null) - - }) - - if (orderedArgValues.size == metaData.inputParam.size) { - - val method = loadMethod.getOrElse(throw new Exception("No method was initialized.")) - val castParameterValues: Array[AnyRef] = ReflectionUtils.castUsingMethodParameterTypes(method, orderedArgValues) - - try { - val output = method.invoke(null, castParameterValues: _*) - val result = metaData.outputParam - .flatMap(elem => elem.getValue(output)) - .map (elem => { - Literal(elem.toString) - }) - Some(result) - } catch { - case e: Throwable => { - logError(s"The following exception occurred when invoking the method ${method.getName}: ${e.getMessage}." + - s"\nThe result will be set to None.", e) - None - } - } - } - else None - - - } - - - - - @throws(classOf[IOException]) - @throws(classOf[ClassNotFoundException]) - private def readObject(in: ObjectInputStream): Unit = { - in.defaultReadObject() - optMethod = None - - } - - override def execute(arguments: Map[Uri, String]): Option[Iterable[Entity]] = { - val inputParams = metaData.inputParam - // casted to List[AnyRef] since method.invoke(...) only accepts reference type but not primitive type of Scala - val paramsOrdered = arguments.groupBy(_._1.value).map(_._2.asInstanceOf[AnyRef]).toList - - val outputParams = metaData.outputParam - - if (paramsOrdered.size == inputParams.size) { - val method = optMethod.get - val castedParameterValues = ReflectionUtils.castUsingMethodParameterTypes(method, paramsOrdered) - val output = method.invoke(null, castedParameterValues: _*) - - if (output != null) { - val result = outputParams.flatMap(elem => elem.getValue(output)) map (elem => Literal(elem.toString)) - Some(result) - } else - None - } else { - //TODO: complain about inputparams size != params ordered - logError(s"Not all input parameters for ${metaData.methodName} could be bound...") - None - } - } - - - override def getMethod: Option[Method] = { - optMethod - } -} diff --git a/src/main/scala/io/rml/framework/core/function/model/Function.scala b/src/main/scala/io/rml/framework/core/function/model/Function.scala index d86de34a..3290ff3b 100644 --- a/src/main/scala/io/rml/framework/core/function/model/Function.scala +++ b/src/main/scala/io/rml/framework/core/function/model/Function.scala @@ -1,30 +1,46 @@ package io.rml.framework.core.function.model +import be.ugent.idlab.knows.functions.agent.{Agent, Arguments} import io.rml.framework.core.internal.Logging import io.rml.framework.core.model.rdf.SerializableRDFQuad -import io.rml.framework.core.model.{Entity, Node, Uri} +import io.rml.framework.core.model.{Entity, Literal, Node} -import java.lang.reflect.Method abstract class Function extends Node with Logging{ - - - def name: Uri = Uri(identifier) - - def getMethod: Option[Method] - - - def execute(paramTriples: List[SerializableRDFQuad]): Option[Iterable[Entity]] - - @deprecated("Please use execute(paramTriples: List[FlinkRDFQuad]) instead") - def execute(argumentsMap: Map[Uri, String]): Option[Iterable[Entity]] - - + def execute(identifier: String, paramTriples: List[SerializableRDFQuad]): Option[Iterable[Entity]] } object Function extends Logging{ - def apply(identifier:String, functionMetaData: FunctionMetaData): Function={ - logDebug("Companion: Function - apply(identifier, functionMetaData)") - DynamicFunction(identifier, functionMetaData) + def execute(identifier: String, paramTriples: List[SerializableRDFQuad], agent: Agent): Option[Iterable[Entity]] = { + // if a group (key: uri) results in a list with 1 element, extract that single element + // otherwise, when a group has a list with more than 1 element, keep it as a list + val argResourcesGroupedByUri = paramTriples.groupBy(_.predicate).map { + pair => { + pair._2.length match { + case 0 => pair._1.uri -> None + case 1 => pair._1.uri -> pair._2.head + case _ => pair._1.uri -> pair._2 + } + } + } + + try { + // create Arguments + val arguments: Arguments = new Arguments(); + argResourcesGroupedByUri.foreach(argPair => { + val parameterName: String = argPair._1.toString; + val parameterValue: String = argPair._2.asInstanceOf[SerializableRDFQuad].`object`.value.value + arguments.add(parameterName, parameterValue) + }) + // execute the funtion using the function agent + val result = agent.execute(identifier, arguments) + Some(List(Literal(result.toString))) + } catch { + case e: Throwable => { + logError(s"The following exception occurred when invoking function ${identifier}: ${e.getMessage}." + + s"\nThe result will be set to None.", e) + None + } + } } } diff --git a/src/main/scala/io/rml/framework/core/function/model/FunctionMetaData.scala b/src/main/scala/io/rml/framework/core/function/model/FunctionMetaData.scala index 344d805c..e69de29b 100644 --- a/src/main/scala/io/rml/framework/core/function/model/FunctionMetaData.scala +++ b/src/main/scala/io/rml/framework/core/function/model/FunctionMetaData.scala @@ -1,20 +0,0 @@ -package io.rml.framework.core.function.model - -/** - * Contains metadata of a function that can be called by an RML Mapping. - * - * @param source string path of the source - * @param className class name containing the [[Function]] - * @param methodName method name of the [[Function]] - * @param inputParam [[List]] of input parameters for the [[Function]] - * @param outputParam [[List]] of expected output parameters from the [[Function]] - */ -case class FunctionMetaData(source: String, className: String, methodName: String, inputParam: List[Parameter], outputParam: List[Parameter]) extends { - - /** - * Used as an id for each inheriting objects for debugging ( maybe equivalence checking? e.g. in TermMap and TripleMap). - * - * @return - */ - def identifier: String = s"($source, $className, $methodName, $inputParam, $outputParam)" -} diff --git a/src/main/scala/io/rml/framework/core/function/model/std/StdRandomFunction.scala b/src/main/scala/io/rml/framework/core/function/model/std/StdRandomFunction.scala deleted file mode 100644 index e69de29b..00000000 diff --git a/src/main/scala/io/rml/framework/core/function/std/StdFunctionLoader.scala b/src/main/scala/io/rml/framework/core/function/std/StdFunctionLoader.scala deleted file mode 100644 index c0d45595..00000000 --- a/src/main/scala/io/rml/framework/core/function/std/StdFunctionLoader.scala +++ /dev/null @@ -1,95 +0,0 @@ -package io.rml.framework.core.function.std - -import io.rml.framework.core.function.model.{FunctionMetaData, Parameter} -import io.rml.framework.core.function.{FunctionLoader, FunctionUtils} -import io.rml.framework.core.model.rdf.{RDFGraph, RDFNode, RDFResource} -import io.rml.framework.core.model.{Literal, Uri} -import io.rml.framework.core.util.Util -import io.rml.framework.core.vocabulary.{DOAPVoc, FunVoc} -import io.rml.framework.shared.{FnOException, RMLException} - - -case class StdFunctionLoader private (functionDescriptionTriplesGraph : RDFGraph) extends FunctionLoader { - - override def parseFunctionMapping(graph: RDFGraph): FunctionLoader = { - logDebug("parsing functions the new way (i.e. using StdFunctionLoader)") - - val fnoFunctionProperty = Uri(FunVoc.FnO.Property.FNO_FUNCTION) - - // subject resources with fno:function property - // these resources have type fnoi:Mapping - val mappings = graph.filterProperties(fnoFunctionProperty) - if(mappings.isEmpty) - throw new RMLException("No function mappings found...") - - val functionDescriptionResources = this.functionDescriptionTriplesGraph.filterResources(Uri(FunVoc.FnO.Class.FNO_FUNCTION)) - logDebug(s"${functionDescriptionResources.length} functionDescriptionResources present") - logDebug(s"The current function description graph contains ${mappings.length} mappings") - for (map <- mappings) { - logDebug(s"Processing mapping: ${map.uri}") - try { - val functionUri = map.listProperties(FunVoc.FnO.Property.FNO_FUNCTION).head.asInstanceOf[RDFResource].uri - - val methodMappingResource = map.listProperties(FunVoc.FnO.Property.FNO_METHOD_MAPPING).head.asInstanceOf[RDFResource] - val methodNode = methodMappingResource.listProperties(FunVoc.FnOMapping.Property.FNOM_METHOD_NAME).head.asInstanceOf[Literal] - val methodName = methodNode.value - val implementationResource = map.listProperties(FunVoc.FnO.Property.FNO_IMPLEMENTATION).head.asInstanceOf[RDFResource] - - val className = Util.getLiteral(implementationResource.listProperties(FunVoc.FnoImplementation.Property.FNOI_CLASS_NAME).head) - val downloadPage = Util.getLiteral(implementationResource.listProperties(DOAPVoc.Property.DOAP_DOWNLOAD_PAGE).head) - logDebug(s"Found map with methodname: ${methodName}, className: ${className}, downloadPage: ${downloadPage}") - - // Get function description resource that corresponds with the current functionUri - // If not present, throw appropriate exception - val functionDescriptionResourceOption = functionDescriptionResources.find(fd => fd.uri == functionUri) - if(functionDescriptionResourceOption.isEmpty) - throw new FnOException(s"No function description resource found with uri: ${functionUri}") - - - // extraction of input parameters - val expectsResource = functionDescriptionResourceOption.get.listProperties(FunVoc.FnO.Property.FNO_EXPECTS).headOption - val inputParameterResources = expectsResource.get.asInstanceOf[RDFResource].getList.asInstanceOf[List[RDFResource]] - val inputParamList = parseParameterResources(inputParameterResources) - - // extraction of output parameters - val returnsResource = functionDescriptionResourceOption.get.listProperties(FunVoc.FnO.Property.FNO_RETURNS).headOption - val outputParameterResources = returnsResource.get.asInstanceOf[RDFResource].getList.asInstanceOf[List[RDFResource]] - val outputParamList = parseParameterResources(outputParameterResources) - - // construct function meta data object and store it in the functionMap - val functionMetaData = FunctionMetaData(downloadPage.get.value, className.get.value, methodName, inputParamList, outputParamList) - this.functionMap.put(functionUri, functionMetaData) - - }catch { - case e@(_: RMLException | _: FnOException) => - logError(e.getMessage) - } - } - this - } - - - def parseParameterResources(parameterResources : List[RDFResource]) : List[Parameter] = { - parameterResources.zipWithIndex.map{ - case (paramResource, paramIndex) => - parseParameter(paramResource, paramIndex) - } - } - - override def parseParameter(inputNode: RDFNode, pos: Int): Parameter = { - val inputResource = inputNode.asInstanceOf[RDFResource] - val paramType = inputResource.listProperties(FunVoc.FnO.Property.FNO_TYPE).headOption - val paramUri = inputResource.listProperties(FunVoc.FnO.Property.FNO_PREDICATE).headOption - - - if(paramType.isEmpty) - throw new FnOException(s"Parameter Type not defined for parameter resource: ${inputResource.uri}") - - if(paramUri.isEmpty) - throw new FnOException(s"Parameter Uri not defined for parameter resource: ${inputResource.uri}") - - val paramTypeResource = paramType.get.asInstanceOf[RDFResource] - val typeClass = FunctionUtils.getTypeClass(paramTypeResource.uri) - Parameter(typeClass, Uri(paramUri.get.identifier), pos) - } -} diff --git a/src/main/scala/io/rml/framework/core/util/ParameterUtil.scala b/src/main/scala/io/rml/framework/core/util/ParameterUtil.scala index c2538f2b..4e4f2e18 100644 --- a/src/main/scala/io/rml/framework/core/util/ParameterUtil.scala +++ b/src/main/scala/io/rml/framework/core/util/ParameterUtil.scala @@ -43,7 +43,8 @@ object ParameterUtil { partitionId: Option[Int] = None, socket: Option[String] = None, outputSink: OutputSinkOption = OutputSinkOption.File, - autoWatermarkInterval: Long = 50L + autoWatermarkInterval: Long = 50L, + functionDescriptionLocations: Option[Seq[String]] = None ) { override def toString: String = { val resultStr: String = @@ -60,7 +61,8 @@ object ParameterUtil { s"Kafka topic: ${topic.getOrElse("/")}\n" + s"Kafka topic partition id: ${partitionId.getOrElse("/")}\n" + s"Output TCP socket: ${socket.getOrElse("/")}\n" + - s"Discard output: ${outputSink.equals(OutputSinkOption.None)}" + s"Function description locations: ${functionDescriptionLocations}\n" + + s"Discard output: ${outputSink.equals(OutputSinkOption.None)}" resultStr } } @@ -122,12 +124,19 @@ object ParameterUtil { .text("If given, Flink's checkpointing is enabled with the given interval. " + "If not given, checkpointing is enabled when writing to a file (this is required to use the flink StreamingFileSink). " + "Otherwise, checkpointing is disabled.") + opt[Long]("auto-watermark-interval").valueName("