diff --git a/docs/_ldio/ldio-transformers/index.md b/docs/_ldio/ldio-transformers/index.md index c52a15ac8..406ca09cf 100644 --- a/docs/_ldio/ldio-transformers/index.md +++ b/docs/_ldio/ldio-transformers/index.md @@ -6,4 +6,8 @@ has_toc: true nav_order: 5 --- -# Linked Data Interactions Orchestrator Transformers \ No newline at end of file +# Linked Data Interactions Orchestrator Transformers + +The LDI Core module contains the components maintained by the VSDS team in order to accommodate the onboarding of LDES onboarders. + +Each component can be wrapped in a desired implementation framework (LDI-orchestrator, NiFi, ...) to be used. diff --git a/docs/_ldio/pipeline-management/index.md b/docs/_ldio/pipeline-management/index.md index 6fdd07db8..00e94e685 100644 --- a/docs/_ldio/pipeline-management/index.md +++ b/docs/_ldio/pipeline-management/index.md @@ -5,100 +5,4 @@ has_children: true has_toc: true nav_order: 1 --- - -# Management of Pipelines - -Pipelines in LDIO can be created in YAML or JSON configuration (although all example configurations are made in YAML, -these can also be formatted in JSON). - -A default pipeline looks as follows: - -```yaml - name: my-first-pipeline - input: - name: fully-qualified name of LDI Input - config: - foo: bar - adapter: - name: fully-qualified name of LDI Adapter - config: - foo: bar - transformers: - - name: fully-qualified name of LDI Transformer - config: - foo: bar - outputs: - - name: fully-qualified name of LDI Transformer - config: - foo: bar -``` - -- Note that one orchestrator can have multiple pipelines -- Note that one pipeline can have multiple LDI Transformers and LDI Outputs - -## Anatomy of a pipeline - -Each pipeline is built up of the following components: - -* [LDIO Input](ldi-inputs): A component that will receive data (not necessarily LD) to then feed the LDIO pipeline. -* [LDIO Adapter](ldi-adapters): To be used in conjunction with the LDIO Input, the LDIO Adapter will transform the - provided content into and internal Linked Data model and sends it down the pipeline. -* [LDIO Transformer](ldi-transformers): A component that takes in a Linked Data model, transforms/modifies it and then - puts it back on the pipeline. -* [LDIO Output](ldi-outputs): A component that will take in Linked Data and will export it to external sources. - -````mermaid -stateDiagram-v2 - direction LR - - LDI_Input --> LDI_Transformer : LD - LDI_Transformer --> LDI_Output : LD - - state LDI_Input { - direction LR - [*] --> LDI_Adapter : Non LD - - state LDI_Adapter { - direction LR - [*] --> adapt - adapt --> [*] - } - - LDI_Adapter --> [*] : LD - } - - state LDI_Transformer { - direction LR - [*] --> transform - transform --> [*] - } - state LDI_Output { - direction LR - [*] --> [*] - } -```` - -## Persistence of Pipelines - -By default, all pipelines defined after startup (via management API) will be lost on restart. - -To prevent this behaviour, add the `orchestrator.directory` property as follows: - -```yaml -orchestrator: - directory: "{directory in application folder}" -``` - -If this directory does not exist, it will be created. - -> **_NOTE:_** An application config can be defined by creating an application YAML file in the LDIO directory -(in docker, this correlates to `/ldio/application.yml`). - - -## Pausing & Resuming LDIO - -Sometimes it might be preferred to pause an LDIO pipeline instead of deleting and recreating it. -The endpoints to manage pipelines can be found [here](pipeline-api.md) - -The exact behaviour of a paused pipeline depends on its input component and can be found in the [documentation of these components](docs/_ldio/ldio-inputs/index.md). -However, it will always complete its current run through the pipeline and then seize sending any output. \ No newline at end of file +# Pipeline Management \ No newline at end of file diff --git a/docs/_ldio/pipeline-management/ldes-client-status.md b/docs/_ldio/pipeline-management/ldes-client-status.md new file mode 100644 index 000000000..0659c9ed6 --- /dev/null +++ b/docs/_ldio/pipeline-management/ldes-client-status.md @@ -0,0 +1,45 @@ +--- +layout: default +parent: Pipeline Management +title: LDES Client Status +nav_order: 4 +--- + +# LDES Client Status + +Just like the LDIO pipelines have a status, so does the [`Ldio:LdesClient`](../ldio-inputs/ldio-ldes-client). The client +status can be fetched when a pipeline that has a running status, and of course when it contains an LDES client as input +component. + +## Overview Of The Status Flow + +```mermaid +graph LR +; + REPLICATING --> SYNCHRONISING; + REPLICATING --> COMPLETED; + SYNCHRONISING --> COMPLETED; + SYNCHRONISING --> ERROR; + REPLICATING --> ERROR; +``` + +The above diagram shows the flow between the different statuses of the client. + +## REPLICATING + +The startup status of the client. This status indicates that the LDES client have not yet fetched all the available +fragments of a view (or views if so configured) + +## SYNCHRONISING + +This status indicates that all the fragments of the configured view(s) have been fetched at least once, and there is at +least one fragment that does not have an immutable state yet. + +## ERROR + +This status indicates that an error has occurred somewhere while `REPLICATING` or `SYNCHRONISING` + +## COMPLETED + +This status indicates that all the fragments of the configured view has been fetched at least once and all those have an +immutable state, or in other words, the end of the LDES has been reached. diff --git a/docs/_ldio/pipeline-management/management-of-pipelines.md b/docs/_ldio/pipeline-management/management-of-pipelines.md new file mode 100644 index 000000000..dd2be92bb --- /dev/null +++ b/docs/_ldio/pipeline-management/management-of-pipelines.md @@ -0,0 +1,103 @@ +--- +layout: default +parent: Pipeline Management +title: Management of Pipelines +nav_order: 1 +--- + +# Management of Pipelines + +Pipelines in LDIO can be created in YAML or JSON configuration (although all example configurations are made in YAML, +these can also be formatted in JSON). + +A default pipeline looks as follows: + +```yaml + name: my-first-pipeline + input: + name: name of LDI Input + config: + foo: bar + adapter: + name: name of LDI Adapter + config: + foo: bar + transformers: + - name: name of LDI Transformer + config: + foo: bar + outputs: + - name: name of LDI Transformer + config: + foo: bar +``` + +- Note that one orchestrator can have multiple pipelines +- Note that one pipeline can have multiple LDI Transformers and LDI Outputs + +## Anatomy of a pipeline + +Each pipeline is built up of the following components: + +* [LDIO Input](ldi-inputs): A component that will receive data (not necessarily LD) to then feed the LDIO pipeline. +* [LDIO Adapter](ldi-adapters): To be used in conjunction with the LDIO Input, the LDIO Adapter will transform the + provided content into and internal Linked Data model and sends it down the pipeline. +* [LDIO Transformer](ldi-transformers): A component that takes in a Linked Data model, transforms/modifies it and then + puts it back on the pipeline. +* [LDIO Output](ldi-outputs): A component that will take in Linked Data and will export it to external sources. + +````mermaid +stateDiagram-v2 + direction LR + + LDI_Input --> LDI_Transformer : LD + LDI_Transformer --> LDI_Output : LD + + state LDI_Input { + direction LR + [*] --> LDI_Adapter : Non LD + + state LDI_Adapter { + direction LR + [*] --> adapt + adapt --> [*] + } + + LDI_Adapter --> [*] : LD + } + + state LDI_Transformer { + direction LR + [*] --> transform + transform --> [*] + } + state LDI_Output { + direction LR + [*] --> [*] + } +```` + +## Persistence of Pipelines + +By default, all pipelines defined after startup (via management API) will be lost on restart. + +To prevent this behaviour, add the `orchestrator.directory` property as follows: + +```yaml +orchestrator: + directory: "{directory in application folder}" +``` + +If this directory does not exist, it will be created. + +> **_NOTE:_** An application config can be defined by creating an application YAML file in the LDIO directory +(in docker, this correlates to `/ldio/application.yml`). + + +## Pausing & Resuming LDIO + +Sometimes it might be preferred to pause an LDIO pipeline instead of deleting and recreating it. +The endpoints to manage pipelines can be found [here](pipeline-api.md) + +The exact behaviour of a paused pipeline depends on its input component and can be found in the [documentation of these components](docs/_ldio/ldio-inputs/index.md). +However, it will always complete its current run through the pipeline and then seize sending any output. \ No newline at end of file diff --git a/docs/_ldio/pipeline-management/openapi.json b/docs/_ldio/pipeline-management/openapi.json index 214b71f57..b0f232034 100644 --- a/docs/_ldio/pipeline-management/openapi.json +++ b/docs/_ldio/pipeline-management/openapi.json @@ -219,6 +219,70 @@ } } }, + "/admin/api/v1/pipeline/ldes-client": { + "get": { + "tags": [ + "LDES Client Status" + ], + "summary": "Get a list of all LDES Client statuses pipelines.", + "operationId": "getStatusses", + "responses": { + "200": { + "description": "A list statuses of all active LDES Client pipelines.", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "$ref": "#/components/schemas/ClientStatusTo" + } + } + } + } + } + } + } + }, + "/admin/api/v1/pipeline/ldes-client/{pipeline}": { + "get": { + "tags": [ + "LDES Client Status" + ], + "summary": "Get the status of a requested LDES Client pipeline.", + "operationId": "getPipelineStatus_2", + "parameters": [ + { + "name": "pipeline", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "404": { + "description": "No LDES Client pipeline exists by that name" + }, + "200": { + "description": "Status of a requested pipeline", + "content": { + "text/plain": { + "schema": { + "type": "string", + "enum": [ + "REPLICATING", + "SYNCHRONISING", + "COMPLETED", + "ERROR" + ] + } + } + } + } + } + } + }, "/admin/api/v1/catalog": { "get": { "tags": [ @@ -372,6 +436,23 @@ } } }, + "ClientStatusTo": { + "type": "object", + "properties": { + "pipeline": { + "type": "string" + }, + "status": { + "type": "string", + "enum": [ + "REPLICATING", + "SYNCHRONISING", + "COMPLETED", + "ERROR" + ] + } + } + }, "LdioCatalog": { "type": "object", "properties": { diff --git a/docs/_ldio/pipeline-management/pipeline-api.md b/docs/_ldio/pipeline-management/pipeline-api.md index b76e170d1..a65615e0d 100644 --- a/docs/_ldio/pipeline-management/pipeline-api.md +++ b/docs/_ldio/pipeline-management/pipeline-api.md @@ -2,6 +2,7 @@ layout: default parent: Pipeline Management title: Pipeline Management API +nav_order: 2 --- diff --git a/docs/_ldio/pipeline-management/pipeline-status.md b/docs/_ldio/pipeline-management/pipeline-status.md index 35c0a0020..4f0a6affc 100644 --- a/docs/_ldio/pipeline-management/pipeline-status.md +++ b/docs/_ldio/pipeline-management/pipeline-status.md @@ -2,24 +2,25 @@ layout: default parent: Pipeline Management title: Pipeline Status +nav_order: 3 --- -# Status +# Pipeline Status An individual ldio-pipeline can be in one of several different statuses. These different statuses and their behaviour are dependent on the input component of the pipeline. -# Overview Of The Status Flow +## Overview Of The Status Flow - -
-graph LR; +```mermaid +graph LR +; INIT --> RUNNING; INIT --> STOPPED; RUNNING --> STOPPED; RUNNING --> HALTED; HALTED --> RUNNING; HALTED --> STOPPED; -
+``` The above diagram shows the flow between the different statuses of the pipeline. diff --git a/docs/_ldio/pipeline-management/startup-config.md b/docs/_ldio/pipeline-management/startup-config.md index 62c992a41..e7b99ce71 100644 --- a/docs/_ldio/pipeline-management/startup-config.md +++ b/docs/_ldio/pipeline-management/startup-config.md @@ -4,7 +4,7 @@ parent: Pipeline Management title: Startup Configuration --- -### Startup Configuration +## Startup Configuration On startup, pipelines can be defined by creating an application YAML file in the LDIO directory (in docker, this correlates to `/ldio/application.yml`) that looks as follows: @@ -14,19 +14,54 @@ orchestrator: pipelines: - name: my-first-pipeline input: - name: fully-qualified name of LDI Input + name: name of LDI Input config: foo: bar adapter: - name: fully-qualified name of LDI Adapter + name: name of LDI Adapter config: foo: bar transformers: - - name: fully-qualified name of LDI Transformer + - name: name of LDI Transformer config: foo: bar outputs: - - name: fully-qualified name of LDI Transformer + - name: name of LDI Output config: foo: bar ```` + +### Since the introduction of dynamic pipelines + +Since version 2.1.0, it is possible to manage pipelines on the fly. If pipelines must be instantiated on startup, those +pipelines can be added to a configured directory. + +First of all, to configure the directory, the `/ldio/application.yml` should look like this: + +````yaml +orchestrator: + directory: +```` + +The folder contains a yaml file for each pipeline. It is the preferred way to call the file the same way as the name of +the pipeline, in this case, that would be `my-first-pipeline.yml`: + +```yaml +name: my-first-pipeline +input: + name: name of LDI Input + config: + foo: bar + adapter: + name: name of LDI Adapter + config: + foo: bar +transformers: + - name: name of LDI Transformer + config: + foo: bar +outputs: + - name: name of LDI Output + config: + foo: bar +``` diff --git a/ldi-orchestrator/ldio-connectors/ldio-ldes-client/src/main/java/be/vlaanderen/informatievlaanderen/ldes/ldio/management/status/ClientStatusController.java b/ldi-orchestrator/ldio-connectors/ldio-ldes-client/src/main/java/be/vlaanderen/informatievlaanderen/ldes/ldio/management/status/ClientStatusController.java index 0269d9261..7a61a29b8 100644 --- a/ldi-orchestrator/ldio-connectors/ldio-ldes-client/src/main/java/be/vlaanderen/informatievlaanderen/ldes/ldio/management/status/ClientStatusController.java +++ b/ldi-orchestrator/ldio-connectors/ldio-ldes-client/src/main/java/be/vlaanderen/informatievlaanderen/ldes/ldio/management/status/ClientStatusController.java @@ -26,7 +26,7 @@ public ClientStatusController(ClientStatusService clientStatusService) { this.clientStatusService = clientStatusService; } - @GetMapping() + @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) @ApiResponse(responseCode = "200", description = "A list statuses of all active LDES Client pipelines.", content = { @Content(mediaType = MediaType.APPLICATION_JSON_VALUE, array = @ArraySchema(schema = @Schema(implementation = ClientStatusTo.class))), }) @@ -35,9 +35,9 @@ public List getStatusses() { return clientStatusService.getClientStatuses(); } - @GetMapping(path = "{pipeline}", produces = "application/json") + @GetMapping(path = "{pipeline}", produces = MediaType.TEXT_PLAIN_VALUE) @ApiResponse(responseCode = "200", description = "Status of a requested pipeline", content = { - @Content(mediaType = MediaType.APPLICATION_JSON_VALUE, schema = @Schema(implementation = ClientStatus.class)), + @Content(mediaType = MediaType.TEXT_PLAIN_VALUE, schema = @Schema(implementation = ClientStatus.class)), }) @ApiResponse(responseCode = "404", description = "No LDES Client pipeline exists by that name", content = { @Content(schema = @Schema()),