chore: ldes client status

docs/_ldio/ldio-transformers/index.md

@@ -6,4 +6,8 @@ has_toc: true
 nav_order: 5
 ---
 
-# Linked Data Interactions Orchestrator Transformers
+# Linked Data Interactions Orchestrator Transformers
+
+The LDI Core module contains the components maintained by the VSDS team in order to accommodate the onboarding of LDES onboarders.
+
+Each component can be wrapped in a desired implementation framework (LDI-orchestrator, NiFi, ...) to be used.

docs/_ldio/pipeline-management/index.md

@@ -5,100 +5,4 @@ has_children: true
 has_toc: true
 nav_order: 1
 ---
-
-# Management of Pipelines
-
-Pipelines in LDIO can be created in YAML or JSON configuration (although all example configurations are made in YAML,
-these can also be formatted in JSON).
-
-A default pipeline looks as follows:
-
-```yaml
-  name: my-first-pipeline
-  input:
-    name: fully-qualified name of LDI Input
-    config:
-      foo: bar
-    adapter:
-      name: fully-qualified name of LDI Adapter
-      config:
-        foo: bar
-  transformers:
-    - name: fully-qualified name of LDI Transformer
-      config:
-        foo: bar
-  outputs:
-    - name: fully-qualified name of LDI Transformer
-      config:
-        foo: bar
-```
-
-- Note that one orchestrator can have multiple pipelines
-- Note that one pipeline can have multiple LDI Transformers and LDI Outputs
-
-## Anatomy of a pipeline
-
-Each pipeline is built up of the following components:
-
-* [LDIO Input](ldi-inputs): A component that will receive data (not necessarily LD) to then feed the LDIO pipeline.
-* [LDIO Adapter](ldi-adapters): To be used in conjunction with the LDIO Input, the LDIO Adapter will transform the
-  provided content into and internal Linked Data model and sends it down the pipeline.
-* [LDIO Transformer](ldi-transformers): A component that takes in a Linked Data model, transforms/modifies it and then
-  puts it back on the pipeline.
-* [LDIO Output](ldi-outputs): A component that will take in Linked Data and will export it to external sources.
-
-````mermaid
-stateDiagram-v2
-    direction LR
-
-    LDI_Input --> LDI_Transformer : LD
-    LDI_Transformer --> LDI_Output : LD
-
-    state LDI_Input {
-        direction LR
-        [*] --> LDI_Adapter : Non LD
-
-        state LDI_Adapter {
-            direction LR
-            [*] --> adapt
-            adapt --> [*]
-        }
-
-        LDI_Adapter --> [*] : LD
-    }
-
-    state LDI_Transformer {
-        direction LR
-        [*] --> transform
-        transform --> [*]
-    }
-    state LDI_Output {
-        direction LR
-        [*] --> [*]
-    }
-````
-
-## Persistence of Pipelines
-
-By default, all pipelines defined after startup (via management API) will be lost on restart.
-
-To prevent this behaviour, add the `orchestrator.directory` property as follows:
-
-```yaml
-orchestrator:
-  directory: "{directory in application folder}"
-```
-
-If this directory does not exist, it will be created.
-
-> **_NOTE:_**  An application config can be defined by creating an application YAML file in the LDIO directory
-(in docker, this correlates to `/ldio/application.yml`).
-
-
-## Pausing & Resuming LDIO
-
-Sometimes it might be preferred to pause an LDIO pipeline instead of deleting and recreating it.
-The endpoints to manage pipelines can be found [here](pipeline-api.md)
-
-The exact behaviour of a paused pipeline depends on its input component and can be found in the [documentation of these components](docs/_ldio/ldio-inputs/index.md).
-However, it will always complete its current run through the pipeline and then seize sending any output.
+# Pipeline Management

docs/_ldio/pipeline-management/ldes-client-status.md

@@ -0,0 +1,45 @@
+---
+layout: default
+parent: Pipeline Management
+title: LDES Client Status
+nav_order: 4
+---
+
+# LDES Client Status
+
+Just like the LDIO pipelines have a status, so does the [`Ldio:LdesClient`](../ldio-inputs/ldio-ldes-client). The client
+status can be fetched when a pipeline that has a running status, and of course when it contains an LDES client as input
+component.
+
+## Overview Of The Status Flow
+
+```mermaid
+graph LR
+;
+    REPLICATING --> SYNCHRONISING;
+    REPLICATING --> COMPLETED;
+    SYNCHRONISING --> COMPLETED;
+    SYNCHRONISING --> ERROR;
+    REPLICATING --> ERROR;
+```
+
+The above diagram shows the flow between the different statuses of the client.
+
+## REPLICATING
+
+The startup status of the client. This status indicates that the LDES client have not yet fetched all the available
+fragments of a view (or views if so configured)
+
+## SYNCHRONISING
+
+This status indicates that all the fragments of the configured view(s) have been fetched at least once, and there is at
+least one fragment that does not have an immutable state yet.
+
+## ERROR
+
+This status indicates that an error has occurred somewhere while `REPLICATING` or `SYNCHRONISING`
+
+## COMPLETED
+
+This status indicates that all the fragments of the configured view has been fetched at least once and all those have an
+immutable state, or in other words, the end of the LDES has been reached.

docs/_ldio/pipeline-management/management-of-pipelines.md

@@ -0,0 +1,103 @@
+---
+layout: default
+parent: Pipeline Management
+title: Management of Pipelines
+nav_order: 1
+---
+
+# Management of Pipelines
+
+Pipelines in LDIO can be created in YAML or JSON configuration (although all example configurations are made in YAML,
+these can also be formatted in JSON).
+
+A default pipeline looks as follows:
+
+```yaml
+  name: my-first-pipeline
+  input:
+    name: name of LDI Input
+    config:
+      foo: bar
+    adapter:
+      name: name of LDI Adapter
+      config:
+        foo: bar
+  transformers:
+    - name: name of LDI Transformer
+      config:
+        foo: bar
+  outputs:
+    - name: name of LDI Transformer
+      config:
+        foo: bar
+```
+
+- Note that one orchestrator can have multiple pipelines
+- Note that one pipeline can have multiple LDI Transformers and LDI Outputs
+
+## Anatomy of a pipeline
+
+Each pipeline is built up of the following components:
+
+* [LDIO Input](ldi-inputs): A component that will receive data (not necessarily LD) to then feed the LDIO pipeline.
+* [LDIO Adapter](ldi-adapters): To be used in conjunction with the LDIO Input, the LDIO Adapter will transform the
+  provided content into and internal Linked Data model and sends it down the pipeline.
+* [LDIO Transformer](ldi-transformers): A component that takes in a Linked Data model, transforms/modifies it and then
+  puts it back on the pipeline.
+* [LDIO Output](ldi-outputs): A component that will take in Linked Data and will export it to external sources.
+
+````mermaid
+stateDiagram-v2
+    direction LR
+
+    LDI_Input --> LDI_Transformer : LD
+    LDI_Transformer --> LDI_Output : LD
+
+    state LDI_Input {
+        direction LR
+        [*] --> LDI_Adapter : Non LD
+
+        state LDI_Adapter {
+            direction LR
+            [*] --> adapt
+            adapt --> [*]
+        }
+
+        LDI_Adapter --> [*] : LD
+    }
+
+    state LDI_Transformer {
+        direction LR
+        [*] --> transform
+        transform --> [*]
+    }
+    state LDI_Output {
+        direction LR
+        [*] --> [*]
+    }
+````
+
+## Persistence of Pipelines
+
+By default, all pipelines defined after startup (via management API) will be lost on restart.
+
+To prevent this behaviour, add the `orchestrator.directory` property as follows:
+
+```yaml
+orchestrator:
+  directory: "{directory in application folder}"
+```
+
+If this directory does not exist, it will be created.
+
+> **_NOTE:_**  An application config can be defined by creating an application YAML file in the LDIO directory
+(in docker, this correlates to `/ldio/application.yml`).
+
+
+## Pausing & Resuming LDIO
+
+Sometimes it might be preferred to pause an LDIO pipeline instead of deleting and recreating it.
+The endpoints to manage pipelines can be found [here](pipeline-api.md)
+
+The exact behaviour of a paused pipeline depends on its input component and can be found in the [documentation of these components](docs/_ldio/ldio-inputs/index.md).
+However, it will always complete its current run through the pipeline and then seize sending any output.

docs/_ldio/pipeline-management/openapi.json

@@ -219,6 +219,70 @@
         }
       }
     },
+    "/admin/api/v1/pipeline/ldes-client": {
+      "get": {
+        "tags": [
+          "LDES Client Status"
+        ],
+        "summary": "Get a list of all LDES Client statuses pipelines.",
+        "operationId": "getStatusses",
+        "responses": {
+          "200": {
+            "description": "A list statuses of all active LDES Client pipelines.",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "array",
+                  "items": {
+                    "$ref": "#/components/schemas/ClientStatusTo"
+                  }
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/admin/api/v1/pipeline/ldes-client/{pipeline}": {
+      "get": {
+        "tags": [
+          "LDES Client Status"
+        ],
+        "summary": "Get the status of a requested LDES Client pipeline.",
+        "operationId": "getPipelineStatus_2",
+        "parameters": [
+          {
+            "name": "pipeline",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string"
+            }
+          }
+        ],
+        "responses": {
+          "404": {
+            "description": "No LDES Client pipeline exists by that name"
+          },
+          "200": {
+            "description": "Status of a requested pipeline",
+            "content": {
+              "text/plain": {
+                "schema": {
+                  "type": "string",
+                  "enum": [
+                    "REPLICATING",
+                    "SYNCHRONISING",
+                    "COMPLETED",
+                    "ERROR"
+                  ]
+                }
+              }
+            }
+          }
+        }
+      }
+    },
     "/admin/api/v1/catalog": {
       "get": {
         "tags": [
@@ -372,6 +436,23 @@
           }
         }
       },
+      "ClientStatusTo": {
+        "type": "object",
+        "properties": {
+          "pipeline": {
+            "type": "string"
+          },
+          "status": {
+            "type": "string",
+            "enum": [
+              "REPLICATING",
+              "SYNCHRONISING",
+              "COMPLETED",
+              "ERROR"
+            ]
+          }
+        }
+      },
       "LdioCatalog": {
         "type": "object",
         "properties": {

docs/_ldio/pipeline-management/pipeline-api.md

@@ -2,6 +2,7 @@
 layout: default
 parent: Pipeline Management
 title: Pipeline Management API
+nav_order: 2
 ---
 
 <html>