Spsh-1553: Implemented the new async workflow for the import & refact…

…oring (#834)

* SPSH-1286: Implemented the repository, entity & aggregate for the history of imports and import transactions updates.

* SPSH-1286: Refactored the import workflow to update the status of importvorgang.

* SPSH-1286: Implemented unit tests and integration tests

* SPSH-1286: Fixed test coverage

* SPSH-1286: Fixed code coverage

* SPSH-1286: Added a reference to the ImportDataItem table and refactored the integration tests.

* First draft in progress

* SPSH-1553: Workaround for import-workflow

* SPSH-1553: Fixed issues with the import-event.handler and added migration file

* SPSH-1553: Fixed issued after merge with main

* SPSH-1553: More code refactoring and foxed tests for the new import workflow.

* SPSH-1553: Fixed the config loader tests

* SPSH-1553: Fixed integration tests

* SPSH-1553: Fixed code coverage

* SPSH-1553: Begrenzung der Anzahl der Nutzer beim Import über die Konfiguration.

* SPSH-1553: Fixed issue with last empty line in the CSV file when the max number of users is checked

* SPSH-1553: PR Review

* SPSH-1553: Fixed test coverage

* SPSH-1553: Merged two migration scripts

* SPSH-1553: Reversed mistake in the values.yaml

* SPSH-1553: Added chart configuration and renamed import config variables for the pipeline

* SPSH-1553: Changed the response type for the import status

---------

Co-authored-by: Caspar Neumann <[email protected]>
Co-authored-by: Youssef Bouchara <[email protected]>

charts/dbildungs-iam-server/config/config.json

@@ -75,7 +75,10 @@
     "REALM": "defrealm"
   },
   "IMPORT": {
-    "IMPORT_FILE_MAXGROESSE_IN_MB": 10
+    "CSV_FILE_MAX_SIZE_IN_MB": 10,
+    "CSV_MAX_NUMBER_OF_USERS": 2000,
+    "PASSPHRASE_SECRET": "44abDqJk2qgwRbpGfO0VZx7DpXeFsm7R",
+    "PASSPHRASE_SALT": "YDp6fYkbUcj4ZkyAOnbAHGQ9O72htc5M"
   },
   "SYSTEM": {
     "RENAME_WAITING_TIME_IN_SECONDS": 3,

charts/dbildungs-iam-server/templates/configmap.yaml

@@ -29,3 +29,5 @@ data:
   ITSLEARNING_ROOT_OEFFENTLICH: '{{ .Values.itslearning.rootOeffentlich }}'
   ITSLEARNING_ROOT_ERSATZ: '{{ .Values.itslearning.rootErsatz }}'
   NODE_OPTIONS: "--max-old-space-size={{ .Values.backend.env.maxOldSpaceSize }}"
+  IMPORT_CSV_FILE_MAX_SIZE_IN_MB: '{{ .Values.import.csvFileMaxSizeInMB }}'
+  IMPORT_CSV_MAX_NUMBER_OF_USERS: '{{ .Values.import.csvMaxNumberOfUsers }}'

charts/dbildungs-iam-server/templates/secret.yaml

@@ -30,4 +30,6 @@ data:
   vidis-region-name: {{ .Values.auth.vidis_region_name }}
   vidis-keycloak-group: {{ .Values.auth.vidis_keycloak_group }}
   vidis-keycloak-role: {{ .Values.auth.vidis_keycloak_role }}
+  import-passphrase-secret: {{ .Values.auth.import_passphrase_secret }}
+  import-passphrase-salt: {{ .Values.auth.import_passphrase_salt }}
 {{- end }}

charts/dbildungs-iam-server/values.yaml

@@ -64,6 +64,8 @@ auth:
   vidis_region_name: ''
   vidis_keycloak_group: ''
   vidis_keycloak_role: ''
+  import_passphrase_secret: ''
+  import_passphrase_salt: ''
 
 backend:
   replicaCount: 1
@@ -193,3 +195,7 @@ cronjobs:
       endpoint: '/api/cron/unlock'
       httpMethod: 'PUT'
       script: 'cron_trigger.sh'
+
+import:
+  csvFileMaxSizeInMB: 10
+  csvMaxNumberOfUsers: 2000

config/config.json

@@ -95,7 +95,10 @@
         "KEYCLOAK_ROLE": "VIDIS-user"
     },
     "IMPORT": {
-        "IMPORT_FILE_MAXGROESSE_IN_MB": 10
+        "CSV_FILE_MAX_SIZE_IN_MB": 10,
+        "CSV_MAX_NUMBER_OF_USERS": 200,
+        "PASSPHRASE_SECRET": "44abDqJk2qgwRbpGfO0VZx7DpXeFsm7R",
+        "PASSPHRASE_SALT": "YDp6fYkbUcj4ZkyAOnbAHGQ9O72htc5M"
     },
     "SYSTEM": {
         "RENAME_WAITING_TIME_IN_SECONDS": 3,