From 2d3a698bedbea8f30b2ac4d198fcd7def5a7c8bb Mon Sep 17 00:00:00 2001 From: "igor.balog" Date: Wed, 26 Jul 2023 14:25:46 +0200 Subject: [PATCH 01/33] OCSP and Audit readme update --- doc/advancedConfiguration/auditlogs.md | 27 ++++++++++++++++++++++++++ doc/advancedConfiguration/ocsp.md | 7 ++++++- 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/doc/advancedConfiguration/auditlogs.md b/doc/advancedConfiguration/auditlogs.md index a3dee9a..1036bd1 100644 --- a/doc/advancedConfiguration/auditlogs.md +++ b/doc/advancedConfiguration/auditlogs.md @@ -1,3 +1,30 @@ ### Audit logs Audit logging is turned **off** by default. If you wish to configure it or even turn off please follow this [document](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.1/doc/AUDIT.md) . + + +## Accessing audit logs + +Access to the audit logs should be allowed only to the person responsible for configuring and setting up TRUE Connector. Lods are stored in docker volumes, and in default configuration those docker volumes are: + +ecc_provider_log +ecc_consumer_log + +Once audit events are turned on, and docker containers are up and running, you can inspect the from the terminal, and access the one for the Excecution Core Container by executing command: + +``` +docker exec -it ecc-consumer /bin/sh +``` + +Once you manage to log into container, you can navigate to */var/log/ecc* and verify that audit tile is present. File might be empty, if there are no actions performed so far, so be sure to have some interaction with the connector, to verify that file is being updated. + +``` +/var/log/ecc $ ls -la +total 12 +drwxr-xr-x 2 nobody nogroup 4096 Jul 25 16:01 . +drwxr-xr-x 1 root root 4096 Jul 19 09:47 .. +-rw-r--r-- 1 nobody nobody 1578 Jul 25 16:09 true_connector_audit_consumer.log +/var/log/ecc $ +``` + +Content of the file is consisted of json entries. \ No newline at end of file diff --git a/doc/advancedConfiguration/ocsp.md b/doc/advancedConfiguration/ocsp.md index 54b6873..97345fb 100644 --- a/doc/advancedConfiguration/ocsp.md +++ b/doc/advancedConfiguration/ocsp.md @@ -9,5 +9,10 @@ OCSP functionality is disabled by default, in order to enable it, set following application.OCSP_RevocationCheckValue=none ``` +| **Property value** | **Description** | +|:---:|:------------| +| *good* | means that OCSP must be supported and certificate is not revoked | +| *unknown* | OCSP check enabled and enforced, but certificate itself does not have OCSP check supported (no URL for checking revoke status). Check will pass in case of certificate not revoker or if cannot be determined. | +| *none* | OCSP check disabled | -When enabled, it will perform remote environemnt certificate check prior to sending request to provider connector. \ No newline at end of file +When enabled, it will perform remote environment certificate check prior to sending request to provider connector. \ No newline at end of file From fe42e932b329344454bd35d0503afa350bd9ab1b Mon Sep 17 00:00:00 2001 From: David Jovanovic Date: Wed, 26 Jul 2023 16:07:28 +0200 Subject: [PATCH 02/33] docs for backup and update --- doc/backup.md | 39 +++++++++++++++++++++++++++++++++++++++ doc/update.md | 3 +++ 2 files changed, 42 insertions(+) create mode 100644 doc/backup.md create mode 100644 doc/update.md diff --git a/doc/backup.md b/doc/backup.md new file mode 100644 index 0000000..6496d27 --- /dev/null +++ b/doc/backup.md @@ -0,0 +1,39 @@ +# Backup & restore data + +To backup and restore data we will be using the official Docker documentation as a reference, found [here](https://docs.docker.com/storage/volumes/#back-up-restore-or-migrate-data-volumes). + +## Backup + +In order to backup your data you can use the following code snippet: + +``` +docker run --rm --volumes-from ecc-consumer -v %cd%:/backup ubuntu tar cvf /backup/backup.tar /var/log/ecc +``` + +Let me explain the options: + + - ecc-consumer - container with the data that you want to backup + - %cd% - current directory in the command prompt ( same as $(pwd) on Linux); the directory where you want the backup to be located + - /backup.tar - name of the backup archive + - /var/log/ecc - directory or file from the container that you wish to backup + +After the process finishes you will find a .tar file with the data. + +## Restore + +Restoring the data is done in a similar way: + +``` +docker run --rm --volumes-from ecc-consumer -v %cd%:/backup ubuntu bash -c "cd /var && tar xvf /backup/backup.tar --strip 1" +``` + + The options are: + + - ecc-consumer - container where you want to restore the data + - %cd% - current directory in the command prompt ( same as $(pwd) on Linux); the directory where the backup is located + - /var - directory where the data will be restored + - /backup.tar - name of the backup archive + + **NOTE** + + If the backup hierarchy looks like this */var/log/ecc* and you wish to keep it when restoring then you have to point the root folder in the command */var*, as it was done in the commands from above. \ No newline at end of file diff --git a/doc/update.md b/doc/update.md new file mode 100644 index 0000000..eefaed2 --- /dev/null +++ b/doc/update.md @@ -0,0 +1,3 @@ +# Updating the TRUE Connector + +To update the TRUE Connector to a newer version you have to download and reconfigure it for your environment as it will always be as a fresh start. To migrate your data to the new version please follow the [backup & restore guide](backup.md) . \ No newline at end of file From e2f08f00419b1977dc4b8ddeb58f935e1bee96f2 Mon Sep 17 00:00:00 2001 From: David Jovanovic Date: Thu, 27 Jul 2023 15:21:42 +0200 Subject: [PATCH 03/33] added backup and update readmes --- .env | 2 ++ README.md | 2 ++ doc/backupAndUpdate/assets.png | Bin 0 -> 6759 bytes doc/{ => backupAndUpdate}/backup.md | 0 doc/backupAndUpdate/update.md | 52 ++++++++++++++++++++++++++++ doc/update.md | 3 -- 6 files changed, 56 insertions(+), 3 deletions(-) create mode 100644 doc/backupAndUpdate/assets.png rename doc/{ => backupAndUpdate}/backup.md (100%) create mode 100644 doc/backupAndUpdate/update.md delete mode 100644 doc/update.md diff --git a/.env b/.env index 1e3219a..86c32da 100644 --- a/.env +++ b/.env @@ -1,3 +1,5 @@ +COMPOSE_PROJECT_NAME=TRUE_Connector + BROKER_URL=https://broker.ids.isst.fraunhofer.de/infrastructure DISABLE_SSL_VALIDATION=true diff --git a/README.md b/README.md index 4d8a10d..ad11c4f 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,8 @@ Please [click here](https://engineering-ing-inf-rd.gitbook.io/true-connector/), * [Self Description API](doc/selfDescriptionAPI/self-description-API.md) * [Changing API password](doc/selfDescriptionAPI/changepassword.md) * [Postman collection](doc/selfDescriptionAPI/postman.md) + * [Backup & restore](doc/backupAndUpdate/backup.md) + * [Updating](doc/backupAndUpdate/update.md) * [Docker image signing and verification](doc/cosign.md) * [Life cycle](doc/life_cycle.md) * [Security](doc/security.md) diff --git a/doc/backupAndUpdate/assets.png b/doc/backupAndUpdate/assets.png new file mode 100644 index 0000000000000000000000000000000000000000..492f17e9da8463412298cde2469cd36ef272f9d7 GIT binary patch literal 6759 zcmbuEXIN89yZ4s`eE`v|C@4r#5Tq+r>4-ETfOH51HxMbI1_*>|1p(;-(p8EA0zx2^ zP;^s56Qm?DK!TJAAwZ;rP{YA}_Vczc=RMc?ux8Dyxvy)@nz`5g`_Dab=B5T*9DE!A z0B{)@-n9ULBcq4)qJNz@EQc#Lz8~I>1X>u}0jdTsQx6-*-E}}Z08o?6x%VgAVVm9G z&^{0V&ISFxj*uS;g#y5(J)^riR-uq(5?9c9O4H}n*aV|TK)jvO`L`Dy3+p9%QpV6c zR^~3+GVjqgat_C3UemCidOit|T=aFE$W>hVV;gBgx!%Wb!dH36%2n{M<4d0Wi{~SI znECO?Dc+v+U3CRiHpyF4nfu)L^@IHNMg#|RPvHyk6m;x1@g{WBJ{j(3-kv<-ge`JJ$Hl8?|=vrA#~bV>~`6Z~SX|8*fx zzh_v)tSisDRPadw9jt1R*zgURgjGwxXZUbV+ey=8yMwB{yaQx3{hN8@!E^YI6RkD> zPeO0Mv3QT=?9eNsA2K$n%`LShY@&>8`sDT4N2IoI0Rucyv9)Vj7QEdEV|Gxsh832- zcLWZlM>U_F5QD9>cO~t6PZvVh#dqD>gECeOPSv@(PdSJ&lZH^spAD?rGqY7Rge&&p z);2@4(1FB|eka~DsVn3wsSQzA++R8im#G!+_A@i9>MC<%l*Vjn<$p17&5{*<;=}BB zsX;V!cXSZ-YHK)UCmi^uTKAu;pOaa+);StgG1e=jFjzmZHuvOR#KDW6STor)YZl_W zf$=@JrB}{lRfrLi zabkL-LtULZ`0|o79icZDb5SJ`qY&y&(;YS2*-ztNEi~8p;pFA$;AIydel`2nxep@A zuSA+da2jSdW>i{|=SJVT1pmnTs$K?;FX-nz;pHSBP{oE>LzI!}ZCrxU8NYguq|9FU zC4k!Z!I!KtG?YO8zSZfK-=ifBI8wsgxew|X6 zYu_q&wy6y9f#EmP?u)EX?luQv2Wwnuo6L}T)?oewVbgl)t34e5oU^jc!4}F0CO1dE zd?~dX+n}(B*bXOdlhG9w9E5+?q$Rfv%_ihC*zl9%59*~O;20mSTOlo7Hzq8%;jE^8nK2eGZ~rWN8vyfhuV*4&<%K0mujiU;=#zYtt6$Z(LixY}`FRlx4$OUBSe`QKj$pe9 z7Da&SK-; z%(!M5)60M@rOa)}y&CRF`h~E#gRvRtvZqEkjAfqruM<5dlYmv5dmW?6u2f6?el{IfYY~4w2%Jw z?>3ccSZk*v?cMTnLM42tG0>Z~l32w+jLWR;95_z?)i@_>zmY#mF{p0dB)iL^MG~Bw zOf|QALn!$7D#nR02}vKWhm3*rJydm?nsGZ68?*S0I*5Z$6kFKOI6S@welp%Qj+%OK z+dYTiwdc77!OUbdOb>%8Cz%J;AFIg%bV2m)LX_vbSCd=LaMrgu{y!o_rJ`>1qT3Fz zUzGcijed2~f0ydT)>M%JH@babwOiNky+H(F!m>anX#?@!oI=~X^qd#}RRk9s5w zn3nc<$RHT#$B5ODR!(83p!J!Q5+gZr%>KE;OuegY-*`kVFOM{b^w)@9R4QAf)k^~& zod4*VfSysFl9;5B6Dp1#TpG5{MLHGdwa<%Mx}6fBk7YOJTxJY?@O+kD;R{;`&@dRn z5pI}r2unU(|1FVZ{wHebzagytZ`0uIhFQOm#{Z?~)NW|*g}#rd4Q%kCS0o#WLu3JA zU8uKwWyQgJZ{)$T-rnA{qMZ%2YZD1`NDyFI!WDy5N)-$UHM{hk?qU2+)B^bYI4z*p z3KOhUrx&)78oO(71i0n@`qA-2Nq8yCB)B$J|M0U9tuha-+?Tv==yL!I*MIlG<@=oh zKp^hs|1%ZxE2j+|N>$B6Y{Hsml#olkC1@uwW=PQTl9unw1k}C;pe-hnT}mCl)X*2O zzwRC0!~S$>RSrfV7O7H-r`O24xV7%w==4zop7EM~xuzb$BS3bXFIf?{GU0Ay<$7oM zV!{nld)Oj$jWWvNB!wQCQArSF7P0|u|5mCKUu0CAc#{h!H4X03UyDq&Zd`@spO6`Q z-*3C)ehiT3>a7#7sywt<)pIQl%_A@>*sW*h&WiWdx)-2a;}+c@RDJ4A4T+!hwL$}! zx8#(|unP7wDa!U`!S6JIO{i>KcGD4yo| z6B~&xY`=guzE|p(acRVW+~W7C3zBd|ej<13nQ4vj3M3P{IyO+$nd^doc9cn9bfHHc zD-lhMM%CD!N{P#ukRBJ=KDH1cI*Lds&u!3CV!R7{B+eV|{GH2r{I-QfCum2lzjRHY zwyLqZ%kIE#w?8%Dxl2uBJYFJ&{U7P>a}t!8jqZK|B{w>vTm$jf^3>5eil(+GCwE_0uN|X~v@@fHUKCr}vXd_X!X6AXv9MLlQ zthN>2K`0xdDaBNe4oZ>!ynnV@S13e=0qK0+$2qu8BStvU+#*4lr+T?LU0vTrMJheq zOigvbO64rZPq_ws*&xYZ3H>@Iop7n-twQa`)Y)*kxj| zh*phQ1krXmlWivL+pEqRT#ep}vD2c@KqY9__j#oEh)KlEOW5*Pcx7#B7uKOOnRf$y z7d9KKpzsz+Bi={cQ^KmcGUVC2= zHI)#_$&OzwmlLd7nmvqfF0_sLS8CCzV~$PJwU(YMd>sVh9Mb4Xm>3>c;f@OS=`LOK z;Yr|3kNQI~4SLp($R`(YnkrsfRm!Cm0bJhr$NLfu!6hN2-8Zm|syaKyp7BUgxXJ-B zh?>T}EVj|iQanNNG!|!z_5@)^Wlf@`Z(If?T|s|rf_d)-pCDCwjwva|WZy`#x0Vr+ z43z1Mw`i#1_o%qlM8ngrMqyMZRhda_u64j^AJNel%HEUS$!7*zJMOSyb0}?Z_=lu% z#A4aUbCkYYOty;YL-&m8O4MzL_U>uY1P3p*jQhF#6H;#fi;-` zC)HTT!8sddpST2;BKJXss=X^s6%8Dx&QUW6lh?cE3w)wA^t1XtSi@MN^!*}-mF&pe zWqv34YRcfA$vXu-ei9N}s7IqC;S`Bz9$&{I`9LTlz~tcqDPpq8%iBD25biT}XLN$! zx4!bys@qI!axHTNDxsQ-PdaWfR&us_DF$x66%1K3Y8iCg^}9n{;j``t4kLVcD9uec z$Dr#Ma~I}yHkV$Pi<7;Ua}_(j>6P`l)xHOv+oP%Fhw6N|;+4HaTtT~Xm&QB(QC`H+ zUfq^F!W>~9@%K0k?nVc+QmHqW_t~u9>2S$yxP7e^eD>(ATf<*D4;fTvA~IsLxDVyr*BGjkNOuUC zJ#qQf6Ir`_|FOpY*4ve^Y%jv<$Ce{#5U*UyCBsr}jHi!^+#?g+zcUxz|Nr?7=n{y( znX)j2Iu88!JbT*kH+fa${(Cu+fRn|Ft&XcX>YH;G6IGQY_vEsdLMq)|M3x{RoStej z8npZpOlf2x(He8lbT+rL>S;^&^Wh`@Ncz4Pa`tO`FI9wd`8y@c(JdKxH1O~Dk6K~; z3hND(#Lg$y{A+Z-;ypynS+mj{QuU#3>+3u0zWX5{UmQkQ-zI8v=-%AsZHqAMvJPS#6-dlgE~fUud|%B3wyN>7^}Ez5@qNR4%IEi1 zaHk07g~vvBmJwx(j6)3d3SimIJSkp=AA0L9>^_$M#Z)KChyT=;V0WmALRzS6+wf9( zD-5tKeGe&`rr#vO+uY{QDwC>7S1gZVl}2^1E|xULma+LGbVx$3NwKQGrm`TDKR*&-lE| zhqc2i#iWfLs-%sJ2T@8yo>keMGJE+VZ;W0?dr`%Ke9d*#y2UjMP_s}Y{ujVWZ$4od zRls6B?eaVmi*k>XtiM1N?H#d+-xTuz# zn4bbW`-1r)$P`n_!StuHssAT2@&3*^3MawG$_Tl1_r(EV%Gm=H@TTR zcAjEaJcHB2ICaom_`5|d;gpzu(~_p&WJ;gE#=e+sXIowH6|dyPI`e$ks8DYv+|8I( z<1#F|A2DJ$liSx|FD+}b6L9p+Xo`u*_@@Ar#^T(&2T$(%OGm2mTY~K%{dZi=A(>1o z`DkT-Y+z;UuSWMjrUNjFN+F3p?WX&eM?Y4G`LYXo{+3;XuJ*Cyt#_^!~BYV z)`>OA2%kEe(ZmCl5VufzxqEcsmguNrFdYwKtwts489$-O`?aHpkgU&<63T60P`)%F zP7&E8KNxw*E0mO8^_*hn#ojhoD8qah(~UW?&*P{^dH;4j=&M_(i>U&~PB*wqr~msQ zwj0t;USO;)S`>_d9O_%CM&?zM{)bMR;f(N4oTgcY9Tu0ZQeaMPd|bi5s%}Q)>uB)f za5xmPIi}eVi~}t;eW+<^$7U^Da>0(+=!~BU*yC84eX$8qlo;nBzM)gV9qHV=am z@zk9=2bntiVAKk)_U@Hz{o*(VtM)moz?&>FSo5TYUpEGM{e z1T|YZ{m==iZ9n$oP4{up!?TBX(BH=p|Ky~9R}VLtm>{al5kTAE?caUBA3)vhm&!=< zLPriM-Ph) +* [Download new version](<#download-new-version>) +* [Configuring the new version](<#configuring-the-new-version>) +* [Restoring the previous data](<#restoring-the-previous-data>) + +### Stop the TRUE Connector + +To update the TRUE Connector to a newer version first stop the TRUE Connector with: + +``` +docker-compose down +``` + +This will shutdown the TRUE Connector and remove the containers but leave it's volumes with the data. If you wish to do a fresh upgrade then also unmount the volumes with: + +``` +docker-compose down -v +``` + +**CAUTION!** +This will delete all your data! Consider doing a [backup](backup.md) beforehand. + +### Download new version + +New versions of the TRUE Connector can be found [here](https://github.com/Engineering-Research-and-Development/true-connector/releases). + +In the assets section you can choose to download it as a zip or tar.gz: + +![Assets](assets.png "Assets") + +Unpack the folder and you are ready to configure it for your environment. + +### Configuring the new version + +The new versions will always be as a fresh start in that regard you have to configure it anew. Please follow the main README.md of the release that you downloaded. + +### Restoring the previous data + +If you have stopped with *docker-compose down*, the volumes of the previous version with the data are still there. Just start the TRUE Connector with: + +``` +docker-compose up -d +``` + +For restoring data from a backup please follow the [backup & restore guide](backup.md) . + +**NOTE** +Future versions may be incompatible with each other and data migration must be done manually. That will be mentioned in the README.md . \ No newline at end of file diff --git a/doc/update.md b/doc/update.md deleted file mode 100644 index eefaed2..0000000 --- a/doc/update.md +++ /dev/null @@ -1,3 +0,0 @@ -# Updating the TRUE Connector - -To update the TRUE Connector to a newer version you have to download and reconfigure it for your environment as it will always be as a fresh start. To migrate your data to the new version please follow the [backup & restore guide](backup.md) . \ No newline at end of file From 83c1b9132f56d89bc887cb17d725e787b09ae214 Mon Sep 17 00:00:00 2001 From: David Jovanovic Date: Tue, 1 Aug 2023 15:28:28 +0200 Subject: [PATCH 04/33] updated platoon uc readme --- doc/PLATOON_USAGE_CONTROL.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/doc/PLATOON_USAGE_CONTROL.md b/doc/PLATOON_USAGE_CONTROL.md index 04cc40f..8866ab0 100644 --- a/doc/PLATOON_USAGE_CONTROL.md +++ b/doc/PLATOON_USAGE_CONTROL.md @@ -10,8 +10,10 @@ In order to switch to PostgreSQL database, following steps are needed: postgres_provider: image: postgres hostname: postgres_provider - ports: - - "5432:5432" + expose: + - "5432" + networks: + - provider env_file: - ./postgres_provider.env volumes: @@ -21,8 +23,10 @@ In order to switch to PostgreSQL database, following steps are needed: postgres_consumer: image: postgres hostname: postgres_consumer - ports: - - "5444:5432" + expose: + - "5432" + networks: + - consumer env_file: - ./postgres_consumer.env volumes: From 9e42027f5ec9bbd66f758aad672b3999ea78ed8c Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Wed, 2 Aug 2023 10:18:49 +0200 Subject: [PATCH 05/33] Modify existing and add new logback xml configurations --- be-dataapp_resources/logback-DATAAPP.xml | 25 +++++++++++++++++++ ecc_resources_consumer/logback-CONSUMER.xml | 24 ++++++++++++------ ecc_resources_provider/logback-PROVIDER.xml | 20 ++++++++++----- .../logback-UCAPP.xml | 22 ++++++++++++++++ .../logback-UCAPP.xml | 22 ++++++++++++++++ 5 files changed, 99 insertions(+), 14 deletions(-) create mode 100644 be-dataapp_resources/logback-DATAAPP.xml create mode 100644 uc-dataapp_resources_consumer/logback-UCAPP.xml create mode 100644 uc-dataapp_resources_provider/logback-UCAPP.xml diff --git a/be-dataapp_resources/logback-DATAAPP.xml b/be-dataapp_resources/logback-DATAAPP.xml new file mode 100644 index 0000000..46a5a9e --- /dev/null +++ b/be-dataapp_resources/logback-DATAAPP.xml @@ -0,0 +1,25 @@ + + + + %d{dd-MM-yyyy HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg %n + + + + + ${LOG_DIR}/dataapp.log + + ${LOG_DIR}/dataapp-%d{yyyy-MM-dd}.%i.gz + 10MB + 30 + 1GB + + + %d{dd-MM-yyyy HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg %n + + + + + + + diff --git a/ecc_resources_consumer/logback-CONSUMER.xml b/ecc_resources_consumer/logback-CONSUMER.xml index 972d453..3144c4a 100644 --- a/ecc_resources_consumer/logback-CONSUMER.xml +++ b/ecc_resources_consumer/logback-CONSUMER.xml @@ -3,26 +3,34 @@ class="ch.qos.logback.core.ConsoleAppender"> - %d{dd-MM-yyyy HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg %n - - + ${LOG_DIR}/true_connector_audit_consumer.log true - - - + + ${LOG_DIR}/true_connector_consumer.log + + ${LOG_DIR}/true_connector_consumer-%d{yyyy-MM-dd}.%i.gz + 10MB + 30 + 1GB + + + %d{dd-MM-yyyy HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg %n + + + - + - + \ No newline at end of file diff --git a/ecc_resources_provider/logback-PROVIDER.xml b/ecc_resources_provider/logback-PROVIDER.xml index 977df43..c76155b 100644 --- a/ecc_resources_provider/logback-PROVIDER.xml +++ b/ecc_resources_provider/logback-PROVIDER.xml @@ -3,26 +3,34 @@ class="ch.qos.logback.core.ConsoleAppender"> - %d{dd-MM-yyyy HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg %n - - + ${LOG_DIR}/true_connector_audit_provider.log true - + + ${LOG_DIR}/true_connector_provider.log + + ${LOG_DIR}/true_connector_provider-%d{yyyy-MM-dd}.%i.gz + 10MB + 30 + 1GB + + + %d{dd-MM-yyyy HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg %n + + - - + diff --git a/uc-dataapp_resources_consumer/logback-UCAPP.xml b/uc-dataapp_resources_consumer/logback-UCAPP.xml new file mode 100644 index 0000000..df657b6 --- /dev/null +++ b/uc-dataapp_resources_consumer/logback-UCAPP.xml @@ -0,0 +1,22 @@ + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/uc-dataapp_resources_provider/logback-UCAPP.xml b/uc-dataapp_resources_provider/logback-UCAPP.xml new file mode 100644 index 0000000..358a005 --- /dev/null +++ b/uc-dataapp_resources_provider/logback-UCAPP.xml @@ -0,0 +1,22 @@ + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file From 5820c3612cbd0df50378b301cf021c637ad3dbf7 Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Wed, 2 Aug 2023 10:19:28 +0200 Subject: [PATCH 06/33] Add logging property and delete unused ones in properties files --- be-dataapp_resources/application-docker.properties | 2 +- ecc_resources_consumer/application-docker.properties | 12 +++--------- ecc_resources_provider/application-docker.properties | 12 +++--------- uc-dataapp_resources_consumer/application.properties | 2 ++ uc-dataapp_resources_provider/application.properties | 2 ++ 5 files changed, 11 insertions(+), 19 deletions(-) diff --git a/be-dataapp_resources/application-docker.properties b/be-dataapp_resources/application-docker.properties index c5603fe..ca19878 100644 --- a/be-dataapp_resources/application-docker.properties +++ b/be-dataapp_resources/application-docker.properties @@ -1,5 +1,6 @@ ### App setup spring.banner.location=classpath:dataapp_banner.txt +logging.config=classpath:logback-DATAAPP.xml spring.devtools.restart.enabled=false @@ -62,6 +63,5 @@ server.tomcat.max-http-post-size=200291456 information.model.version=4.2.7 -application.security.username=proxy # encoded 'password' application.security.password=$2a$10$MQ5grDaIqDpBjMlG78PFduv.AMRe9cs0CNm/V4cgUubrqdGTFCH3m diff --git a/ecc_resources_consumer/application-docker.properties b/ecc_resources_consumer/application-docker.properties index 3f172e3..cffc003 100644 --- a/ecc_resources_consumer/application-docker.properties +++ b/ecc_resources_consumer/application-docker.properties @@ -14,7 +14,6 @@ application.logging.auditableEvents=NONE application.targetDirectory=/cert/ #SSL Settings for https communication #use https instead of http, if false the ssl settings can be empty -server.ssl.enabled=${SERVER_SSL_ENABLED} application.ssl.key-store.name=${KEYSTORE_NAME} application.ssl.key-store-password=${KEYSTORE_PASSWORD} application.ssl.key-password=${KEY_PASSWORD} @@ -69,8 +68,6 @@ application.trustStorePassword=${TRUSTORE_PASSWORD} application.daps.orbiter.privateKey= application.daps.orbiter.password= -#Disable SSL validation -application.disableSslVerification=${DISABLE_SSL_VALIDATION} #enable protocol validation (default=true) #correct protocol is required in Forward-To (https/http://example.com || wss://example.com || idscp://example.com) @@ -124,16 +121,12 @@ application.openDataAppReceiverRouter=${MULTIPART_EDGE} application.eccHttpSendRouter=${MULTIPART_ECC} #Camel - Spring Boot -camel.springboot.main-run-controller=true camel.ssl.config.key-managers.key-password=${application.ssl.key-password} camel.ssl.config.key-managers.key-store.resource=${application.targetDirectory}${application.ssl.key-store.name} camel.ssl.config.key-managers.key-store.password=${application.ssl.key-store-password} -#Camel - HTTP4 -camel.component.http4.use-global-ssl-context-parameters=${REST_ENABLE_HTTPS} #Camel - Jetty -camel.component.jetty.use-global-ssl-context-parameters=${REST_ENABLE_HTTPS} camel.component.jetty.keystore=${application.targetDirectory}${application.ssl.key-store.name} camel.component.jetty.ssl-password=${application.ssl.key-store-password} camel.component.jetty.ssl-key-password=${application.ssl.key-password} @@ -176,7 +169,8 @@ application.selfdescription.description=Data Consumer Connector description application.selfdescription.title=Data Consumer Connector title application.selfdescription.curator=http://consumer.curatorURI.com application.selfdescription.maintainer=http://consumer.maintainerURI.com -application.selfdescription.filelocation=/home/nobody/app/sd +application.selfdescription.filelocation=/home/nobody/data/sd +application.selfdescription.inboundModelVersion=4.0.0,4.1.0,4.1.2,4.2.0,4.2.1,4.2.2,4.2.3,4.2.4,4.2.5,4.2.6,4.2.7 #For logging the response over WSS set to DEBUG, else leave empty #logging.level.it.eng.idsa.businesslogic.processor.receiver= @@ -192,7 +186,7 @@ application.healthcheck.threshold.audit=5 application.healthcheck.threshold.cpu=0.5 application.healthcheck.threshold.memory=10.5 -application.healthcheck.dataapp=https://be-dataapp-consume:8083/about/version +application.healthcheck.dataapp=https://be-dataapp-consume:8183/about/version application.healthcheck.usagecontrol=https://uc-dataapp-consumer:8080/platoontec/PlatoonDataUsage/1.0/about/version application.healthcheck.daps=${application.dapsJWKSUrl} diff --git a/ecc_resources_provider/application-docker.properties b/ecc_resources_provider/application-docker.properties index 219c6ff..70e3346 100644 --- a/ecc_resources_provider/application-docker.properties +++ b/ecc_resources_provider/application-docker.properties @@ -14,7 +14,6 @@ application.logging.auditableEvents=NONE application.targetDirectory=/cert/ #SSL Settings for https communication #use https instead of http, if false the ssl settings can be empty -server.ssl.enabled=${SERVER_SSL_ENABLED} application.ssl.key-store.name=${KEYSTORE_NAME} application.ssl.key-store-password=${KEYSTORE_PASSWORD} application.ssl.key-password=${KEY_PASSWORD} @@ -69,8 +68,6 @@ application.trustStorePassword=${TRUSTORE_PASSWORD} application.daps.orbiter.privateKey= application.daps.orbiter.password= -#Disable SSL validation -application.disableSslVerification=${DISABLE_SSL_VALIDATION} #enable protocol validation (default=true) #correct protocol is required in Forward-To (https/http://example.com || wss://example.com || idscp://example.com) @@ -124,16 +121,12 @@ application.openDataAppReceiverRouter=${MULTIPART_EDGE} application.eccHttpSendRouter=${MULTIPART_ECC} #Camel - Spring Boot -camel.springboot.main-run-controller=true camel.ssl.config.key-managers.key-password=${application.ssl.key-password} camel.ssl.config.key-managers.key-store.resource=${application.targetDirectory}${application.ssl.key-store.name} camel.ssl.config.key-managers.key-store.password=${application.ssl.key-store-password} -#Camel - HTTP4 -camel.component.http4.use-global-ssl-context-parameters=${REST_ENABLE_HTTPS} #Camel - Jetty -camel.component.jetty.use-global-ssl-context-parameters=${REST_ENABLE_HTTPS} camel.component.jetty.keystore=${application.targetDirectory}${application.ssl.key-store.name} camel.component.jetty.ssl-password=${application.ssl.key-store-password} camel.component.jetty.ssl-key-password=${application.ssl.key-password} @@ -176,7 +169,8 @@ application.selfdescription.description=Data Provider Connector description application.selfdescription.title=Data Provider Connector title application.selfdescription.curator=http://provider.curatorURI.com application.selfdescription.maintainer=http://provider.maintainerURI.com -application.selfdescription.filelocation=/home/nobody/app/sd +application.selfdescription.filelocation=/home/nobody/data/sd +application.selfdescription.inboundModelVersion=4.0.0,4.1.0,4.1.2,4.2.0,4.2.1,4.2.2,4.2.3,4.2.4,4.2.5,4.2.6,4.2.7 #For logging the response over WSS set to DEBUG, else leave empty #logging.level.it.eng.idsa.businesslogic.processor.receiver= @@ -192,7 +186,7 @@ application.healthcheck.threshold.audit=5 application.healthcheck.threshold.cpu=0.5 application.healthcheck.threshold.memory=10.5 -application.healthcheck.dataapp=https://be-dataapp-provider:8083/about/version +application.healthcheck.dataapp=https://be-dataapp-provider:8183/about/version application.healthcheck.usagecontrol=https://uc-dataapp-provider:8080/platoontec/PlatoonDataUsage/1.0/about/version application.healthcheck.daps=${application.dapsJWKSUrl} diff --git a/uc-dataapp_resources_consumer/application.properties b/uc-dataapp_resources_consumer/application.properties index 1089593..01ae01c 100644 --- a/uc-dataapp_resources_consumer/application.properties +++ b/uc-dataapp_resources_consumer/application.properties @@ -1,5 +1,7 @@ spring.banner.location=classpath:true-connector-uc-banner.txt +logging.config=classpath:logback-UCAPP.xml + springfox.documentation.open-api.v3.path=/api-docs server.servlet.contextPath=/platoontec/PlatoonDataUsage/1.0 # diff --git a/uc-dataapp_resources_provider/application.properties b/uc-dataapp_resources_provider/application.properties index d38328f..0448a95 100644 --- a/uc-dataapp_resources_provider/application.properties +++ b/uc-dataapp_resources_provider/application.properties @@ -1,5 +1,7 @@ spring.banner.location=classpath:true-connector-uc-banner.txt +logging.config=classpath:logback-UCAPP.xml + springfox.documentation.open-api.v3.path=/api-docs server.servlet.contextPath=/platoontec/PlatoonDataUsage/1.0 # From 20438960a294b2bdcc29276c6dfcb461453f0a26 Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Wed, 2 Aug 2023 10:20:31 +0200 Subject: [PATCH 07/33] Add script for creating and pre populating docker volume --- prepopulate_be_dataapp_data_provider.sh | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100755 prepopulate_be_dataapp_data_provider.sh diff --git a/prepopulate_be_dataapp_data_provider.sh b/prepopulate_be_dataapp_data_provider.sh new file mode 100755 index 0000000..d549c63 --- /dev/null +++ b/prepopulate_be_dataapp_data_provider.sh @@ -0,0 +1,22 @@ +#!/bin/bash + +# Remove the existing Docker volume (ignoring errors if it doesn't exist) +docker volume rm be_dataapp_data_provider || true + +# Replace 'be-dataapp_provider_datalake' with the actual volume name you want to create +docker volume create be_dataapp_data_provider + +# Change the ownership of the local folder 'be-dataapp_data_provider' to nobody:nogroup +chown -R nobody:nogroup "$(pwd)/be-dataapp_data_provider" + +# Create the 'datalake' directory inside the 'be_dataapp_data_provider' volume with the desired ownership +docker run --rm -v "be_dataapp_data_provider:/target_data" alpine sh -c "mkdir -p /target_data/datalake && chown -R nobody:nogroup /target_data/datalake" + +# Copy data from the 'be-dataapp_data_provider' folder to the 'target_data/datalake' directory inside the Docker volume +docker run --rm -v "$(pwd)/be-dataapp_data_provider:/source_data" -v "be_dataapp_data_provider:/target_data" alpine sh -c "cp -r /source_data/* /target_data/datalake/" + +# Create the 'log' directory inside the 'be_dataapp_data_provider' volume with the desired ownership +docker run --rm -v "be_dataapp_data_provider:/target_data" alpine sh -c "mkdir -p /target_data/log/dataapp && chown -R nobody:nogroup /target_data/log/dataapp" + +# Change the ownership of the Docker volume contents to 'nobody:nogroup' +docker run --rm -v be_dataapp_data_provider:/target_data alpine chown -R nobody:nogroup /target_data From 288d10cda136e44b5a397f61bf6d192e90245b58 Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Wed, 2 Aug 2023 10:22:10 +0200 Subject: [PATCH 08/33] Update docker-compose.yml --- docker-compose.yml | 53 ++++++++++++++++++++-------------------------- 1 file changed, 23 insertions(+), 30 deletions(-) diff --git a/docker-compose.yml b/docker-compose.yml index ab25c3b..40b4379 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -15,24 +15,21 @@ services: - consumer container_name: ecc-provider ports: - - "${PROVIDER_PORT}:8449" #Port for exposing HTTP endpoints - - "8889:8889" #Exposed port for receiving data from another connector (REST) - - "8086:8086" #Exposed port for receiving data from another connector (WS) + - "${PROVIDER_PORT}:8449" #Port for exposing HTTP endpoints + - "8889:8889" #Exposed port for receiving data from another connector (REST) + - "8086:8086" #Exposed port for receiving data from another connector (WS) environment: - "SPRING_PROFILES_ACTIVE=docker" - - DATA_APP_ENDPOINT=${PROVIDER_DATA_APP_ENDPOINT} #Data APP enpoint for consuming received data - - MULTIPART_EDGE=${PROVIDER_MULTIPART_EDGE} #Data APP endpoint multipart/mixed content type + - DATA_APP_ENDPOINT=${PROVIDER_DATA_APP_ENDPOINT} #Data APP enpoint for consuming received data + - MULTIPART_EDGE=${PROVIDER_MULTIPART_EDGE} #Data APP endpoint multipart/mixed content type - MULTIPART_ECC=${MULTIPART_ECC} - - REST_ENABLE_HTTPS=${REST_ENABLE_HTTPS} - IDSCP2=${IDSCP2} - WS_EDGE=${PROVIDER_WS_EDGE} - WS_ECC=${WS_ECC} - UC_DATAAPP_URI=http://uc-dataapp-provider:8080/platoontec/PlatoonDataUsage/1.0/ - BROKER_URL=${BROKER_URL} - - DISABLE_SSL_VALIDATION=${DISABLE_SSL_VALIDATION} - CACHE_TOKEN=${CACHE_TOKEN} - FETCH_TOKEN_ON_STARTUP=${FETCH_TOKEN_ON_STARTUP} - - SERVER_SSL_ENABLED=${SERVER_SSL_ENABLED} - KEYSTORE_NAME=${KEYSTORE_NAME} - KEY_PASSWORD=${KEY_PASSWORD} - KEYSTORE_PASSWORD=${KEYSTORE_PASSWORD} @@ -111,21 +108,20 @@ services: environment: - "SPRING_PROFILES_ACTIVE=docker" - DATA_APP_MULTIPART=${PROVIDER_MULTIPART_EDGE} - - SERVER_SSL_ENABLED=${SERVER_SSL_ENABLED} - KEYSTORE_NAME=${KEYSTORE_NAME} - KEY_PASSWORD=${KEY_PASSWORD} - KEYSTORE_PASSWORD=${KEYSTORE_PASSWORD} - ALIAS=${ALIAS} - ECC_HOSTNAME=ecc-provider + - EXTRACT_PAYLOAD_FROM_RESPONSE=${EXTRACT_PAYLOAD_FROM_RESPONSE} - ECC_PORT=8887 - TZ=Europe/Rome - ISSUER_CONNECTOR_URI=${PROVIDER_ISSUER_CONNECTOR_URI} - - EXTRACT_PAYLOAD_FROM_RESPONSE=${EXTRACT_PAYLOAD_FROM_RESPONSE} volumes: - ./be-dataapp_resources:/config - - ./be-dataapp_data_provider:/home/nobody/data + - be_dataapp_data_provider:/home/nobody/data/ - ./ecc_cert:/cert - + ecc-consumer: image: rdlabengpa/ids_execution_core_container:v1.14.1 deploy: @@ -141,24 +137,21 @@ services: - provider container_name: ecc-consumer ports: - - "${CONSUMER_PORT}:8449" #Port for exposing HTTP endpoints - - "8890:8889" #Exposed port for receiving data from another connector (REST) - - "8087:8086" #Exposed port for receiving data from another connector (WS) + - "${CONSUMER_PORT}:8449" #Port for exposing HTTP endpoints + - "8890:8889" #Exposed port for receiving data from another connector (REST) + - "8087:8086" #Exposed port for receiving data from another connector (WS) environment: - "SPRING_PROFILES_ACTIVE=docker" - - DATA_APP_ENDPOINT=${CONSUMER_DATA_APP_ENDPOINT} #Data APP enpoint for consuming received data - - MULTIPART_EDGE=${CONSUMER_MULTIPART_EDGE} #Data APP endpoint multipart/mixed content type + - DATA_APP_ENDPOINT=${CONSUMER_DATA_APP_ENDPOINT} #Data APP enpoint for consuming received data + - MULTIPART_EDGE=${CONSUMER_MULTIPART_EDGE} #Data APP endpoint multipart/mixed content type - MULTIPART_ECC=${MULTIPART_ECC} - - REST_ENABLE_HTTPS=${REST_ENABLE_HTTPS} - IDSCP2=${IDSCP2} - WS_EDGE=${CONSUMER_WS_EDGE} - WS_ECC=${WS_ECC} - UC_DATAAPP_URI=http://uc-dataapp-consumer:8080/platoontec/PlatoonDataUsage/1.0/ - BROKER_URL=${BROKER_URL} - - DISABLE_SSL_VALIDATION=${DISABLE_SSL_VALIDATION} - CACHE_TOKEN=${CACHE_TOKEN} - FETCH_TOKEN_ON_STARTUP=${FETCH_TOKEN_ON_STARTUP} - - SERVER_SSL_ENABLED=${SERVER_SSL_ENABLED} - KEYSTORE_NAME=${KEYSTORE_NAME} - KEY_PASSWORD=${KEY_PASSWORD} - KEYSTORE_PASSWORD=${KEYSTORE_PASSWORD} @@ -237,7 +230,6 @@ services: environment: - "SPRING_PROFILES_ACTIVE=docker" - DATA_APP_MULTIPART=${CONSUMER_MULTIPART_EDGE} - - SERVER_SSL_ENABLED=${SERVER_SSL_ENABLED} - KEYSTORE_NAME=${KEYSTORE_NAME} - KEY_PASSWORD=${KEY_PASSWORD} - KEYSTORE_PASSWORD=${KEYSTORE_PASSWORD} @@ -251,17 +243,18 @@ services: - ./be-dataapp_resources:/config - ./ecc_cert:/cert - be_dataapp_data_consumer:/home/nobody/data - + networks: consumer: {} provider: {} volumes: - ecc_provider_log : {} - ecc_provider_sd : {} - uc_provider_data : {} - be_dataapp_data_provider : {} - ecc_consumer_log : {} - ecc_consumer_sd : {} - uc_consumer_data : {} - be_dataapp_data_consumer : {} + ecc_provider_log: {} + ecc_provider_sd: {} + uc_provider_data: {} + ecc_consumer_log: {} + ecc_consumer_sd: {} + uc_consumer_data: {} + be_dataapp_data_consumer: {} + be_dataapp_data_provider: + external: true From 222fad05d870f8a182654b03f9579a9653b2f65b Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Wed, 2 Aug 2023 10:23:15 +0200 Subject: [PATCH 09/33] Update TRUE connector postman collection --- TRUE Connector v1.postman_collection.json | 28 +++++++++++------------ 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/TRUE Connector v1.postman_collection.json b/TRUE Connector v1.postman_collection.json index 8c86f31..0c89f95 100644 --- a/TRUE Connector v1.postman_collection.json +++ b/TRUE Connector v1.postman_collection.json @@ -1,9 +1,9 @@ { "info": { - "_postman_id": "d69b4d98-a2fb-44a4-826d-99da07c6cb54", + "_postman_id": "77fdf588-02b3-4d62-b604-bbfd055e03e6", "name": "TRUE Connector v1", "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json", - "_exporter_id": "2326106" + "_exporter_id": "2326106", }, "item": [ { @@ -924,7 +924,7 @@ "basic": [ { "key": "username", - "value": "proxy", + "value": "idsUser", "type": "string" }, { @@ -1120,7 +1120,7 @@ "basic": [ { "key": "username", - "value": "admin", + "value": "apiUser", "type": "string" }, { @@ -1291,13 +1291,13 @@ "type": "basic", "basic": [ { - "key": "password", - "value": "password", + "key": "username", + "value": "idsUser", "type": "string" }, { - "key": "username", - "value": "proxy", + "key": "password", + "value": "password", "type": "string" } ] @@ -1555,7 +1555,7 @@ "basic": [ { "key": "username", - "value": "proxy", + "value": "idsUser", "type": "string" }, { @@ -1663,13 +1663,13 @@ "type": "basic", "basic": [ { - "key": "password", - "value": "password", + "key": "username", + "value": "idsUser", "type": "string" }, { - "key": "username", - "value": "proxy", + "key": "password", + "value": "password", "type": "string" } ] @@ -1701,7 +1701,7 @@ "basic": [ { "key": "username", - "value": "admin", + "value": "apiUser", "type": "string" }, { From 15bcced3f0418a2c617096c8668a4f3ed113a776 Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Wed, 2 Aug 2023 10:23:24 +0200 Subject: [PATCH 10/33] Update documentation --- doc/TRUEConnector/start-stop.md | 15 +++++++++++++++ doc/TRUEConnector/volumes.md | 17 ++++++++++++++--- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/doc/TRUEConnector/start-stop.md b/doc/TRUEConnector/start-stop.md index 0aa9f71..03bfd8e 100644 --- a/doc/TRUEConnector/start-stop.md +++ b/doc/TRUEConnector/start-stop.md @@ -1,5 +1,20 @@ ### Starting and stopping containers +To setup the TRUE connector for starting container, execute the following command: + +``` +sudo ./prepopulate_be_dataapp_data_provider.sh + +``` +With this command, you will create external `be_dataapp_data_provider` volume. + +***NOTE:*** If you're using Linux, check if script is executable, if not, run the next command: + +``` +chmod +x prepopulate_be_dataapp_data_provider.sh + +``` + To start docker container, open terminal and execute following command: ``` diff --git a/doc/TRUEConnector/volumes.md b/doc/TRUEConnector/volumes.md index 8b581fc..fb6f329 100644 --- a/doc/TRUEConnector/volumes.md +++ b/doc/TRUEConnector/volumes.md @@ -16,7 +16,18 @@ be_dataapp_data_consumer Those volumes will store data needed for corresponding service, like log files, self description file, Usage Control H2 database (default configuration) and dataApp resource storage. -If you need to have some files present in volume, for example provider dataApp shares some file, you can either +Volume `be_dataapp_data_provider` is external volume. In order to create it, please execute the script `prepopulate_be_dataapp_data_provider.sh` running next command: -* create volume, mount it to some "dummy" docker container, copy file into volume, stop "dummy" container and you will have file present in volume, and when you start TRUE Connector, it will load already populated dataApp resource volume, or -* you can change using volume and mount folder instead. +``` +sudo ./prepopulate_be_dataapp_data_provider.sh + +``` + +***NOTE:*** If you're using Linux, check if script is executable, if not, run the next command: + +``` +chmod +x prepopulate_be_dataapp_data_provider.sh + +``` + +Running this script, all files present on the host in folder `be-dataapp_data_provder` will be present in volume, so if you need to have some files present in volume, please put them in this folder before running script, and all of them will be present in `DataApp Provider` application. From 6c6483407105ac78d25b7a6aae7adf9f3ed2e2d4 Mon Sep 17 00:00:00 2001 From: David Jovanovic Date: Wed, 2 Aug 2023 10:59:30 +0200 Subject: [PATCH 11/33] updated backup and update docs to match current changes --- doc/backupAndUpdate/backup.md | 10 +++++----- doc/backupAndUpdate/update.md | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/backupAndUpdate/backup.md b/doc/backupAndUpdate/backup.md index 6496d27..6e03e47 100644 --- a/doc/backupAndUpdate/backup.md +++ b/doc/backupAndUpdate/backup.md @@ -7,7 +7,7 @@ To backup and restore data we will be using the official Docker documentation as In order to backup your data you can use the following code snippet: ``` -docker run --rm --volumes-from ecc-consumer -v %cd%:/backup ubuntu tar cvf /backup/backup.tar /var/log/ecc +docker run --rm --volumes-from ecc-consumer -v %cd%:/backup ubuntu tar cvf /backup/backup.tar /home/nobody/data/log ``` Let me explain the options: @@ -15,7 +15,7 @@ Let me explain the options: - ecc-consumer - container with the data that you want to backup - %cd% - current directory in the command prompt ( same as $(pwd) on Linux); the directory where you want the backup to be located - /backup.tar - name of the backup archive - - /var/log/ecc - directory or file from the container that you wish to backup + - /home/nobody/data/log - directory or file from the container that you wish to backup After the process finishes you will find a .tar file with the data. @@ -24,16 +24,16 @@ After the process finishes you will find a .tar file with the data. Restoring the data is done in a similar way: ``` -docker run --rm --volumes-from ecc-consumer -v %cd%:/backup ubuntu bash -c "cd /var && tar xvf /backup/backup.tar --strip 1" +docker run --rm --volumes-from ecc-consumer -v %cd%:/backup ubuntu bash -c "cd /home && tar xvf /backup/backup.tar --strip 1" ``` The options are: - ecc-consumer - container where you want to restore the data - %cd% - current directory in the command prompt ( same as $(pwd) on Linux); the directory where the backup is located - - /var - directory where the data will be restored + - /home - directory where the data will be restored - /backup.tar - name of the backup archive **NOTE** - If the backup hierarchy looks like this */var/log/ecc* and you wish to keep it when restoring then you have to point the root folder in the command */var*, as it was done in the commands from above. \ No newline at end of file + If the backup hierarchy looks like this */home/nobody/data/log* and you wish to keep it when restoring then you have to point the root folder in the command */home*, as it was done in the commands from above. \ No newline at end of file diff --git a/doc/backupAndUpdate/update.md b/doc/backupAndUpdate/update.md index 0f4326b..907eca5 100644 --- a/doc/backupAndUpdate/update.md +++ b/doc/backupAndUpdate/update.md @@ -22,7 +22,7 @@ docker-compose down -v ``` **CAUTION!** -This will delete all your data! Consider doing a [backup](backup.md) beforehand. +This will delete all your data except be_dataapp_data_provider volume, since that one is made outside of the docker compose! Consider doing a [backup](backup.md) beforehand. ### Download new version From 6e10461e6e8b8b1d168da6fe131627259c500a4f Mon Sep 17 00:00:00 2001 From: "igor.balog" Date: Wed, 2 Aug 2023 11:51:53 +0200 Subject: [PATCH 12/33] Prerequisite and audit log update --- README.md | 1 + doc/TRUEConnector/prerequisite.md | 55 ++++++++++++++++++++++++++ doc/advancedConfiguration/auditlogs.md | 27 ++++++++----- 3 files changed, 73 insertions(+), 10 deletions(-) create mode 100644 doc/TRUEConnector/prerequisite.md diff --git a/README.md b/README.md index 4d8a10d..3bc97e6 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,7 @@ Please [click here](https://engineering-ing-inf-rd.gitbook.io/true-connector/), * [TRUE Connector]() * [Table of Contents]() + * [Prerequisite](doc/TRUEConnector/prerequisite.md) * [Introduction](doc/TRUEConnector/introduction.md) * [System requirements](doc/TRUEConnector/system-requirements.md) * [Volumes](doc/TRUEConnector/volumes.md) diff --git a/doc/TRUEConnector/prerequisite.md b/doc/TRUEConnector/prerequisite.md new file mode 100644 index 0000000..ab9d138 --- /dev/null +++ b/doc/TRUEConnector/prerequisite.md @@ -0,0 +1,55 @@ +# Prerequisite + +To have secure and certification compliant environment, following prerequisites are mandatory to be performed before setting up TRUE Connector: + +* NTP time server of the machine, where TRUE Connector will be deployed, has to be enabled and configured correctly. This will allow that once certificates are checked, correct time will be used to verify certificate, expired or not. This applies for both DAPS and TLS1.3 certificates. Connector will rely on OS time when checking certificates +* Docker is mandatory "OS service" for running connector +* verify [System requirements](system-requirements.md) before starting the connector. + +## Securing docker host + +* The host OS should be audited and secure; OS should be as minimal as possible and it should be preferably used to host our Docker exclusively. There should not coexist other services like web servers or web applications so that attacker could not exploit it or lead to potential exploit (minimal threat attack surface). +* Monitoring mechanism (Linux auditd service for example) should be installed and configured as prerequisite before deploying connector. This will capture if someone tries to make changes on property files used by the connector. +* make sure to create rules to monitor folders and property files of the TRUE Connector (for example auditctl -w /xxxx/TRUEConnector/* -k trueconnector, depending on the location where TRUE Connector is deployed) +* make sure to create rules for monitoring docker service (dockerd, /run/containerc, /var/lib/docker, /etc/docker, docker.service...) This might differ based on OS distribution +* rules for auditing should be persisted (/etc/audit/audit.d/rules/audit.rules file, depending on the OS distribution, location might differ) + +* User responsible for setting up environment where connector will run should isolate or disable other services. +* OS user for running docker should not be root user; be sure to create new user, assign new user to docker group, that user can run docker compose; add it to sudoers, if required by the OS distribution +* disable password login to the server for newly created user and allow only key-based authentication to the server +* disable access for the root user by using a password when connecting to the server via ssh + +## Securing docker deamon + +User namespace configuration???? + + +* 2 types of certificate are required: DAPS and TLS. +DAPS certificate should be obtained from Certified Authority responsible for the Dataspace, while TLS certificate can be self signed or signed by some CA. More information about TLS certificate can be found [here](../security.md). + + +* make sure to create rules for mounted docker volumes + + +# Post configuration steps + +Once TRUE Connector is successfully configured and is up and running, responsible user for setting up environment and configuring connector should generate new passwords for 2 type of users required for operating with connector. More information how to do this can be found [here](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.2/doc/SECURITY.md#change-default-password). + +Make sure to update following properties to address your usecase: + + +.env file + +``` +PROVIDER_ISSUER_CONNECTOR_URI=http://w3id.org/engrd/connector/provider +CONSUMER_ISSUER_CONNECTOR_URI=http://w3id.org/engrd/connector/consumer +``` +and in ecc_resources_consumer and ecc_resources_provider application.property file: + +``` +application.selfdescription.description=Data Consumer Connector description +application.selfdescription.title=Data Consumer Connector title +application.selfdescription.curator=http://consumer.curatorURI.com +application.selfdescription.maintainer=http://consumer.maintainerURI.com + +``` diff --git a/doc/advancedConfiguration/auditlogs.md b/doc/advancedConfiguration/auditlogs.md index 1036bd1..e19ebc4 100644 --- a/doc/advancedConfiguration/auditlogs.md +++ b/doc/advancedConfiguration/auditlogs.md @@ -1,30 +1,37 @@ ### Audit logs -Audit logging is turned **off** by default. If you wish to configure it or even turn off please follow this [document](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.1/doc/AUDIT.md) . +Audit logging is turned **off** by default. If you wish to configure it or even turn off please follow this [document](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.2/doc/AUDIT.md) . ## Accessing audit logs -Access to the audit logs should be allowed only to the person responsible for configuring and setting up TRUE Connector. Lods are stored in docker volumes, and in default configuration those docker volumes are: +Access to the audit logs should be allowed only to the person responsible for configuring and setting up TRUE Connector. Logs are stored in docker volumes, and in default configuration those docker volumes are: -ecc_provider_log -ecc_consumer_log +ecc_provider +ecc_consumer -Once audit events are turned on, and docker containers are up and running, you can inspect the from the terminal, and access the one for the Excecution Core Container by executing command: +Once audit events are turned on, and docker containers are up and running, you can inspect the from the terminal, and access the one for the Execution Core Container by executing command: ``` docker exec -it ecc-consumer /bin/sh ``` -Once you manage to log into container, you can navigate to */var/log/ecc* and verify that audit tile is present. File might be empty, if there are no actions performed so far, so be sure to have some interaction with the connector, to verify that file is being updated. +Once you manage to connect to container, you can navigate to */home/nobody/data/log/ecc* and verify that log and audit files are present. File might be empty, if there are no actions are performed so far, sure to have some interaction with the connector, to verify that file is being updated. ``` -/var/log/ecc $ ls -la +/home/nobody/data/log/ecc $ ls -la total 12 drwxr-xr-x 2 nobody nogroup 4096 Jul 25 16:01 . drwxr-xr-x 1 root root 4096 Jul 19 09:47 .. --rw-r--r-- 1 nobody nobody 1578 Jul 25 16:09 true_connector_audit_consumer.log -/var/log/ecc $ +-rw-r--r-- 1 nobody nobody 1579 Jul 27 10:17 true_connector_audit_consumer.log +-rw-r--r-- 1 nobody nobody 55019 Jul 27 10:37 true_connector_consumer.log +/home/nobody/data/log/ecc $ ``` -Content of the file is consisted of json entries. \ No newline at end of file +Content of the file is consisted of json entries. + +## Trace log file + +Trace log file contains logs that can be useful when debugging or investigating what was incorrect and why connector is responding as expected. In cases when connector does not start, or if response is not expected, some rejection message is returned instead any other response, good starting point is to check content of this log file and get information why connector is not responding as expected. + +User might try to solve the problem, in most cases problem will be configuration related or that invalid values are passed. \ No newline at end of file From c567c66b92cefffdc0d397d8d4787d66497f176f Mon Sep 17 00:00:00 2001 From: "igor.balog" Date: Wed, 2 Aug 2023 15:44:16 +0200 Subject: [PATCH 13/33] Prerequisite update --- doc/TRUEConnector/prerequisite.md | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/doc/TRUEConnector/prerequisite.md b/doc/TRUEConnector/prerequisite.md index ab9d138..6bd8596 100644 --- a/doc/TRUEConnector/prerequisite.md +++ b/doc/TRUEConnector/prerequisite.md @@ -13,24 +13,19 @@ To have secure and certification compliant environment, following prerequisites * make sure to create rules to monitor folders and property files of the TRUE Connector (for example auditctl -w /xxxx/TRUEConnector/* -k trueconnector, depending on the location where TRUE Connector is deployed) * make sure to create rules for monitoring docker service (dockerd, /run/containerc, /var/lib/docker, /etc/docker, docker.service...) This might differ based on OS distribution * rules for auditing should be persisted (/etc/audit/audit.d/rules/audit.rules file, depending on the OS distribution, location might differ) +* make sure to create rules for mounted docker volumes (to be able to keep track of changes made over files present in those volumes) -* User responsible for setting up environment where connector will run should isolate or disable other services. -* OS user for running docker should not be root user; be sure to create new user, assign new user to docker group, that user can run docker compose; add it to sudoers, if required by the OS distribution -* disable password login to the server for newly created user and allow only key-based authentication to the server -* disable access for the root user by using a password when connecting to the server via ssh - -## Securing docker deamon -User namespace configuration???? +* User responsible for setting up environment where connector will run should isolate or disable other services. +* OS user for running docker should not be root user; be sure to create new user, assign new user to docker group, that user can run docker compose +* disable password login to the server for newly created user and allow only key-based authentication for accessing the server where connector will run +* disable access for the root user by using a password when connecting to the server via ssh (key-based auth only) * 2 types of certificate are required: DAPS and TLS. DAPS certificate should be obtained from Certified Authority responsible for the Dataspace, while TLS certificate can be self signed or signed by some CA. More information about TLS certificate can be found [here](../security.md). -* make sure to create rules for mounted docker volumes - - # Post configuration steps Once TRUE Connector is successfully configured and is up and running, responsible user for setting up environment and configuring connector should generate new passwords for 2 type of users required for operating with connector. More information how to do this can be found [here](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.2/doc/SECURITY.md#change-default-password). From 7b4abe04049d47e1bfa70949343fb9015aa4049d Mon Sep 17 00:00:00 2001 From: "igor.balog" Date: Wed, 9 Aug 2023 12:13:16 +0200 Subject: [PATCH 14/33] Added user managemend readme --- README.md | 1 + doc/user_management.md | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+) create mode 100644 doc/user_management.md diff --git a/README.md b/README.md index 7a05f41..1f9b98d 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,7 @@ Please [click here](https://engineering-ing-inf-rd.gitbook.io/true-connector/), * [Self Description API](doc/selfDescriptionAPI/self-description-API.md) * [Changing API password](doc/selfDescriptionAPI/changepassword.md) * [Postman collection](doc/selfDescriptionAPI/postman.md) + * [User management](doc/user_management.md) * [Backup & restore](doc/backupAndUpdate/backup.md) * [Updating](doc/backupAndUpdate/update.md) * [Docker image signing and verification](doc/cosign.md) diff --git a/doc/user_management.md b/doc/user_management.md new file mode 100644 index 0000000..2bb1e86 --- /dev/null +++ b/doc/user_management.md @@ -0,0 +1,24 @@ +# User management + +TRUE Connector implements simple user management; 2 users, idsUser and apiUser are present in 2 property files, and are responsible for: + +idsUser - interacts with Basic DataApp, initiates communication with connector +apiUser - interacts with Execution Core Container, makes modification for Self Description document + +With their responsibilities, idsUser can be found and managed by modifying DataApp property file, + +``` +application.security.password=$2a$10$MQ5grDaIqDpBjMlG78PFduv.AMRe9cs0CNm/V4cgUubrqdGTFCH3m + +``` + +while apiUser is present in ecc property file. + +``` +application.user.api.password=$2a$10$MQ5grDaIqDpBjMlG78PFduv.AMRe9cs0CNm/V4cgUubrqdGTFCH3m +``` + + +## Modifying password for a user + +Once new password is generated, (described [here](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.1/doc/SECURITY.md#change-default-password)) user should send encoded password to the operations user, which should be the only one who can modify connector property file. That user will update property file and restart TRUE Connector, so that new password will be loaded by the connector. \ No newline at end of file From ff364477d0c25a05fd3f0a777332355d9d67c552 Mon Sep 17 00:00:00 2001 From: "igor.balog" Date: Wed, 9 Aug 2023 15:56:40 +0200 Subject: [PATCH 15/33] Added security issues implemented --- doc/traceability_matrix.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/doc/traceability_matrix.md b/doc/traceability_matrix.md index 567b5c7..7bc64b6 100644 --- a/doc/traceability_matrix.md +++ b/doc/traceability_matrix.md @@ -29,4 +29,18 @@ Based on the severity (Common Vulnerability Scoring System is used) of the issue | Moderate | < 1 month | | Low | < 2 months | -For issues that are currently reported, you can always check Security tab for specific subcomponent and in Dependabot section find all opened issues. \ No newline at end of file +For issues that are currently reported, you can always check Security tab for specific subcomponent and in Dependabot section find all opened issues. + + +## Security issues implemented + +Fixes for security issues should be covered with tests. Once issue is fixed, new GitHub Action should be created and added to the existing set of tests (when applicable). + +| Description | Fixed version TC | Component | +|:------------|:---------:|:---------:| +| com.auth0:jwks-rsa from 0.21.1 to 0.22.1 | 1.0.1 | ECC | +| net.logstash.logback:logstash-logback-encoder from 7.0.1 to 7.3 | 1.0.1 | ECC | +| com.auth0:java-jwt from 3.19.1 to 3.19.3 | 1.0.1 | ECC | +| org.bitbucket.b_c:jose4j:0.7.8 to 0.9.3 | 1.0.1 | ECC | +| TLS 1.3 as mandatory way for communication | 1.0.1 | ECC, DA, UC | + From e85bcef1c0e33fd2d23c698957fb3042db4f7d06 Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Wed, 9 Aug 2023 17:42:11 +0200 Subject: [PATCH 16/33] Update TRUE Connector postman collection --- TRUE Connector v1.postman_collection.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/TRUE Connector v1.postman_collection.json b/TRUE Connector v1.postman_collection.json index 0c89f95..2d69d11 100644 --- a/TRUE Connector v1.postman_collection.json +++ b/TRUE Connector v1.postman_collection.json @@ -1,9 +1,9 @@ { "info": { - "_postman_id": "77fdf588-02b3-4d62-b604-bbfd055e03e6", + "_postman_id": "d69b4d98-a2fb-44a4-826d-99da07c6cb54", "name": "TRUE Connector v1", "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json", - "_exporter_id": "2326106", + "_exporter_id": "2326106" }, "item": [ { From 0dbc59a2d5a8177130649accdc300439b6a657e8 Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Wed, 9 Aug 2023 17:42:21 +0200 Subject: [PATCH 17/33] Update prepoulate script --- prepopulate_be_dataapp_data_provider.sh | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/prepopulate_be_dataapp_data_provider.sh b/prepopulate_be_dataapp_data_provider.sh index d549c63..f0e7827 100755 --- a/prepopulate_be_dataapp_data_provider.sh +++ b/prepopulate_be_dataapp_data_provider.sh @@ -1,22 +1,22 @@ #!/bin/bash # Remove the existing Docker volume (ignoring errors if it doesn't exist) -docker volume rm be_dataapp_data_provider || true +docker volume rm be_dataapp_provider_data || true -# Replace 'be-dataapp_provider_datalake' with the actual volume name you want to create -docker volume create be_dataapp_data_provider +# Replace 'be-be_dataapp_provider_data' with the actual volume name you want to create +docker volume create be_dataapp_provider_data # Change the ownership of the local folder 'be-dataapp_data_provider' to nobody:nogroup chown -R nobody:nogroup "$(pwd)/be-dataapp_data_provider" -# Create the 'datalake' directory inside the 'be_dataapp_data_provider' volume with the desired ownership -docker run --rm -v "be_dataapp_data_provider:/target_data" alpine sh -c "mkdir -p /target_data/datalake && chown -R nobody:nogroup /target_data/datalake" +# Create the 'datalake' directory inside the 'be_dataapp_provider_data' volume with the desired ownership +docker run --rm -v "be_dataapp_provider_data:/target_data" alpine sh -c "mkdir -p /target_data/datalake && chown -R nobody:nogroup /target_data/datalake" # Copy data from the 'be-dataapp_data_provider' folder to the 'target_data/datalake' directory inside the Docker volume -docker run --rm -v "$(pwd)/be-dataapp_data_provider:/source_data" -v "be_dataapp_data_provider:/target_data" alpine sh -c "cp -r /source_data/* /target_data/datalake/" +docker run --rm -v "$(pwd)/be-dataapp_data_provider:/source_data" -v "be_dataapp_provider_data:/target_data" alpine sh -c "cp -r /source_data/* /target_data/datalake/" -# Create the 'log' directory inside the 'be_dataapp_data_provider' volume with the desired ownership -docker run --rm -v "be_dataapp_data_provider:/target_data" alpine sh -c "mkdir -p /target_data/log/dataapp && chown -R nobody:nogroup /target_data/log/dataapp" +# Create the 'log' directory inside the 'be_dataapp_provider_data' volume with the desired ownership +docker run --rm -v "be_dataapp_provider_data:/target_data" alpine sh -c "mkdir -p /target_data/log/dataapp && chown -R nobody:nogroup /target_data/log/dataapp" # Change the ownership of the Docker volume contents to 'nobody:nogroup' -docker run --rm -v be_dataapp_data_provider:/target_data alpine chown -R nobody:nogroup /target_data +docker run --rm -v be_dataapp_provider_data:/target_data alpine chown -R nobody:nogroup /target_data From a94688e07f9eaa2c6e845eb7c4be42a4518d35a8 Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Wed, 9 Aug 2023 17:42:42 +0200 Subject: [PATCH 18/33] Update docker-compose file with new volumes and versions --- docker-compose.yml | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-) diff --git a/docker-compose.yml b/docker-compose.yml index 40b4379..7497fba 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,7 +1,7 @@ version: '3.1' services: ecc-provider: - image: rdlabengpa/ids_execution_core_container:v1.14.1 + image: rdlabengpa/ids_execution_core_container:v1.14.2 deploy: resources: limits: @@ -43,13 +43,12 @@ services: volumes: - ./ecc_resources_provider:/config - ./ecc_cert:/cert - - ecc_provider_log:/var/log/ecc - - ecc_provider_sd:/home/nobody/app/sd + - ecc_provider_data:/home/nobody/data/ extra_hosts: - "ecc-provider:172.17.0.1" uc-dataapp-provider: - image: rdlabengpa/ids_uc_data_app_platoon:v1.7.2 + image: rdlabengpa/ids_uc_data_app_platoon:v1.7.3 deploy: resources: limits: @@ -90,7 +89,7 @@ services: - ./uc-dataapp-pip_resources_provider:/etc be-dataapp-provider: - image: rdlabengpa/ids_be_data_app:v0.3.0 + image: rdlabengpa/ids_be_data_app:v0.3.1 deploy: resources: limits: @@ -119,11 +118,11 @@ services: - ISSUER_CONNECTOR_URI=${PROVIDER_ISSUER_CONNECTOR_URI} volumes: - ./be-dataapp_resources:/config - - be_dataapp_data_provider:/home/nobody/data/ + - be_dataapp_provider_data:/home/nobody/data/ - ./ecc_cert:/cert ecc-consumer: - image: rdlabengpa/ids_execution_core_container:v1.14.1 + image: rdlabengpa/ids_execution_core_container:v1.14.2 deploy: resources: limits: @@ -165,13 +164,12 @@ services: volumes: - ./ecc_resources_consumer:/config - ./ecc_cert:/cert - - ecc_consumer_log:/var/log/ecc - - ecc_consumer_sd:/sd + - ecc_consumer_data:/home/nobody/data/ extra_hosts: - "ecc-consumer:172.17.0.1" uc-dataapp-consumer: - image: rdlabengpa/ids_uc_data_app_platoon:v1.7.2 + image: rdlabengpa/ids_uc_data_app_platoon:v1.7.3 deploy: resources: limits: @@ -212,7 +210,7 @@ services: - ./uc-dataapp-pip_resources_consumer:/etc be-dataapp-consumer: - image: rdlabengpa/ids_be_data_app:v0.3.0 + image: rdlabengpa/ids_be_data_app:v0.3.1 deploy: resources: limits: @@ -242,19 +240,17 @@ services: volumes: - ./be-dataapp_resources:/config - ./ecc_cert:/cert - - be_dataapp_data_consumer:/home/nobody/data + - be_dataapp_consumer_data:/home/nobody/data networks: consumer: {} provider: {} volumes: - ecc_provider_log: {} - ecc_provider_sd: {} + ecc_provider_data: {} uc_provider_data: {} - ecc_consumer_log: {} - ecc_consumer_sd: {} + ecc_consumer_data: {} uc_consumer_data: {} - be_dataapp_data_consumer: {} - be_dataapp_data_provider: + be_dataapp_consumer_data: {} + be_dataapp_provider_data: external: true From fde9630ab7ef23eb9d6575bd028f4ae683cf752b Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Wed, 9 Aug 2023 17:43:05 +0200 Subject: [PATCH 19/33] Update documentation --- doc/TEST_API.md | 60 +++++++++---------- doc/TRUEConnector/component-overview.md | 8 +-- doc/TRUEConnector/start-stop.md | 18 +++--- doc/TRUEConnector/volumes.md | 12 ++-- .../contract_agreement_request.md | 4 +- .../contract_request_message.md | 4 +- .../description_request_message.md | 4 +- .../get_offered_resource.md | 4 +- .../get_offered_resource_granted.md | 4 +- doc/cosign.md | 18 +++--- doc/exchange-data.md | 4 +- 11 files changed, 69 insertions(+), 71 deletions(-) diff --git a/doc/TEST_API.md b/doc/TEST_API.md index e7dcf56..805169d 100644 --- a/doc/TEST_API.md +++ b/doc/TEST_API.md @@ -69,13 +69,13 @@ User can also use it to check if TRUE Connector is ready for use (if component i Example of the request: ``` -curl --location 'http://localhost:8081/about/version' +curl --location -k 'https://localhost:8090/about/version' ``` and expected response: ``` -0.3.0-SNAPSHOT +1.14.2-SNAPSHOT ``` ## Self Description API @@ -93,9 +93,9 @@ Required header element - contractOffer of type URI Request example: ``` -curl --location 'http://localhost:8081/api/contractOffer/' \ +curl --location -k 'https://localhost:8090/api/contractOffer/' \ --header 'contractOffer: https://w3id.org/idsa/autogen/contractOffer/1e902a98-7858-4336-9607-64b9e243a76c' \ ---header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' +--header 'Authorization: Basic YXBpVXNlcjpwYXNzd29yZA==' ``` If requested contract offer is present in self description, following response is returned: @@ -179,9 +179,9 @@ NOTE: It is required to provide context in json representation, otherwise, reque Example request for adding contact offer: ``` -curl --location 'http://localhost:8081/api/contractOffer/' \ +curl --location -k 'https://localhost:8090/api/contractOffer/' \ --header 'resource: https://w3id.org/idsa/autogen/textResource/a329a2fd-1002-4753-822e-89561f148839' \ ---header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \ +--header 'Authorization: Basic YXBpVXNlcjpwYXNzd29yZA==' \ --header 'Content-Type: application/json' \ --data-raw '{ "@context" : { @@ -276,9 +276,9 @@ Required header element - contractOffer of type URI Request example: ``` -curl --location --request DELETE 'http://localhost:8081/api/contractOffer/' \ +curl --location -k --request DELETE 'https://localhost:8090/api/contractOffer/' \ --header 'contractOffer: https://w3id.org/idsa/autogen/contractOffer/1e902a98-7858-4336-9607-64b9e243a76c' \ ---header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' +--header 'Authorization: Basic YXBpVXNlcjpwYXNzd29yZA==' ``` Expected response - self description document, without contract offer, if such existed. @@ -292,9 +292,9 @@ Required header element - resource of type URI Request example: ``` -curl --location 'http://localhost:8081/api/offeredResource/' \ +curl --location -k 'https://localhost:8090/api/offeredResource/' \ --header 'resource: https://w3id.org/idsa/autogen/textResource/a329a2fd-1002-4753-822e-89561f148839' \ ---header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' +--header 'Authorization: Basic YXBpVXNlcjpwYXNzd29yZA==' ``` And successful response: @@ -448,9 +448,9 @@ NOTE: It is required to provide context in json representation, otherwise, reque Example request for adding offered resource: ``` -curl --location --request POST 'http://localhost:8081/api/offeredResource/' \ +curl --location -k --request POST 'https://localhost:8090/api/offeredResource/' \ --header 'resource: https://w3id.org/idsa/autogen/textResource/6e7c04f2-a09d-41b2-8334-013877bbda12' \ ---header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \ +--header 'Authorization: Basic YXBpVXNlcjpwYXNzd29yZA==' \ --header 'Content-Type: application/json' \ --data-raw '{ "@context": { @@ -633,9 +633,9 @@ Required header element - resource of type URI Example request: ``` -curl --location --request DELETE 'http://localhost:8081/api/offeredResource/' \ +curl --location -k --request DELETE 'https://localhost:8090/api/offeredResource/' \ --header 'resource: https://w3id.org/idsa/autogen/textResource/6e7c04f2-a09d-41b2-8334-013877bbda12' \ ---header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' +--header 'Authorization: Basic YXBpVXNlcjpwYXNzd29yZA==' ``` Expected successful response - self description document, without offered resource, if such existed. @@ -657,9 +657,9 @@ Required header element - representation of type URI Request example: ``` -curl --location 'http://localhost:8081/api/representation/' \ +curl --location -k 'https://localhost:8090/api/representation/' \ --header 'representation: https://w3id.org/idsa/autogen/textRepresentation/09b9b628-77ee-40a2-98c4-79b559370cda' \ ---header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' +--header 'Authorization: Basic YXBpVXNlcjpwYXNzd29yZA==' ``` Successful response: @@ -721,9 +721,9 @@ NOTE: It is required to provide context in json representation, otherwise, reque Example request for adding contact offer: ``` -curl --location 'http://localhost:8081/api/representation/' \ +curl --location -k 'https://localhost:800-/api/representation/' \ --header 'resource: https://w3id.org/idsa/autogen/textResource/424e2559-50ad-411e-bea0-93ff6550aa80' \ ---header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \ +--header 'Authorization: Basic YXBpVXNlcjpwYXNzd29yZA==' \ --header 'Content-Type: application/json' \ --data-raw '{ "@context": { @@ -799,9 +799,9 @@ Required header element - representation of type URI Request example: ``` -curl --location --request DELETE 'http://localhost:8081/api/representation/' \ +curl --location -k --request DELETE 'https://localhost:8090/api/representation/' \ --header 'representation: https://w3id.org/idsa/autogen/textRepresentation/09b9b628-77ee-40a2-98c4-79b559370cda' \ ---header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' +--header 'Authorization: Basic YXBpVXNlcjpwYXNzd29yZA==' ``` Expected response - self description document, without contract offer, if such existed. @@ -811,8 +811,8 @@ Expected response - self description document, without contract offer, if such e Example request: ``` -curl --location 'http://localhost:8081/api/selfDescription/' \ ---header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' +curl --location -k 'https://localhost:8090/api/selfDescription/' \ +--header 'Authorization: Basic YXBpVXNlcjpwYXNzd29yZA==' ``` Expected response is connector Self Description document, with all elements. @@ -846,9 +846,9 @@ multipart - mixed Example request: ``` -curl --location --request POST 'https://localhost:8887/incoming-data-app/multipartMessageBodyBinary' \ +curl --location -k --request POST 'https://localhost:8887/incoming-data-app/multipartMessageBodyBinary' \ --header 'Forward-To: https://localhost:8889/data' \ ---header 'Authorization: Basic Y29ubmVjdG9yOnBhc3N3b3Jk' \ +--header 'Authorization: Basic aWRzVXNlcjpwYXNzd29yZA==' \ --header 'Content-Type: text/plain' \ --data-raw '--9RDrAvgB92_-w2A-YY7av8i7GEQcKogs7pjm Content-Disposition: form-data; name="header" @@ -900,9 +900,9 @@ multipart - form Example request: ``` -curl --location --request POST 'https://localhost:8887/incoming-data-app/multipartMessageBodyFormData' \ +curl --location -k --request POST 'https://localhost:8887/incoming-data-app/multipartMessageBodyFormData' \ --header 'Forward-To: https://localhost:8889/data' \ ---header 'Authorization: Basic Y29ubmVjdG9yOnBhc3N3b3Jk' \ +--header 'Authorization: Basic aWRzVXNlcjpwYXNzd29yZA==' \ --form 'header="{ \"@context\" : { \"ids\" : \"https://w3id.org/idsa/core/\", @@ -948,7 +948,7 @@ multipart - http-header This request is a bit specific, since it is required to convert IDS message to http headers (logic that DataApp proxy request do for you) and when conversion is done correct this is how request looks like, depending of the Messagetype and its mandatory fields: ``` -curl --location 'https://localhost:8887/incoming-data-app/multipartMessageHttpHeader' \ +curl --location -k 'https://localhost:8887/incoming-data-app/multipartMessageHttpHeader' \ --header 'Forward-To: https://localhost:8889/data' \ --header 'IDS-CorrelationMessage: http://correlationMessage' \ --header 'IDS-Id: https://w3id.org/idsa/autogen/ArtifactRequestMessage/e5939da0-7240-499b-ac1b-2c6ac5718933' \ @@ -963,7 +963,7 @@ curl --location 'https://localhost:8887/incoming-data-app/multipartMessageHttpHe --header 'IDS-SecurityToken-Type: ids:DynamicAttributeToken' \ --header 'IDS-SenderAgent: http://w3id.org/engrd/connector/' \ --header 'IDS-TransferContract: https://w3id.org/idsa/autogen/contractAgreement/39f9cc50-5d9b-4d12-80dc-23e03f3cc1f8' \ ---header 'Authorization: Basic Y29ubmVjdG9yOnBhc3N3b3Jk' \ +--header 'Authorization: Basic aWRzVXNlcjpwYXNzd29yZA==' \ --header 'Content-Type: text/plain' \ --data 'PAYLOAD' ``` @@ -977,8 +977,8 @@ There are convenient endpoints to initiate flow with Broker. They can be trigger Example proxy request: ``` -curl --location 'https://localhost:8184/proxy' \ ---header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \ +curl --location -k 'https://localhost:8184/proxy' \ +--header 'Authorization: Basic aWRzVXNlcjpwYXNzd29yZA==' \ --header 'Content-Type: application/json' \ --data '{ "multipart": "form", diff --git a/doc/TRUEConnector/component-overview.md b/doc/TRUEConnector/component-overview.md index c846483..7f530b4 100644 --- a/doc/TRUEConnector/component-overview.md +++ b/doc/TRUEConnector/component-overview.md @@ -4,10 +4,10 @@ TRUE Connector is build using Java11, and use following libraries: | Component | Version | | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | -| [Execution core container](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/releases/tag/1.14.1) | 1.14.1 | -| [Basic data app](https://github.com/Engineering-Research-and-Development/true-connector-basic_data_app/releases/tag/0.3.0) | 0.3.0 | -| [Usage control app](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/releases/tag/1.7.2) | 1.7.2 | -| [Pip](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/tree/1.7.2/Docker_Tecnalia_DataUsage/pip) | 1.0.0 | +| [Execution core container](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/releases/tag/1.14.2) | 1.14.2 | +| [Basic data app](https://github.com/Engineering-Research-and-Development/true-connector-basic_data_app/releases/tag/0.3.1) | 0.3.1 | +| [Usage control app](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/releases/tag/1.7.3) | 1.7.3 | +| [Pip](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/tree/1.7.3/Docker_Tecnalia_DataUsage/pip) | 1.0.0 | | [Multipart Message Library](https://github.com/Engineering-Research-and-Development/true-connector-multipart_message_library/releases/tag/1.0.17) | 1.0.17 | | [Websocket Message Streamer](https://github.com/Engineering-Research-and-Development/true-connector-websocket_message_streamer/releases/tag/1.0.17) | 1.0.17 | | [Information model](https://github.com/International-Data-Spaces-Association/InformationModel) | 4.2.7 | diff --git a/doc/TRUEConnector/start-stop.md b/doc/TRUEConnector/start-stop.md index 03bfd8e..2f5ccc8 100644 --- a/doc/TRUEConnector/start-stop.md +++ b/doc/TRUEConnector/start-stop.md @@ -63,15 +63,15 @@ ecc-consumer | INFO: Initializing Spring DispatcherServlet 'dispatcherSe You can also check using _docker ps_ command to verify that containers are up and running: ``` -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -5c7bd4b70295 rdlabengpa/ids_execution_core_container:v1.13.1 "/bin/sh -c 'java -j…" About a minute ago Up About a minute (healthy) 0.0.0.0:8087->8086/tcp, 0.0.0.0:8091->8449/tcp, 0.0.0.0:8890->8889/tcp ecc-consumer -8883c669e3ed rdlabengpa/ids_execution_core_container:v1.13.1 "/bin/sh -c 'java -j…" About a minute ago Up About a minute (healthy) 0.0.0.0:8086->8086/tcp, 0.0.0.0:8889->8889/tcp, 0.0.0.0:8090->8449/tcp ecc-provider -752af4b94096 rdlabengpa/ids_uc_data_app_platoon_pip:v1.0.0 "java -jar pip.jar" About a minute ago Up About a minute 0/tcp uc-dataapp-pip-provider -095bf6919593 rdlabengpa/ids_uc_data_app_platoon:v1.7.1 "/bin/sh -c 'java -j…" About a minute ago Up About a minute 8080/tcp uc-dataapp-provider -59c3b560d8b6 rdlabengpa/ids_be_data_app:v0.2.7 "/bin/sh -c 'java -j…" About a minute ago Up About a minute (unhealthy) 0.0.0.0:8184->8183/tcp, 0.0.0.0:9001->9000/tcp be-dataapp-consumer -3f8979a433cf rdlabengpa/ids_uc_data_app_platoon:v1.7.1 "/bin/sh -c 'java -j…" About a minute ago Up About a minute 8080/tcp uc-dataapp-consumer -49d5c7e81cff rdlabengpa/ids_uc_data_app_platoon_pip:v1.0.0 "java -jar pip.jar" About a minute ago Up About a minute 0/tcp uc-dataapp-pip-consumer -f8de175d72f0 rdlabengpa/ids_be_data_app:v0.2.7 "/bin/sh -c 'java -j…" About a minute ago Up About a minute (unhealthy) 0.0.0.0:8183->8183/tcp, 0.0.0.0:9000->9000/tcp be-dataapp-provider +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +bc693e1fdb90 rdlabengpa/ids_execution_core_container:1.14.2 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 0.0.0.0:8087->8086/tcp, :::8087->8086/tcp, 0.0.0.0:8091->8449/tcp, :::8091->8449/tcp, 0.0.0.0:8890->8889/tcp, :::8890->8889/tcp ecc-consumer +28dc87213f68 rdlabengpa/ids_be_data_app:0.3.1 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 0.0.0.0:8184->8183/tcp, :::8184->8183/tcp, 0.0.0.0:9001->9000/tcp, :::9001->9000/tcp be-dataapp-consumer +9eb157ceb37b rdlabengpa/ids_be_data_app:0.3.1 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 0.0.0.0:8183->8183/tcp, :::8183->8183/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp be-dataapp-provider +44bc21187460 rdlabengpa/ids_execution_core_container:1.14.2 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 0.0.0.0:8086->8086/tcp, :::8086->8086/tcp, 0.0.0.0:8889->8889/tcp, :::8889->8889/tcp, 0.0.0.0:8090->8449/tcp, :::8090->8449/tcp ecc-provider +b3f4cdb77ed6 rdlabengpa/ids_uc_data_app_platoon:1.7.3 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 8080/tcp uc-dataapp-consumer +a36748901ce1 rdlabengpa/ids_uc_data_app_platoon_pip:v1.0.0 "java -jar pip.jar" 3 hours ago Up 3 hours 0/tcp uc-dataapp-pip-provider +d6f77ad9762d rdlabengpa/ids_uc_data_app_platoon:1.7.3 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 8080/tcp uc-dataapp-provider +bb0bb9668931 rdlabengpa/ids_uc_data_app_platoon_pip:v1.0.0 "java -jar pip.jar" 3 hours ago Up 3 hours 0/tcp uc-dataapp-pip-consumer ``` diff --git a/doc/TRUEConnector/volumes.md b/doc/TRUEConnector/volumes.md index fb6f329..728fb5e 100644 --- a/doc/TRUEConnector/volumes.md +++ b/doc/TRUEConnector/volumes.md @@ -3,20 +3,18 @@ Following docker volumes will be created ``` -ecc_provider_log -ecc_provider_sd +ecc_provider_data uc_provider_data -be_dataapp_data_provider +be_dataapp_provider_data -ecc_consumer_log -ecc_consumer_sd +ecc_consumer_data uc_consumer_data -be_dataapp_data_consumer +be_dataapp_consumer_data ``` Those volumes will store data needed for corresponding service, like log files, self description file, Usage Control H2 database (default configuration) and dataApp resource storage. -Volume `be_dataapp_data_provider` is external volume. In order to create it, please execute the script `prepopulate_be_dataapp_data_provider.sh` running next command: +Volume `be_dataapp_provider_data` is external volume. In order to create it, please execute the script `prepopulate_be_dataapp_data_provider.sh` running next command: ``` sudo ./prepopulate_be_dataapp_data_provider.sh diff --git a/doc/contractNegotiation/contract_agreement_request.md b/doc/contractNegotiation/contract_agreement_request.md index 22bea21..6cc8134 100644 --- a/doc/contractNegotiation/contract_agreement_request.md +++ b/doc/contractNegotiation/contract_agreement_request.md @@ -7,9 +7,9 @@ Postman collection will preset required fields from previous request. Multipart form - Contract Agreement request ``` -curl --location --request POST 'https://localhost:8184/proxy' \ +curl --location -k --request POST 'https://localhost:8184/proxy' \ --header 'Content-Type: application/json' \ ---header 'Authorization: Basic cHJveHk6cGFzc3dvcmQ=' \ +--header 'Authorization: Basic aWRzVXNlcjpwYXNzd29yZA==' \ --data-raw '{ "multipart": "form", "Forward-To": "https://ecc-provider:8889/data", diff --git a/doc/contractNegotiation/contract_request_message.md b/doc/contractNegotiation/contract_request_message.md index 4db12ea..da709b9 100644 --- a/doc/contractNegotiation/contract_request_message.md +++ b/doc/contractNegotiation/contract_request_message.md @@ -7,9 +7,9 @@ Contract Request Message is initial message sent in Contract Negotiation flow. I Multipart form - Contract Request Message ``` -curl --location --request POST 'https://localhost:8184/proxy' \ +curl --location -k --request POST 'https://localhost:8184/proxy' \ --header 'Content-Type: application/json' \ ---header 'Authorization: Basic cHJveHk6cGFzc3dvcmQ=' \ +--header 'Authorization: Basic aWRzVXNlcjpwYXNzd29yZA==' \ --data-raw '{ "multipart": "form", "Forward-To": "https://ecc-provider:8889/data", diff --git a/doc/contractNegotiation/description_request_message.md b/doc/contractNegotiation/description_request_message.md index 11d0065..7189cde 100644 --- a/doc/contractNegotiation/description_request_message.md +++ b/doc/contractNegotiation/description_request_message.md @@ -7,9 +7,9 @@ Before start of negotiation process, Description Request Message is sent to iden Multipart form - Description Request Message ``` -curl --location 'https://localhost:8184/proxy' \ +curl --location -k 'https://localhost:8184/proxy' \ --header 'Content-Type: application/json' \ ---header 'Authorization: Basic cHJveHk6cGFzc3dvcmQ=' \ +--header 'Authorization: Basic aWRzVXNlcjpwYXNzd29yZA==' \ --data '{ "multipart": "form", "Forward-To": "https://ecc-provider:8889/data", diff --git a/doc/contractNegotiation/get_offered_resource.md b/doc/contractNegotiation/get_offered_resource.md index f8a4ec0..1b26daa 100644 --- a/doc/contractNegotiation/get_offered_resource.md +++ b/doc/contractNegotiation/get_offered_resource.md @@ -9,9 +9,9 @@ We can query the resource with ArtifactRequestMessage: Multipart form - Artifact Request Message ``` -curl --location --request POST 'https://localhost:8184/proxy' \ +curl --location -k --request POST 'https://localhost:8184/proxy' \ --header 'Content-Type: application/json' \ ---header 'Authorization: Basic Y29ubmVjdG9yOnBhc3N3b3Jk' \ +--header 'Authorization: Basic aWRzVXNlcjpwYXNzd29yZA==' \ --data-raw '{ "multipart": "form", "Forward-To": "https://ecc-provider:8889/data", diff --git a/doc/contractNegotiation/get_offered_resource_granted.md b/doc/contractNegotiation/get_offered_resource_granted.md index c82cc29..6bab222 100644 --- a/doc/contractNegotiation/get_offered_resource_granted.md +++ b/doc/contractNegotiation/get_offered_resource_granted.md @@ -9,9 +9,9 @@ Postman collection will preset required fields from previous request. Multipart Form - Artifact Request Message ``` -curl --location --request POST 'http://localhost:8184/proxy' \ +curl --location -k --request POST 'http://localhost:8184/proxy' \ --header 'Content-Type: application/json' \ ---header 'Authorization: Basic cHJveHk6cGFzc3dvcmQ=' \ +--header 'Authorization: Basic aWRzVXNlcjpwYXNzd29yZA==' \ --data-raw '{ "multipart": "form", "Forward-To": "http://ecc-provider:8889/data", diff --git a/doc/cosign.md b/doc/cosign.md index d4900de..ba2a693 100644 --- a/doc/cosign.md +++ b/doc/cosign.md @@ -4,11 +4,11 @@ Docker images that are part of the TRUE Connector are signed using [cosign](http Signed images starts with following versions: -**rdlabengpa/ids\_execution\_core\_container:v1.14.1**\ +**rdlabengpa/ids\_execution\_core\_container:v1.14.2**\ -**rdlabengpa/ids\_be\_data\_app:v0.3.0**\ +**rdlabengpa/ids\_be\_data\_app:v0.3.1**\ -**rdlabengpa/ids\_uc\_data\_app\_platoon:v1.7.2**\ +**rdlabengpa/ids\_uc\_data\_app\_platoon:v1.7.3**\ **rdlabengpa/ids\_uc\_data\_app\_platoon\_pip:v1.0.0**\ @@ -16,9 +16,9 @@ Signed images starts with following versions: Once images are downloaded, you can verify the signature by executing following command, (trueconn.pub file can be found in the root of this repo) and response should be like following ``` -cosign verify --key trueconn.pub rdlabengpa/ids_execution_core_container:v1.14.1 +cosign verify --key trueconn.pub rdlabengpa/ids_execution_core_container:v1.14.2 -Verification for index.docker.io/rdlabengpa/ids_execution_core_container:v1.14.1 -- +Verification for index.docker.io/rdlabengpa/ids_execution_core_container:v1.14.2 -- The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key @@ -40,9 +40,9 @@ The following checks were performed on each of these signatures: ``` ``` -cosign verify --key trueconn.pub rdlabengpa/ids_be_data_app:v0.3.0 +cosign verify --key trueconn.pub rdlabengpa/ids_be_data_app:v0.3.1 -Verification for index.docker.io/rdlabengpa/ids_be_data_app:v0.3.0 -- +Verification for index.docker.io/rdlabengpa/ids_be_data_app:v0.3.1 -- The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key @@ -64,9 +64,9 @@ The following checks were performed on each of these signatures: ``` ``` -cosign verify --key trueconn.pub rdlabengpa/ids_uc_data_app_platoon:v1.7.2 +cosign verify --key trueconn.pub rdlabengpa/ids_uc_data_app_platoon:v1.7.3 -Verification for index.docker.io/rdlabengpa/ids_uc_data_app_platoon:v1.7.2 -- +Verification for index.docker.io/rdlabengpa/ids_uc_data_app_platoon:v1.7.3 -- The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key diff --git a/doc/exchange-data.md b/doc/exchange-data.md index 6e88603..6c43b6d 100644 --- a/doc/exchange-data.md +++ b/doc/exchange-data.md @@ -7,9 +7,9 @@ With default configuration, you can use following curl command, to get data from Multipart Form request ``` -curl --location 'https://localhost:8184/proxy' \ +curl --location -k 'https://localhost:8184/proxy' \ --header 'Content-Type: application/json' \ ---header 'Authorization: Basic cHJveHk6cGFzc3dvcmQ=' \ +--header 'Authorization: Basic aWRzVXNlcjpwYXNzd29yZA==' \ --data '{ "multipart": "form", "Forward-To": "https://ecc-provider:8889/data", From f6457364e434e7c5a3b273784ee66f25af9141cc Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Wed, 9 Aug 2023 19:47:14 +0200 Subject: [PATCH 20/33] Add -k to the first curl command, and fix Authorization value --- doc/TEST_API.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/TEST_API.md b/doc/TEST_API.md index 805169d..73b7108 100644 --- a/doc/TEST_API.md +++ b/doc/TEST_API.md @@ -17,9 +17,9 @@ _transferContract_ - of type String, should be formatted as URI\ Example request: ``` -curl --location --request POST 'https://localhost:8184/proxy' \ +curl --location -k --request POST 'https://localhost:8184/proxy' \ --header 'Content-Type: application/json' \ ---header 'Authorization: Basic Y29ubmVjdG9yOnBhc3N3b3Jk' \ +--header 'Authorization: Basic aWRzVXNlcjpwYXNzd29yZA==' \ --data-raw '{ "multipart": "form", "Forward-To": "https://connectora:8080/api/ids/data", From 02361f7eeec2e4a83905fa4197f47ec87705391a Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Tue, 22 Aug 2023 08:56:13 +0200 Subject: [PATCH 21/33] Modify current versions of modules in doc --- doc/PLATOON_USAGE_CONTROL.md | 2 +- doc/TRUEConnector/component-overview.md | 2 +- doc/advancedConfiguration/broker.md | 2 +- doc/advancedConfiguration/extendedjwt.md | 2 +- doc/contributingTC.md | 2 +- doc/cosign.md | 6 +++--- doc/rest_api/REST_API.md | 4 ++-- doc/security.md | 2 +- doc/selfDescriptionAPI/self-description-API.md | 2 +- docker-compose.yml | 4 ++-- 10 files changed, 14 insertions(+), 14 deletions(-) diff --git a/doc/PLATOON_USAGE_CONTROL.md b/doc/PLATOON_USAGE_CONTROL.md index 8866ab0..c82140c 100644 --- a/doc/PLATOON_USAGE_CONTROL.md +++ b/doc/PLATOON_USAGE_CONTROL.md @@ -105,5 +105,5 @@ POSTGRES_DB=usagecontrol_consumer # Usage control examples -For more information and examples of policies compatible with Platoon UC app, please check [README](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/blob/1.7.2/README.md) +For more information and examples of policies compatible with Platoon UC app, please check [README](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/blob/1.7.4/README.md) diff --git a/doc/TRUEConnector/component-overview.md b/doc/TRUEConnector/component-overview.md index 7f530b4..b54c0c1 100644 --- a/doc/TRUEConnector/component-overview.md +++ b/doc/TRUEConnector/component-overview.md @@ -6,7 +6,7 @@ TRUE Connector is build using Java11, and use following libraries: | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | [Execution core container](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/releases/tag/1.14.2) | 1.14.2 | | [Basic data app](https://github.com/Engineering-Research-and-Development/true-connector-basic_data_app/releases/tag/0.3.1) | 0.3.1 | -| [Usage control app](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/releases/tag/1.7.3) | 1.7.3 | +| [Usage control app](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/releases/tag/1.7.3) | 1.7.4 | | [Pip](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/tree/1.7.3/Docker_Tecnalia_DataUsage/pip) | 1.0.0 | | [Multipart Message Library](https://github.com/Engineering-Research-and-Development/true-connector-multipart_message_library/releases/tag/1.0.17) | 1.0.17 | | [Websocket Message Streamer](https://github.com/Engineering-Research-and-Development/true-connector-websocket_message_streamer/releases/tag/1.0.17) | 1.0.17 | diff --git a/doc/advancedConfiguration/broker.md b/doc/advancedConfiguration/broker.md index fa9aa18..1fc4c97 100644 --- a/doc/advancedConfiguration/broker.md +++ b/doc/advancedConfiguration/broker.md @@ -13,4 +13,4 @@ TRUE Connector can register itself on startup, and also unregister when shutting application.selfdescription.registrateOnStartup=true ``` -Information on how TRUE Connector can interact with Broker, can be found on following [link](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.1/doc/BROKER.md) +Information on how TRUE Connector can interact with Broker, can be found on following [link](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.2/doc/BROKER.md) diff --git a/doc/advancedConfiguration/extendedjwt.md b/doc/advancedConfiguration/extendedjwt.md index e516108..5726126 100644 --- a/doc/advancedConfiguration/extendedjwt.md +++ b/doc/advancedConfiguration/extendedjwt.md @@ -1,3 +1,3 @@ ### Extended jwt validation -TRUE Connector can check additional claims from jwToken. For more information please check the [following link](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.1/doc/TRANSPORTCERTSSHA256.md) +TRUE Connector can check additional claims from jwToken. For more information please check the [following link](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.2/doc/TRANSPORTCERTSSHA256.md) diff --git a/doc/contributingTC.md b/doc/contributingTC.md index ca25c90..71e85da 100644 --- a/doc/contributingTC.md +++ b/doc/contributingTC.md @@ -18,4 +18,4 @@ should at least include the following information: * Steps to reproduce (system specs included) * Relevant logs and/or media (optional): e.g. an image -For more details about branches, naming conventions and some suggestions, take a look at following [Developer instructions](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/tree/1.14.1#developer-guide-section) \ No newline at end of file +For more details about branches, naming conventions and some suggestions, take a look at following [Developer instructions](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/tree/1.14.2#developer-guide-section) \ No newline at end of file diff --git a/doc/cosign.md b/doc/cosign.md index ba2a693..864bd9f 100644 --- a/doc/cosign.md +++ b/doc/cosign.md @@ -8,7 +8,7 @@ Signed images starts with following versions: **rdlabengpa/ids\_be\_data\_app:v0.3.1**\ -**rdlabengpa/ids\_uc\_data\_app\_platoon:v1.7.3**\ +**rdlabengpa/ids\_uc\_data\_app\_platoon:v1.7.4**\ **rdlabengpa/ids\_uc\_data\_app\_platoon\_pip:v1.0.0**\ @@ -64,9 +64,9 @@ The following checks were performed on each of these signatures: ``` ``` -cosign verify --key trueconn.pub rdlabengpa/ids_uc_data_app_platoon:v1.7.3 +cosign verify --key trueconn.pub rdlabengpa/ids_uc_data_app_platoon:v1.7.4 -Verification for index.docker.io/rdlabengpa/ids_uc_data_app_platoon:v1.7.3 -- +Verification for index.docker.io/rdlabengpa/ids_uc_data_app_platoon:v1.7.4 -- The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key diff --git a/doc/rest_api/REST_API.md b/doc/rest_api/REST_API.md index fb3bc1c..a2ce7ee 100644 --- a/doc/rest_api/REST_API.md +++ b/doc/rest_api/REST_API.md @@ -3,7 +3,7 @@ The TRUE Connector will use two protocols (http and https) as described by the Docker Compose File. Overview of all available endpoints: -*NOTE* Endpoints are protected with credentials, for more details, please check [this link](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.1/doc/SECURITY.md) +*NOTE* Endpoints are protected with credentials, for more details, please check [this link](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.2/doc/SECURITY.md) | **Method** | **Endpoint** | **Usage** | **Returns** | | ---------- | ------------ | --------- | ----------- | @@ -38,5 +38,5 @@ Swagger UI for representation CRUD operations: ![Resource representation](Resource_Representation_Swagger.jpg "Resource representation swagger API") -On following [link](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.1/doc/SELF_DESCRIPTION.md), you can find more detailed explanation of endpoints, with example requests. +On following [link](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.2/doc/SELF_DESCRIPTION.md), you can find more detailed explanation of endpoints, with example requests. diff --git a/doc/security.md b/doc/security.md index aa6667c..89006fb 100644 --- a/doc/security.md +++ b/doc/security.md @@ -47,7 +47,7 @@ Once certificate is generated, following instruction from previous link, you can TRUE Connector has several ways to check the integrity: * [Docker cosing check](cosign.md) - * [Healthcheck](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.1/doc/HEALTHCHECK.md) + * [Healthcheck](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.2/doc/HEALTHCHECK.md) * Verification of the components itself, that will check if current version of subcomponent is verified or not; Each component (Execution Core Container, Basic DataApp and Platoon Usage Control) should on startup log somethign like following: diff --git a/doc/selfDescriptionAPI/self-description-API.md b/doc/selfDescriptionAPI/self-description-API.md index d0e53fa..22165af 100644 --- a/doc/selfDescriptionAPI/self-description-API.md +++ b/doc/selfDescriptionAPI/self-description-API.md @@ -1,6 +1,6 @@ ## Self Description API -To manage your Self Description Document please check following [link](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.1/doc/SELF\_DESCRIPTION.md) +To manage your Self Description Document please check following [link](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.2/doc/SELF\_DESCRIPTION.md) You can copy existing valid self-description.json document to following location **/ecc\_resources\_consumer** or **/ecc\_resources\_provider** folders, for consumer or provider. diff --git a/docker-compose.yml b/docker-compose.yml index 7497fba..575edd5 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -48,7 +48,7 @@ services: - "ecc-provider:172.17.0.1" uc-dataapp-provider: - image: rdlabengpa/ids_uc_data_app_platoon:v1.7.3 + image: rdlabengpa/ids_uc_data_app_platoon:v1.7.4 deploy: resources: limits: @@ -169,7 +169,7 @@ services: - "ecc-consumer:172.17.0.1" uc-dataapp-consumer: - image: rdlabengpa/ids_uc_data_app_platoon:v1.7.3 + image: rdlabengpa/ids_uc_data_app_platoon:v1.7.4 deploy: resources: limits: From 360c34199d5635a41cabf9ad3e8bfc63f1953ddb Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Tue, 22 Aug 2023 09:29:13 +0200 Subject: [PATCH 22/33] Update cosigin vales to the lastest versions of modules --- doc/cosign.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/cosign.md b/doc/cosign.md index 864bd9f..00dfefe 100644 --- a/doc/cosign.md +++ b/doc/cosign.md @@ -30,7 +30,7 @@ The following checks were performed on each of these signatures: "docker-reference": "index.docker.io/rdlabengpa/ids_execution_core_container" }, "image": { - "docker-manifest-digest": "sha256:13db7775bc9bfb1f5b85f1d4930f64665993aa06567c379e4ebfd73be10a9fb4" + "docker-manifest-digest": "sha256:40416b055eb7f8841f58be6501a55364583cec25cf87768747cb117ba5f872d0" }, "type": "cosign container image signature" }, @@ -54,7 +54,7 @@ The following checks were performed on each of these signatures: "docker-reference": "index.docker.io/rdlabengpa/ids_be_data_app" }, "image": { - "docker-manifest-digest": "sha256:718eded2099437494e9d2995205282c8d66465a267c0188a838d66f6a3b05bac" + "docker-manifest-digest": "sha256:0f722e89aabf52e6b7e40791f4579566b03789c9a5c5ee462f23553eba13cfd4" }, "type": "cosign container image signature" }, @@ -78,7 +78,7 @@ The following checks were performed on each of these signatures: "docker-reference": "index.docker.io/rdlabengpa/ids_uc_data_app_platoon" }, "image": { - "docker-manifest-digest": "sha256:12e83927ec001d65ad68e3f7f53572cde92cccc5d933a936599d1f133b342ada" + "docker-manifest-digest": "sha256:f8a0f0a221b9b7c880417e49f4da9064725490ab8f905a7c4ad833276e851f40" }, "type": "cosign container image signature" }, From 12b8d827a85029a199973a418dc46b4dd1d3be2d Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Tue, 22 Aug 2023 10:21:37 +0200 Subject: [PATCH 23/33] Fix link for UC App --- doc/TRUEConnector/component-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/TRUEConnector/component-overview.md b/doc/TRUEConnector/component-overview.md index b54c0c1..fe3162c 100644 --- a/doc/TRUEConnector/component-overview.md +++ b/doc/TRUEConnector/component-overview.md @@ -6,7 +6,7 @@ TRUE Connector is build using Java11, and use following libraries: | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | [Execution core container](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/releases/tag/1.14.2) | 1.14.2 | | [Basic data app](https://github.com/Engineering-Research-and-Development/true-connector-basic_data_app/releases/tag/0.3.1) | 0.3.1 | -| [Usage control app](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/releases/tag/1.7.3) | 1.7.4 | +| [Usage control app](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/releases/tag/1.7.4) | 1.7.4 | | [Pip](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/tree/1.7.3/Docker_Tecnalia_DataUsage/pip) | 1.0.0 | | [Multipart Message Library](https://github.com/Engineering-Research-and-Development/true-connector-multipart_message_library/releases/tag/1.0.17) | 1.0.17 | | [Websocket Message Streamer](https://github.com/Engineering-Research-and-Development/true-connector-websocket_message_streamer/releases/tag/1.0.17) | 1.0.17 | From b097a88e7e2f767dd22ae9faff61212279047475 Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Tue, 22 Aug 2023 10:21:53 +0200 Subject: [PATCH 24/33] Fix link to usageControl.md --- doc/contractNegotiation/contract-negotiation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/contractNegotiation/contract-negotiation.md b/doc/contractNegotiation/contract-negotiation.md index 6fe7458..8defff8 100644 --- a/doc/contractNegotiation/contract-negotiation.md +++ b/doc/contractNegotiation/contract-negotiation.md @@ -1,6 +1,6 @@ ## Contract Negotiation - simple flow -Usage Control is disabled by default. If you want to enable it (mandatory for contract negotiation), please check ["Enabling usage control"](usagecontrol.md). +Usage Control is disabled by default. If you want to enable it (mandatory for contract negotiation), please check ["Enabling usage control"](../advancedConfiguration/usagecontrol.md). If mandatory, for other connectors, you can perform contract negotiation with other connector (not TRUE Connector) or with TRUE Connector. There is default contract offer that will be sent if ContractRequestMessage is received. It will allow consuming of resource. From c2fb9c230b86e5ef0821f72cb925e8e520c3b23a Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Tue, 22 Aug 2023 11:16:21 +0200 Subject: [PATCH 25/33] Delete the SERVER_SSL_ENABLED description --- doc/modifyingConfiguration/ssl.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/doc/modifyingConfiguration/ssl.md b/doc/modifyingConfiguration/ssl.md index c78a68d..f3e07db 100644 --- a/doc/modifyingConfiguration/ssl.md +++ b/doc/modifyingConfiguration/ssl.md @@ -9,9 +9,3 @@ KEY_PASSWORD={your_certificate_key} KEYSTORE_PASSWORD={your_certificate_password} ALIAS={your_certificate_alias} ``` - -If you want to use http and not https, simply disable following property - -``` -SERVER_SSL_ENABLED=false -``` From 52cb99716142ab3e6e427cfd913001c43f380abe Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Tue, 22 Aug 2023 11:16:38 +0200 Subject: [PATCH 26/33] Update ROADMAP.md --- doc/roadmap.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/roadmap.md b/doc/roadmap.md index faf2c68..7120712 100644 --- a/doc/roadmap.md +++ b/doc/roadmap.md @@ -5,8 +5,8 @@ This list follows no timeline, instead, individual tasks can be priority-assigne | Priority | Task | Status | Note | |:---:|:------------|:------------|:------------| | 1 | Dataspace Protocol investigation | Ongoing | | -| 1 | Dataspace Protocol (compliance with EDC)| | | -| 2 | User Interface | | | +| 1 | Dataspace Protocol (compliance with EDC)| Ongoing | | +| 2 | User Interface | Ongoing | | | 1 | Audit logging | Done | | | 3 | Kubernetes deployment | Done | | | 2 | Basic Usage Control | Done | | \ No newline at end of file From 6e3eefeb91212164ff5411e87c0ecd2147f1f7b9 Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Tue, 22 Aug 2023 14:01:35 +0200 Subject: [PATCH 27/33] Modify command --- prepopulate_be_dataapp_data_provider.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prepopulate_be_dataapp_data_provider.sh b/prepopulate_be_dataapp_data_provider.sh index f0e7827..82b1ceb 100755 --- a/prepopulate_be_dataapp_data_provider.sh +++ b/prepopulate_be_dataapp_data_provider.sh @@ -19,4 +19,4 @@ docker run --rm -v "$(pwd)/be-dataapp_data_provider:/source_data" -v "be_dataapp docker run --rm -v "be_dataapp_provider_data:/target_data" alpine sh -c "mkdir -p /target_data/log/dataapp && chown -R nobody:nogroup /target_data/log/dataapp" # Change the ownership of the Docker volume contents to 'nobody:nogroup' -docker run --rm -v be_dataapp_provider_data:/target_data alpine chown -R nobody:nogroup /target_data +docker run --rm -v "be_dataapp_provider_data:/target_data" alpine sh -c "chown -R nobody:nogroup /target_data" From 2b758f7a9eba147d33e3119bac6271a70e00d60d Mon Sep 17 00:00:00 2001 From: David Jovanovic Date: Tue, 22 Aug 2023 14:25:24 +0200 Subject: [PATCH 28/33] updated docs --- .env | 10 ++----- doc/TRUEConnector/start-stop.md | 30 ++++++++++++++++++--- doc/TRUEConnector/volumes.md | 16 ++--------- doc/backupAndUpdate/backup.md | 8 +++--- prepopulate_be_dataapp_data_provider_win.sh | 18 +++++++++++++ 5 files changed, 52 insertions(+), 30 deletions(-) create mode 100644 prepopulate_be_dataapp_data_provider_win.sh diff --git a/.env b/.env index 86c32da..d5fe922 100644 --- a/.env +++ b/.env @@ -2,9 +2,6 @@ COMPOSE_PROJECT_NAME=TRUE_Connector BROKER_URL=https://broker.ids.isst.fraunhofer.de/infrastructure -DISABLE_SSL_VALIDATION=true -#Enable HTTPS -SERVER_SSL_ENABLED=true #SSL settings KEYSTORE_NAME=ssl-server.jks KEY_PASSWORD=changeit @@ -12,15 +9,12 @@ KEYSTORE_PASSWORD=changeit ALIAS=execution-core-container #TRUSTORE (used also by IDSCP2) -TRUSTORE_NAME= -TRUSTORE_PASSWORD= +TRUSTORE_NAME=truststoreEcc.jks +TRUSTORE_PASSWORD=allpassword CACHE_TOKEN=false FETCH_TOKEN_ON_STARTUP=false -#Camel -REST_ENABLE_HTTPS=true - # REST Communication type between ECC - mixed | form | http-header MULTIPART_ECC=form diff --git a/doc/TRUEConnector/start-stop.md b/doc/TRUEConnector/start-stop.md index 2f5ccc8..c77bbb9 100644 --- a/doc/TRUEConnector/start-stop.md +++ b/doc/TRUEConnector/start-stop.md @@ -2,19 +2,44 @@ To setup the TRUE connector for starting container, execute the following command: +# For Linux + ``` sudo ./prepopulate_be_dataapp_data_provider.sh ``` With this command, you will create external `be_dataapp_data_provider` volume. -***NOTE:*** If you're using Linux, check if script is executable, if not, run the next command: +***NOTE:*** Check if script is executable, if not, run the next command: ``` chmod +x prepopulate_be_dataapp_data_provider.sh ``` +# For Windows + +In the *prepopulate_be_dataapp_data_provider_win.sh* change the *FULL_PATH* with the full path where the TRUE Connector is located: + +``` +docker run --rm -v "FULL_PATH/be-dataapp_data_provider:/source_data" -v "be_dataapp_provider_data:/target_data" alpine sh -c "cp -r /source_data/* /target_data/datalake/" +``` + +For example: + +``` +docker run --rm -v "C:/true-connector/be-dataapp_data_provider:/source_data" -v "be_dataapp_provider_data:/target_data" alpine sh -c "cp -r /source_data/* /target_data/datalake/" +``` + +And run the next command: + +``` +sh prepopulate_be_dataapp_data_provider_win.sh +``` + + +# For both systems + To start docker container, open terminal and execute following command: ``` @@ -92,9 +117,6 @@ Consumer connector: _https://localhost:8091/about/version_ -or self description document:\ -_https://localhost:8090/_ - Ctrl+C is used to exit from log inspection (you will be returned to the terminal). To stop containers, execute following: diff --git a/doc/TRUEConnector/volumes.md b/doc/TRUEConnector/volumes.md index 728fb5e..d87bd14 100644 --- a/doc/TRUEConnector/volumes.md +++ b/doc/TRUEConnector/volumes.md @@ -14,18 +14,6 @@ be_dataapp_consumer_data Those volumes will store data needed for corresponding service, like log files, self description file, Usage Control H2 database (default configuration) and dataApp resource storage. -Volume `be_dataapp_provider_data` is external volume. In order to create it, please execute the script `prepopulate_be_dataapp_data_provider.sh` running next command: +Volume `be_dataapp_provider_data` is an external volume which was created following the [start-stop.md](start-stop.md) . -``` -sudo ./prepopulate_be_dataapp_data_provider.sh - -``` - -***NOTE:*** If you're using Linux, check if script is executable, if not, run the next command: - -``` -chmod +x prepopulate_be_dataapp_data_provider.sh - -``` - -Running this script, all files present on the host in folder `be-dataapp_data_provder` will be present in volume, so if you need to have some files present in volume, please put them in this folder before running script, and all of them will be present in `DataApp Provider` application. +Following the instructions, all files present on the host in the folder `be-dataapp_data_provder` will be present in the volume, so if you need to have some files present in volume, please put them in this folder before running the script and all of them will be present in `DataApp Provider` application. diff --git a/doc/backupAndUpdate/backup.md b/doc/backupAndUpdate/backup.md index 6e03e47..343e081 100644 --- a/doc/backupAndUpdate/backup.md +++ b/doc/backupAndUpdate/backup.md @@ -7,13 +7,13 @@ To backup and restore data we will be using the official Docker documentation as In order to backup your data you can use the following code snippet: ``` -docker run --rm --volumes-from ecc-consumer -v %cd%:/backup ubuntu tar cvf /backup/backup.tar /home/nobody/data/log +docker run --rm --volumes-from ecc-consumer -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /home/nobody/data/log ``` Let me explain the options: - ecc-consumer - container with the data that you want to backup - - %cd% - current directory in the command prompt ( same as $(pwd) on Linux); the directory where you want the backup to be located + - $(pwd) - current directory in the command prompt (use %cd% on Windows); the directory where you want the backup to be located - /backup.tar - name of the backup archive - /home/nobody/data/log - directory or file from the container that you wish to backup @@ -24,13 +24,13 @@ After the process finishes you will find a .tar file with the data. Restoring the data is done in a similar way: ``` -docker run --rm --volumes-from ecc-consumer -v %cd%:/backup ubuntu bash -c "cd /home && tar xvf /backup/backup.tar --strip 1" +docker run --rm --volumes-from ecc-consumer -v $(pwd):/backup ubuntu bash -c "cd /home && tar xvf /backup/backup.tar --strip 1" ``` The options are: - ecc-consumer - container where you want to restore the data - - %cd% - current directory in the command prompt ( same as $(pwd) on Linux); the directory where the backup is located + - $(pwd) - current directory in the command prompt (use %cd% on Windows); the directory where the backup is located - /home - directory where the data will be restored - /backup.tar - name of the backup archive diff --git a/prepopulate_be_dataapp_data_provider_win.sh b/prepopulate_be_dataapp_data_provider_win.sh new file mode 100644 index 0000000..e360168 --- /dev/null +++ b/prepopulate_be_dataapp_data_provider_win.sh @@ -0,0 +1,18 @@ + +# Remove the existing Docker volume (ignoring errors if it doesn't exist) +docker volume rm be_dataapp_provider_data || true + +# Replace 'be-be_dataapp_provider_data' with the actual volume name you want to create +docker volume create be_dataapp_provider_data + +# Create the 'datalake' directory inside the 'be_dataapp_provider_data' volume with the desired ownership +docker run --rm -v "be_dataapp_provider_data:/target_data" alpine sh -c "mkdir -p /target_data/datalake && chown -R nobody:nogroup /target_data/datalake" + +# Copy data from the 'be-dataapp_data_provider' folder to the 'target_data/datalake' directory inside the Docker volume +docker run --rm -v "FULL_PATH/be-dataapp_data_provider:/source_data" -v "be_dataapp_provider_data:/target_data" alpine sh -c "cp -r /source_data/* /target_data/datalake/" + +# Create the 'log' directory inside the 'be_dataapp_provider_data' volume with the desired ownership +docker run --rm -v "be_dataapp_provider_data:/target_data" alpine sh -c "mkdir -p /target_data/log/dataapp && chown -R nobody:nogroup /target_data/log/dataapp" + +# Change the ownership of the Docker volume contents to 'nobody:nogroup' +docker run --rm -v "be_dataapp_provider_data:/target_data" alpine sh -c "chown -R nobody:nogroup /target_data" \ No newline at end of file From 923ef8b705f6a30c4125129b743d800beb7b4c2d Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Tue, 22 Aug 2023 16:01:57 +0200 Subject: [PATCH 29/33] Update sha values in cosign.md --- doc/cosign.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/cosign.md b/doc/cosign.md index 00dfefe..937bd58 100644 --- a/doc/cosign.md +++ b/doc/cosign.md @@ -54,7 +54,7 @@ The following checks were performed on each of these signatures: "docker-reference": "index.docker.io/rdlabengpa/ids_be_data_app" }, "image": { - "docker-manifest-digest": "sha256:0f722e89aabf52e6b7e40791f4579566b03789c9a5c5ee462f23553eba13cfd4" + "docker-manifest-digest": "sha256:905071836b33b7af28727f53574257a218a9b7c93c476f7c1bcaa07b0c7ac24a" }, "type": "cosign container image signature" }, @@ -78,7 +78,7 @@ The following checks were performed on each of these signatures: "docker-reference": "index.docker.io/rdlabengpa/ids_uc_data_app_platoon" }, "image": { - "docker-manifest-digest": "sha256:f8a0f0a221b9b7c880417e49f4da9064725490ab8f905a7c4ad833276e851f40" + "docker-manifest-digest": "sha256:00b61c089c106750ed8e3f5d6761f9188c5c44276b47d85cef63d8c1df3e37f0" }, "type": "cosign container image signature" }, From 7f292c83d61baac7466e12e171131868f3238a16 Mon Sep 17 00:00:00 2001 From: Marko Stojanovic Date: Tue, 22 Aug 2023 16:09:38 +0200 Subject: [PATCH 30/33] Modify ecc sha --- doc/cosign.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/cosign.md b/doc/cosign.md index 937bd58..c0e24d6 100644 --- a/doc/cosign.md +++ b/doc/cosign.md @@ -30,7 +30,7 @@ The following checks were performed on each of these signatures: "docker-reference": "index.docker.io/rdlabengpa/ids_execution_core_container" }, "image": { - "docker-manifest-digest": "sha256:40416b055eb7f8841f58be6501a55364583cec25cf87768747cb117ba5f872d0" + "docker-manifest-digest": "sha256:d28ec86e5ee3c9c5b992dd3445fa3301d77a83b6c244b7a8577f2b4e7b8f5d52" }, "type": "cosign container image signature" }, From d1ee7d3690c27535d3c6eaf48c19595c43287b44 Mon Sep 17 00:00:00 2001 From: David Jovanovic Date: Tue, 22 Aug 2023 16:31:42 +0200 Subject: [PATCH 31/33] updated postman collection --- TRUE Connector v1.postman_collection.json | 151 +++++++++++++++++++++- 1 file changed, 147 insertions(+), 4 deletions(-) diff --git a/TRUE Connector v1.postman_collection.json b/TRUE Connector v1.postman_collection.json index 2d69d11..fcb75e9 100644 --- a/TRUE Connector v1.postman_collection.json +++ b/TRUE Connector v1.postman_collection.json @@ -1,9 +1,9 @@ { "info": { - "_postman_id": "d69b4d98-a2fb-44a4-826d-99da07c6cb54", + "_postman_id": "de10144c-f87c-4bd2-84e1-5b8247e5266a", "name": "TRUE Connector v1", "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json", - "_exporter_id": "2326106" + "_exporter_id": "21815221" }, "item": [ { @@ -12,6 +12,54 @@ { "name": "ContractOffer", "item": [ + { + "name": "SelfDescription", + "event": [ + { + "listen": "test", + "script": { + "exec": [ + "var jsonData = pm.response.json();\r", + "pm.response.to.be.ok;\r", + "//store applicants self-description for further tests\r", + "pm.collectionVariables.set(\"APPLICANT_SELF_DESCRIPTION\", jsonData)\r", + "\r", + "pm.test(\"ContractOffer\", function () {\r", + " // catalog\r", + " pm.expect(jsonData[\"ids:resourceCatalog\"][0][\"@id\"]).to.exist;\r", + " pm.collectionVariables.set(\"resourceCatalog\", jsonData[\"ids:resourceCatalog\"][0][\"@id\"]) \r", + " // resource\r", + " pm.expect(jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"@id\"]).to.exist;\r", + " pm.collectionVariables.set(\"offeredResource\", jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"@id\"])\r", + " // contractOffer\r", + " pm.expect(jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"ids:contractOffer\"][0][\"@id\"]).to.exist;\r", + " pm.collectionVariables.set(\"contractOffer\", jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"ids:contractOffer\"][0][\"@id\"])\r", + " // representation\r", + " pm.expect(jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"ids:representation\"][0][\"@id\"]).to.exist;\r", + " pm.collectionVariables.set(\"representation\", jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"ids:representation\"][0][\"@id\"])\r", + " });" + ], + "type": "text/javascript" + } + } + ], + "request": { + "method": "GET", + "header": [], + "url": { + "raw": "{{Self_Description_URL}}/api/selfDescription/", + "host": [ + "{{Self_Description_URL}}" + ], + "path": [ + "api", + "selfDescription", + "" + ] + } + }, + "response": [] + }, { "name": "ContractOffer", "event": [ @@ -26,7 +74,6 @@ "//http://w3id.org/engrd/connector/artifact/test\r", " pm.expect(pm.response.text()).to.include(\"ids:Permission\");\r", " pm.expect(pm.response.text()).to.include(\"ids:ContractOffer\");\r", - " pm.expect(pm.response.text()).to.include(\"ids:Constraint\");\r", " pm.expect(pm.response.text()).to.include(\"http://w3id.org/engrd/connector/artifact\");\r", " });" ], @@ -233,6 +280,54 @@ { "name": "OfferedResource", "item": [ + { + "name": "SelfDescription", + "event": [ + { + "listen": "test", + "script": { + "exec": [ + "var jsonData = pm.response.json();\r", + "pm.response.to.be.ok;\r", + "//store applicants self-description for further tests\r", + "pm.collectionVariables.set(\"APPLICANT_SELF_DESCRIPTION\", jsonData)\r", + "\r", + "pm.test(\"ContractOffer\", function () {\r", + " // catalog\r", + " pm.expect(jsonData[\"ids:resourceCatalog\"][0][\"@id\"]).to.exist;\r", + " pm.collectionVariables.set(\"resourceCatalog\", jsonData[\"ids:resourceCatalog\"][0][\"@id\"]) \r", + " // resource\r", + " pm.expect(jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"@id\"]).to.exist;\r", + " pm.collectionVariables.set(\"offeredResource\", jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"@id\"])\r", + " // contractOffer\r", + " pm.expect(jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"ids:contractOffer\"][0][\"@id\"]).to.exist;\r", + " pm.collectionVariables.set(\"contractOffer\", jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"ids:contractOffer\"][0][\"@id\"])\r", + " // representation\r", + " pm.expect(jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"ids:representation\"][0][\"@id\"]).to.exist;\r", + " pm.collectionVariables.set(\"representation\", jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"ids:representation\"][0][\"@id\"])\r", + " });" + ], + "type": "text/javascript" + } + } + ], + "request": { + "method": "GET", + "header": [], + "url": { + "raw": "{{Self_Description_URL}}/api/selfDescription/", + "host": [ + "{{Self_Description_URL}}" + ], + "path": [ + "api", + "selfDescription", + "" + ] + } + }, + "response": [] + }, { "name": "OfferedResource", "event": [ @@ -293,7 +388,7 @@ "//http://w3id.org/engrd/connector/artifact/postman\r", " pm.expect(pm.response.text()).to.include(\"http://w3id.org/engrd/connector/artifact/postman\");\r", " var list = jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"].length;\r", - " tests[\"Validate total of 3 offered resources\"] = list === 3;\r", + " tests[\"Validate total of 3 offered resources\"] = list > 2;\r", " });\r", "" ], @@ -435,6 +530,54 @@ { "name": "Representation", "item": [ + { + "name": "SelfDescription", + "event": [ + { + "listen": "test", + "script": { + "exec": [ + "var jsonData = pm.response.json();\r", + "pm.response.to.be.ok;\r", + "//store applicants self-description for further tests\r", + "pm.collectionVariables.set(\"APPLICANT_SELF_DESCRIPTION\", jsonData)\r", + "\r", + "pm.test(\"ContractOffer\", function () {\r", + " // catalog\r", + " pm.expect(jsonData[\"ids:resourceCatalog\"][0][\"@id\"]).to.exist;\r", + " pm.collectionVariables.set(\"resourceCatalog\", jsonData[\"ids:resourceCatalog\"][0][\"@id\"]) \r", + " // resource\r", + " pm.expect(jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"@id\"]).to.exist;\r", + " pm.collectionVariables.set(\"offeredResource\", jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"@id\"])\r", + " // contractOffer\r", + " pm.expect(jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"ids:contractOffer\"][0][\"@id\"]).to.exist;\r", + " pm.collectionVariables.set(\"contractOffer\", jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"ids:contractOffer\"][0][\"@id\"])\r", + " // representation\r", + " pm.expect(jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"ids:representation\"][0][\"@id\"]).to.exist;\r", + " pm.collectionVariables.set(\"representation\", jsonData[\"ids:resourceCatalog\"][0][\"ids:offeredResource\"][0][\"ids:representation\"][0][\"@id\"])\r", + " });" + ], + "type": "text/javascript" + } + } + ], + "request": { + "method": "GET", + "header": [], + "url": { + "raw": "{{Self_Description_URL}}/api/selfDescription/", + "host": [ + "{{Self_Description_URL}}" + ], + "path": [ + "api", + "selfDescription", + "" + ] + } + }, + "response": [] + }, { "name": "Representation", "event": [ From 968b4b964553083f71a10c999d298e1789023a1a Mon Sep 17 00:00:00 2001 From: David Jovanovic Date: Tue, 22 Aug 2023 16:40:13 +0200 Subject: [PATCH 32/33] updated versions in docs --- doc/TEST_API.md | 2 +- doc/TRUEConnector/component-overview.md | 2 +- doc/TRUEConnector/start-stop.md | 4 ++-- doc/exchange-data.md | 2 +- doc/rest-api.md | 2 +- doc/user_management.md | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/TEST_API.md b/doc/TEST_API.md index 73b7108..89f7b2c 100644 --- a/doc/TEST_API.md +++ b/doc/TEST_API.md @@ -75,7 +75,7 @@ curl --location -k 'https://localhost:8090/about/version' and expected response: ``` -1.14.2-SNAPSHOT +1.14.2 ``` ## Self Description API diff --git a/doc/TRUEConnector/component-overview.md b/doc/TRUEConnector/component-overview.md index fe3162c..d11c376 100644 --- a/doc/TRUEConnector/component-overview.md +++ b/doc/TRUEConnector/component-overview.md @@ -7,7 +7,7 @@ TRUE Connector is build using Java11, and use following libraries: | [Execution core container](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/releases/tag/1.14.2) | 1.14.2 | | [Basic data app](https://github.com/Engineering-Research-and-Development/true-connector-basic_data_app/releases/tag/0.3.1) | 0.3.1 | | [Usage control app](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/releases/tag/1.7.4) | 1.7.4 | -| [Pip](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/tree/1.7.3/Docker_Tecnalia_DataUsage/pip) | 1.0.0 | +| [Pip](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/tree/1.7.4/Docker_Tecnalia_DataUsage/pip) | 1.0.0 | | [Multipart Message Library](https://github.com/Engineering-Research-and-Development/true-connector-multipart_message_library/releases/tag/1.0.17) | 1.0.17 | | [Websocket Message Streamer](https://github.com/Engineering-Research-and-Development/true-connector-websocket_message_streamer/releases/tag/1.0.17) | 1.0.17 | | [Information model](https://github.com/International-Data-Spaces-Association/InformationModel) | 4.2.7 | diff --git a/doc/TRUEConnector/start-stop.md b/doc/TRUEConnector/start-stop.md index c77bbb9..4cf03c7 100644 --- a/doc/TRUEConnector/start-stop.md +++ b/doc/TRUEConnector/start-stop.md @@ -93,9 +93,9 @@ bc693e1fdb90 rdlabengpa/ids_execution_core_container:1.14.2 "/bin/sh -c 'jav 28dc87213f68 rdlabengpa/ids_be_data_app:0.3.1 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 0.0.0.0:8184->8183/tcp, :::8184->8183/tcp, 0.0.0.0:9001->9000/tcp, :::9001->9000/tcp be-dataapp-consumer 9eb157ceb37b rdlabengpa/ids_be_data_app:0.3.1 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 0.0.0.0:8183->8183/tcp, :::8183->8183/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp be-dataapp-provider 44bc21187460 rdlabengpa/ids_execution_core_container:1.14.2 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 0.0.0.0:8086->8086/tcp, :::8086->8086/tcp, 0.0.0.0:8889->8889/tcp, :::8889->8889/tcp, 0.0.0.0:8090->8449/tcp, :::8090->8449/tcp ecc-provider -b3f4cdb77ed6 rdlabengpa/ids_uc_data_app_platoon:1.7.3 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 8080/tcp uc-dataapp-consumer +b3f4cdb77ed6 rdlabengpa/ids_uc_data_app_platoon:1.7.4 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 8080/tcp uc-dataapp-consumer a36748901ce1 rdlabengpa/ids_uc_data_app_platoon_pip:v1.0.0 "java -jar pip.jar" 3 hours ago Up 3 hours 0/tcp uc-dataapp-pip-provider -d6f77ad9762d rdlabengpa/ids_uc_data_app_platoon:1.7.3 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 8080/tcp uc-dataapp-provider +d6f77ad9762d rdlabengpa/ids_uc_data_app_platoon:1.7.4 "/bin/sh -c 'java -j…" 3 hours ago Up 3 hours (healthy) 8080/tcp uc-dataapp-provider bb0bb9668931 rdlabengpa/ids_uc_data_app_platoon_pip:v1.0.0 "java -jar pip.jar" 3 hours ago Up 3 hours 0/tcp uc-dataapp-pip-consumer ``` diff --git a/doc/exchange-data.md b/doc/exchange-data.md index 6c43b6d..1f89d9a 100644 --- a/doc/exchange-data.md +++ b/doc/exchange-data.md @@ -28,7 +28,7 @@ _NOTE_: even that this curl command is exported from Postman, it is noticed seve If this happens, please check body of the request in Postman, and if body is empty, simply copy everything enclosed between\ _--data-raw '_ and _'_ -For more details on request samples, please check following link [Backend DataApp Usage](https://github.com/Engineering-Research-and-Development/market4.0-data\_app\_test\_BE/blob/0.3.0/README.md) +For more details on request samples, please check following link [Backend DataApp Usage](https://github.com/Engineering-Research-and-Development/market4.0-data\_app\_test\_BE/blob/0.3.1/README.md) Be sure to use correct configuration/ports for sender and receiver Data App and Execution Core Container (check .env file). diff --git a/doc/rest-api.md b/doc/rest-api.md index 703dc91..3d7e541 100644 --- a/doc/rest-api.md +++ b/doc/rest-api.md @@ -2,4 +2,4 @@ Detailed description of API endpoints provided by TRUE Connector can be found in [link](rest\_api/REST\_API.md) -Bare in mind that all endpoints of the TRUE Connector will require authorization. Please follow [this link](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.1/doc/SECURITY.md) to get more information about providing correct credentials for desired request/functionality. \ No newline at end of file +Bare in mind that all endpoints of the TRUE Connector will require authorization. Please follow [this link](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.2/doc/SECURITY.md) to get more information about providing correct credentials for desired request/functionality. \ No newline at end of file diff --git a/doc/user_management.md b/doc/user_management.md index 2bb1e86..14baaf1 100644 --- a/doc/user_management.md +++ b/doc/user_management.md @@ -21,4 +21,4 @@ application.user.api.password=$2a$10$MQ5grDaIqDpBjMlG78PFduv.AMRe9cs0CNm/V4cgUub ## Modifying password for a user -Once new password is generated, (described [here](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.1/doc/SECURITY.md#change-default-password)) user should send encoded password to the operations user, which should be the only one who can modify connector property file. That user will update property file and restart TRUE Connector, so that new password will be loaded by the connector. \ No newline at end of file +Once new password is generated, (described [here](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.2/doc/SECURITY.md#change-default-password)) user should send encoded password to the operations user, which should be the only one who can modify connector property file. That user will update property file and restart TRUE Connector, so that new password will be loaded by the connector. \ No newline at end of file From e5eaf5eff502390c770158c4bf18bbf6cbfab910 Mon Sep 17 00:00:00 2001 From: David Jovanovic Date: Tue, 22 Aug 2023 16:48:47 +0200 Subject: [PATCH 33/33] update roadmap --- doc/roadmap.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/roadmap.md b/doc/roadmap.md index 7120712..451e393 100644 --- a/doc/roadmap.md +++ b/doc/roadmap.md @@ -4,8 +4,8 @@ This list follows no timeline, instead, individual tasks can be priority-assigne | Priority | Task | Status | Note | |:---:|:------------|:------------|:------------| -| 1 | Dataspace Protocol investigation | Ongoing | | -| 1 | Dataspace Protocol (compliance with EDC)| Ongoing | | +| 1 | Dataspace Protocol investigation | | | +| 1 | Dataspace Protocol (compliance with EDC)| | | | 2 | User Interface | Ongoing | | | 1 | Audit logging | Done | | | 3 | Kubernetes deployment | Done | |