Certification fixes (#30)

* Add application.firewall.isEnabled property to all relevant properties files * Update .env file * Update docker-compose to use new property * Add firewall.properties to ecc and dataapp resources * Update documentation * Add information about user credentials persistence and password encoding * Added info about self decription properties * Added connectorId ecc change Fixed port for pip services * Resource management update Added JDK_JAVA_OPTIONS param * Update documentation related to the new version of UC * New UC app security related features * Fix datalake path in Data App properties file * New cosign hash values --------- Co-authored-by: Marko Stojanovic <[email protected]>
International-Data-Spaces-Association · Oct 4, 2023 · 5d0b6d6 · 5d0b6d6
1 parent 89e5489
commit 5d0b6d6
Show file tree

Hide file tree

Showing 32 changed files with 346 additions and 103 deletions.
diff --git a/.env b/.env
@@ -1,8 +1,8 @@
-COMPOSE_PROJECT_NAME=TRUE_Connector
+COMPOSE_PROJECT_NAME=trueconnector
 
 BROKER_URL=https://broker.ids.isst.fraunhofer.de/infrastructure
 
-#SSL settings
+#TLS settings
 KEYSTORE_NAME=ssl-server.jks
 KEY_PASSWORD=changeit
 KEYSTORE_PASSWORD=changeit
@@ -27,6 +27,7 @@ IDSCP2=false
 EXTRACT_PAYLOAD_FROM_RESPONSE=true
 
 ### PROVIDER Configuration
+PROVIDER_ECC_SELF_DESCRIPTION_URL=https://ecc-provider
 PROVIDER_DAPS_KEYSTORE_NAME=
 PROVIDER_DAPS_KEYSTORE_PASSWORD=
 PROVIDER_DAPS_KEYSTORE_ALIAS=
@@ -38,8 +39,12 @@ PROVIDER_DATA_APP_ENDPOINT=https://be-dataapp-provider:8083/data
 #PROVIDER_DATA_APP_ENDPOINT=https://be-dataapp-provider:9000/incoming-data-app/routerBodyBinary
 PROVIDER_WS_EDGE=false
 PROVIDER_ISSUER_CONNECTOR_URI=http://w3id.org/engrd/connector/provider
+PROVIDER_DATA_APP_FIREWALL=false
+PROVIDER_ECC_FIREWALL=false
+
 
 ### CONSUMER Configuration
+CONSUMER_ECC_SELF_DESCRIPTION_URL=https://ecc-consumer
 CONSUMER_DAPS_KEYSTORE_NAME=
 CONSUMER_DAPS_KEYSTORE_PASSWORD=
 CONSUMER_DAPS_KEYSTORE_ALIAS=
@@ -49,3 +54,5 @@ CONSUMER_MULTIPART_EDGE=form
 CONSUMER_DATA_APP_ENDPOINT=https://be-dataapp-consumer:8083/data
 CONSUMER_WS_EDGE=false
 CONSUMER_ISSUER_CONNECTOR_URI=http://w3id.org/engrd/connector/consumer
+CONSUMER_DATA_APP_FIREWALL=false
+CONSUMER_ECC_FIREWALL=false
diff --git a/README.md b/README.md
@@ -46,6 +46,7 @@ Please [click here](https://engineering-ing-inf-rd.gitbook.io/true-connector/),
     * [Usage Control](doc/advancedConfiguration/usagecontrol.md)
     * [MyData Usage Control](doc/advancedConfiguration/mydata.md)
     * [Audit logs](doc/advancedConfiguration/auditlogs.md)
+    * [Firewall](doc/advancedConfiguration/firewall.md)
   * [Contract Negotiation - simple flow](doc/contractNegotiation/contract-negotiation.md)
     * [Get offered resource](doc/contractNegotiation/get_offered_resource.md)
     * [Description Request Message](doc/contractNegotiation/description_request_message.md)

diff --git a/SUMMARY.md b/SUMMARY.md
@@ -1,6 +1,7 @@
 # Table of contents
 
 * [TRUE Connector](README.md)
+  * [Prerequisite](doc/TRUEConnector/prerequisite.md)
   * [Introduction](doc/TRUEConnector/introduction.md)
   * [System requirements](doc/TRUEConnector/system-requirements.md)
   * [Volumes](doc/TRUEConnector/volumes.md)
@@ -27,6 +28,7 @@
   * [Usage Control](doc/advancedConfiguration/usagecontrol.md)
   * [MyData Usage Control](doc/advancedConfiguration/mydata.md)
   * [Audit logs](doc/advancedConfiguration/auditlogs.md)
+  * [Firewall](doc/advancedConfiguration/firewall.md)
 * [Contract Negotiation - simple flow](doc/contractNegotiation/contract-negotiation.md)
   * [Get offered resource](doc/contractNegotiation/get_offered_resource.md)
   * [Description Request Message](doc/contractNegotiation/description_request_message.md)
@@ -41,10 +43,8 @@
   * [MYDATA\_USAGE\_CONTROL](doc/MYDATA\_USAGE\_CONTROL.md)
   * [PLATOON\_USAGE\_CONTROL](doc/PLATOON\_USAGE\_CONTROL.md)
   * [Test cases](doc/TEST\_API.md)
-  * [rest\_api](doc/rest\_api/README.md)
-    * [REST API](doc/rest\_api/REST\_API.md)
-  * [testbed](doc/testbed/README.md)
-    * [Testbed](doc/testbed/TESTBED.md)
+  * [REST API](doc/rest\_api/REST\_API.md)
+  * [Testbed](doc/testbed/TESTBED.md)
   * [Step to replicate True Connector installation in minikube.](kubernetes/README.md)
   * [Docker image signing and verification](doc/cosign.md)
 * [Life cycle](doc/life_cycle.md)

diff --git a/be-dataapp_resources/application-docker.properties b/be-dataapp_resources/application-docker.properties
@@ -19,6 +19,9 @@ server.ssl.key-store-provider=SUN
 server.ssl.key-store-type=JKS
 server.ssl.trust-store-type=JKS
 
+#Firewall
+application.firewall.isEnabled=${FIREWALL}
+
 application.dataapp.http.config=${DATA_APP_MULTIPART}
 
 application.ecc.protocol=https
@@ -43,7 +46,7 @@ application.encodePayload=false
 application.extractPayloadFromResponse=${EXTRACT_PAYLOAD_FROM_RESPONSE}
 
 application.fileSenderPort=9000
-application.dataLakeDirectory=/home/nobody/data
+application.dataLakeDirectory=/home/nobody/data/datalake
 
 #checkSum verification - true | false
 application.verifyCheckSum=false

diff --git a/be-dataapp_resources/firewall.properties b/be-dataapp_resources/firewall.properties
@@ -0,0 +1,18 @@
+#Set which HTTP header names should be allowed (if want to allow all header names, keep it empty)
+allowedHeaderNames=
+#Set which values in header names should have the exact value and allowed (if want to allow any values keep it empty)
+allowedHeaderValues=
+#Set which HTTP methods should be allowed (if want to allow all header names, keep it empty)
+allowedMethods=GET,POST
+#Set if a backslash "\" or a URL encoded backslash "%5C" should be allowed in the path or not
+allowBackSlash=true
+#Set if a slash "/" that is URL encoded "%2F" should be allowed in the path or not
+allowUrlEncodedSlash=true
+#Set if double slash "//" that is URL encoded "%2F%2F" should be allowed in the path or not
+allowUrlEncodedDoubleSlash=true
+#Set if semicolon is allowed in the URL (i.e. matrix variables)
+allowSemicolon=true
+#Set if a percent "%" that is URL encoded "%25" should be allowed in the path or not
+allowUrlEncodedPercent=true
+#if a period "." that is URL encoded "%2E" should be allowed in the path or not
+allowUrlEncodedPeriod=true
diff --git a/doc/PLATOON_USAGE_CONTROL.md b/doc/PLATOON_USAGE_CONTROL.md
@@ -105,5 +105,5 @@ POSTGRES_DB=usagecontrol_consumer
 
 # Usage control examples
 
-For more information and examples of policies compatible with Platoon UC app, please check [README](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/blob/1.7.4/README.md)
+For more information and examples of policies compatible with Platoon UC app, please check [README](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/blob/1.7.5/README.md)
 
diff --git a/doc/TEST_API.md b/doc/TEST_API.md
@@ -75,7 +75,7 @@ curl --location -k 'https://localhost:8090/about/version'
 and expected response:
 
 ```
-1.14.2
+1.14.3
 ```
 
 ## Self Description API

diff --git a/doc/TRUEConnector/component-overview.md b/doc/TRUEConnector/component-overview.md
@@ -4,10 +4,10 @@ TRUE Connector is build using Java11, and use following libraries:
 
 | Component                                                                                                                         						| Version       |
 | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
-| [Execution core container](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/releases/tag/1.14.2)   		| 1.14.2 		|
-| [Basic data app](https://github.com/Engineering-Research-and-Development/true-connector-basic_data_app/releases/tag/0.3.1) 								| 0.3.1 		|
-| [Usage control app](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/releases/tag/1.7.4)   						| 1.7.4 		|
-| [Pip](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/tree/1.7.4/Docker_Tecnalia_DataUsage/pip) 				| 1.0.0 		|
+| [Execution core container](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/releases/tag/1.14.3)   		| 1.14.3 		|
+| [Basic data app](https://github.com/Engineering-Research-and-Development/true-connector-basic_data_app/releases/tag/0.3.2) 								| 0.3.2 		|
+| [Usage control app](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/releases/tag/1.7.5)   						| 1.7.5 		|
+| [Pip](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/tree/1.7.5/Docker_Tecnalia_DataUsage/pip) 				| 1.0.0 		|
 | [Multipart Message Library](https://github.com/Engineering-Research-and-Development/true-connector-multipart_message_library/releases/tag/1.0.17)   		| 1.0.17 		|
 | [Websocket Message Streamer](https://github.com/Engineering-Research-and-Development/true-connector-websocket_message_streamer/releases/tag/1.0.17) 		| 1.0.17 		|
 | [Information model](https://github.com/International-Data-Spaces-Association/InformationModel)                                    						| 4.2.7         |

diff --git a/doc/TRUEConnector/default-configuration.md b/doc/TRUEConnector/default-configuration.md
@@ -8,7 +8,8 @@ TRUE Connector comes pre-configured with following:
 * Disabled Usage control
 * Disabled Clearing House
 * Disabled validate protocol in Forward-To header
-* Disabled hostname validation
 * Disabled CheckSum validation
+* Disabled Firewall
+
 
 If you wish to change this configuration, please check chapter [Modifying configuration](../modifyingConfiguration/modify-configuration.md)
diff --git a/doc/TRUEConnector/prerequisite.md b/doc/TRUEConnector/prerequisite.md
@@ -1,12 +1,12 @@
-# Prerequisite
+## Prerequisite <a href="doc/#prerequisite" id="prerequisite"></a>
 
 To have secure and certification compliant environment, following prerequisites are mandatory to be performed before setting up TRUE Connector:
 
 * NTP time server of the machine, where TRUE Connector will be deployed, has to be enabled and configured correctly. This will allow that once certificates are checked, correct time will be used to verify certificate, expired or not. This applies for both DAPS and TLS1.3 certificates. Connector will rely on OS time when checking certificates
 * Docker is mandatory "OS service" for running connector
 * verify [System requirements](system-requirements.md) before starting the connector.
 
-## Securing docker host
+### Securing docker host
 
 * The host OS should be audited and secure; OS should be as minimal as possible and it should be preferably used to host our Docker exclusively. There should not coexist other services like web servers or web applications so that attacker could not exploit it or lead to potential exploit (minimal threat attack surface).
 * Monitoring mechanism (Linux auditd service for example) should be installed and configured as prerequisite before deploying connector. This will capture if someone tries to make changes on property files used by the connector.
@@ -20,15 +20,16 @@ To have secure and certification compliant environment, following prerequisites
 * OS user for running docker should not be root user; be sure to create new user, assign new user to docker group, that user can run docker compose
 * disable password login to the server for newly created user and allow only key-based authentication for accessing the server where connector will run
 * disable access for the root user by using a password when connecting to the server via ssh (key-based auth only)
+* in case of adding some additional, more configurable and robust firewall, be sure to restrict access to the /api/* endpoints to only internal network, since those endpoints should not be exposed to the outside world, but intended to be used by "internal" user, to make modifications to the self description document.
 
 
 * 2 types of certificate are required: DAPS and TLS. 
 DAPS certificate should be obtained from Certified Authority responsible for the Dataspace, while TLS certificate can be self signed or signed by some CA. More information about TLS certificate can be found [here](../security.md).
 
 
-# Post configuration steps
+## Post configuration steps
 
-Once TRUE Connector is successfully configured and is up and running, responsible user for setting up environment and configuring connector should generate new passwords for 2 type of users required for operating with connector. More information how to do this can be found [here](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.2/doc/SECURITY.md#change-default-password). 
+Once TRUE Connector is successfully configured and is up and running, responsible user for setting up environment and configuring connector should generate new passwords for 2 type of users required for operating with connector. More information how to do this can be found [here](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.3/doc/SECURITY.md#change-default-password). 
 
 Make sure to update following properties to address your usecase:
 

diff --git a/doc/TRUEConnector/start-stop.md b/doc/TRUEConnector/start-stop.md
@@ -89,13 +89,13 @@ You can also check using _docker ps_ command to verify that containers are up an
 
 ```
 CONTAINER ID   IMAGE                                             COMMAND                  CREATED       STATUS                 PORTS                                                                                                                             NAMES
-bc693e1fdb90   rdlabengpa/ids_execution_core_container:1.14.2   "/bin/sh -c 'java -j…"   3 hours ago   Up 3 hours (healthy)   0.0.0.0:8087->8086/tcp, :::8087->8086/tcp, 0.0.0.0:8091->8449/tcp, :::8091->8449/tcp, 0.0.0.0:8890->8889/tcp, :::8890->8889/tcp   ecc-consumer
-28dc87213f68   rdlabengpa/ids_be_data_app:0.3.1                "/bin/sh -c 'java -j…"   3 hours ago   Up 3 hours (healthy)   0.0.0.0:8184->8183/tcp, :::8184->8183/tcp, 0.0.0.0:9001->9000/tcp, :::9001->9000/tcp                                              be-dataapp-consumer
-9eb157ceb37b   rdlabengpa/ids_be_data_app:0.3.1                "/bin/sh -c 'java -j…"   3 hours ago   Up 3 hours (healthy)   0.0.0.0:8183->8183/tcp, :::8183->8183/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp                                              be-dataapp-provider
-44bc21187460   rdlabengpa/ids_execution_core_container:1.14.2   "/bin/sh -c 'java -j…"   3 hours ago   Up 3 hours (healthy)   0.0.0.0:8086->8086/tcp, :::8086->8086/tcp, 0.0.0.0:8889->8889/tcp, :::8889->8889/tcp, 0.0.0.0:8090->8449/tcp, :::8090->8449/tcp   ecc-provider
-b3f4cdb77ed6   rdlabengpa/ids_uc_data_app_platoon:1.7.4       "/bin/sh -c 'java -j…"   3 hours ago   Up 3 hours (healthy)   8080/tcp                                                                                                                          uc-dataapp-consumer
+bc693e1fdb90   rdlabengpa/ids_execution_core_container:1.14.3   "/bin/sh -c 'java -j…"   3 hours ago   Up 3 hours (healthy)   0.0.0.0:8087->8086/tcp, :::8087->8086/tcp, 0.0.0.0:8091->8449/tcp, :::8091->8449/tcp, 0.0.0.0:8890->8889/tcp, :::8890->8889/tcp   ecc-consumer
+28dc87213f68   rdlabengpa/ids_be_data_app:0.3.2                "/bin/sh -c 'java -j…"   3 hours ago   Up 3 hours (healthy)   0.0.0.0:8184->8183/tcp, :::8184->8183/tcp, 0.0.0.0:9001->9000/tcp, :::9001->9000/tcp                                              be-dataapp-consumer
+9eb157ceb37b   rdlabengpa/ids_be_data_app:0.3.2                "/bin/sh -c 'java -j…"   3 hours ago   Up 3 hours (healthy)   0.0.0.0:8183->8183/tcp, :::8183->8183/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp                                              be-dataapp-provider
+44bc21187460   rdlabengpa/ids_execution_core_container:1.14.3   "/bin/sh -c 'java -j…"   3 hours ago   Up 3 hours (healthy)   0.0.0.0:8086->8086/tcp, :::8086->8086/tcp, 0.0.0.0:8889->8889/tcp, :::8889->8889/tcp, 0.0.0.0:8090->8449/tcp, :::8090->8449/tcp   ecc-provider
+b3f4cdb77ed6   rdlabengpa/ids_uc_data_app_platoon:1.7.5       "/bin/sh -c 'java -j…"   3 hours ago   Up 3 hours (healthy)   8080/tcp                                                                                                                          uc-dataapp-consumer
 a36748901ce1   rdlabengpa/ids_uc_data_app_platoon_pip:v1.0.0     "java -jar pip.jar"      3 hours ago   Up 3 hours             0/tcp                                                                                                                             uc-dataapp-pip-provider
-d6f77ad9762d   rdlabengpa/ids_uc_data_app_platoon:1.7.4        "/bin/sh -c 'java -j…"   3 hours ago   Up 3 hours (healthy)   8080/tcp                                                                                                                          uc-dataapp-provider
+d6f77ad9762d   rdlabengpa/ids_uc_data_app_platoon:1.7.5        "/bin/sh -c 'java -j…"   3 hours ago   Up 3 hours (healthy)   8080/tcp                                                                                                                          uc-dataapp-provider
 bb0bb9668931   rdlabengpa/ids_uc_data_app_platoon_pip:v1.0.0     "java -jar pip.jar"      3 hours ago   Up 3 hours             0/tcp                                                                                                                             uc-dataapp-pip-consumer
 
 ```

diff --git a/doc/TRUEConnector/system-requirements.md b/doc/TRUEConnector/system-requirements.md
@@ -3,6 +3,63 @@
 In order to run TRUE Connector following minimal system requirements are needed:
 
 * CPU: newer 4 core (8 threads)
-* Memory: at least 2GB dedicated to TRUE Connector (1024MB - for ECC services, 512MB for DataApp and 512MB for Usage Control services)
+* Memory: at least 2GB dedicated to TRUE Connector per instance (1024MB - for ECC services, 512MB for DataApp, 256MB for Usage Control services and 256MB for Usage Control PIP services)
 
-This values can be considered as initial values, and if required, they can be increased or reduced, keeping the functionality of TRUE Connector unchanged.
+This values can be considered as initial values, and if required, they can be increased or reduced, keeping the functionality of TRUE Connector unchanged.
+
+Default resources, provided to docker containers are following (defined in docker-compose.yml):
+
+```
+ ecc-*:
+    deploy:
+      resources:
+        limits:
+          cpus: "1"
+          memory: 1024M
+    logging:
+      options:
+        max-size: "200M"
+...
+
+ uc-dataapp-*:
+    deploy:
+      resources:
+        limits:
+          cpus: "1"
+          memory: 256M
+    logging:
+      options:
+        max-size: "100M"
+...
+
+ uc-dataapp-pip-*:
+    deploy:
+      resources:
+        limits:
+          cpus: "1"
+          memory: 256M
+    logging:
+      options:
+        max-size: "100M"
+...
+
+  be-dataapp-*:
+    deploy:
+      resources:
+        limits:
+          cpus: "1"
+          memory: 512M
+    logging:
+      options:
+        max-size: "100M"
+...
+
+```
+
+In case that you need to assign more memory to some specific service, this can be done by increasing memory amount in deploy section for specific service.
+In case of *java.lang.OutOfMemoryError: Java heap space* be sure to pass following environment variable to "problematic" service:
+
+
+- "JDK_JAVA_OPTIONS=-Xmx1024m"
+
+Variables defined in deploy resource part and this passed to JVM needs to be correlated, meaning that you first need to delegate memory to docker service and then to assign memory JVM from that amount.
diff --git a/doc/advancedConfiguration/auditlogs.md b/doc/advancedConfiguration/auditlogs.md
@@ -1,6 +1,6 @@
 ### Audit logs <a href="#auditlogs" id="auditlogs"></a>
 
-Audit logging is turned **off** by default. If you wish to configure it or even turn off please follow this [document](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.2/doc/AUDIT.md) .
+Audit logging is turned **off** by default. If you wish to configure it or even turn off please follow this [document](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.3/doc/AUDIT.md) .
 
 
 ## Accessing audit logs

diff --git a/doc/advancedConfiguration/broker.md b/doc/advancedConfiguration/broker.md
@@ -13,4 +13,4 @@ TRUE Connector can register itself on startup, and also unregister when shutting
 application.selfdescription.registrateOnStartup=true
 ```
 
-Information on how TRUE Connector can interact with Broker, can be found on following [link](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.2/doc/BROKER.md)
+Information on how TRUE Connector can interact with Broker, can be found on following [link](https://github.com/Engineering-Research-and-Development/true-connector-execution\_core\_container/blob/1.14.3/doc/BROKER.md)
diff --git a/doc/advancedConfiguration/extendedjwt.md b/doc/advancedConfiguration/extendedjwt.md
@@ -1,3 +1,3 @@
 ### Extended jwt validation <a href="#extendedjwt" id="extendedjwt"></a>
 
-TRUE Connector can check additional claims from jwToken. For more information please check the [following link](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.2/doc/TRANSPORTCERTSSHA256.md)
+TRUE Connector can check additional claims from jwToken. For more information please check the [following link](https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container/blob/1.14.3/doc/TRANSPORTCERTSSHA256.md)
Original file line number	Diff line number	Diff line change
Expand Up		@@ -105,5 +105,5 @@ POSTGRES_DB=usagecontrol_consumer

		# Usage control examples

		For more information and examples of policies compatible with Platoon UC app, please check [README](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/blob/1.7.4/README.md)
		For more information and examples of policies compatible with Platoon UC app, please check [README](https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app_platoon/blob/1.7.5/README.md)
-Original file line number
+Diff line change
@@ Expand Up / @@ -75,7 +75,7 @@ curl --location -k 'https://localhost:8090/about/version' @@
     and expected response:
     ```
-.14.2
+.14.3
     ```
     ## Self Description API
@@ Expand Down @@