SymbIoteCloud from docker

Cloud Deployment

This document explains how to run the symbIoTe cloud for L1/L2 integration depending on your needs. There are two ways to run docker:

1. directly (on linux)
2. docker-machine (previously boot2docker) - in virtual machine

This influences how to expose port in virtual machine to local machine. So, there are 2 ways to run ngrok and 2 ways to expose port in production environment.

1. Preparation steps

1.1. Register user and configure your platform.

When getting the platform configuration you have to enter the following options:

• Compliance Level: L1 or L2 (depending on the compliance level desired)
• Deployment Type: Docker

1.2. Install prerequisites:

• docker (18.03.x),
• docker-compose (1.21.x),
• docker-machine (0.14.x),
• bash,
• curl,
• wget

1.3. Create folder for your deployment

• mkdir symbiote-cloud
• change to that directory (e.g. cd symbiote-cloud)
• unzip the configuration files to the directory from the .zip downloaded earlier
• copy the docker-compose files for the desired compliance level:
  • docker-compose-swarm-L1.yml and docker-compose-prod-swarm-L1.yml for L1 integration. Also, docker-compose-ngrok-swarm-L1.yml to run ngrok.
  • docker-compose-swarm-L2.yml and docker-compose-prod-swarm-L2.yml for L2 integration. Also, docker-compose-ngrok-swarm-L2.yml to run ngrok.
• Uncomment and configure the proxy settings in the docker compose files for L1/L2 (docker-compose-swarm-L1.yml, docker-compose-swarm-L2.yml) if you are behind a proxy.
• example of implementation of ResourceAccessProxyPlugin:
  • L1: docker-compose-swarm-custom.yml
  • L2: docker-compose-swarm-custom.yml
• In some YML files are parts that are configuring PROXY. If you use proxy it should be uncommented and changed according to your proxy server and if you do not use it please comment those parts by putting # in the beginning of line. Parameters related to proxy are: JAVA_HTTP_PROXY, JAVA_HTTPS_PROXY, JAVA_SOCKS_PROXY and JAVA_NON_PROXY_HOSTS.

2. Configuring and starting components depending on your preferred deployment

Here is table with instructions based on preferred deployment:

Environment	docker directly (on linux)	docker-machine
production environment	section 2.1	section 2.2
hackathon environment (ngrok)	section 2.3	section 2.4

Important Note:

The compose files have been configured to block incoming external traffic to components other than the nginx when docker compose is used. However, this is not possible when using docker swarm. Consequently, the Platform Owners need to block the incoming external traffic to components by using other means e.g. iptables in all the swarm nodes.

2.1. Production, without docker-machine (boot2docker)

1. Obtain certificate as described in https://github.com/symbiote-h2020/SymbioteCloud/wiki/2.1-Configuration-of-NGINX#2111-obtaining-the-ssl-certificate

   • Create dir nginx-certificates
     $ mkdir nginx-certificates

   • Copy certificates to nginx-certificates
     $ sudo cp /etc/letsencrypt/live/{your domain}/fullchain.pem nginx-certificates/
$ sudo cp /etc/letsencrypt/live/{your domain}/privkey.pem nginx-certificates/

2. Run docker swarm init if the node is not a swarm manager. We use the swarm mode so that secrets are encrypted during transit and at rest. Docker secrets are only available to swarm services and not to standalone containers.

3. Run:

   • docker stack deploy -c docker-compose-swarm-L1.yml -c docker-compose-prod-swarm-L1.yml -c docker-compose-swarm-custom.yml symbiote-app to deploy the application in L1 compliance or
   • docker stack deploy -c docker-compose-swarm-L2.yml -c docker-compose-prod-swarm-L2.yml -c docker-compose-swarm-custom.yml symbiote-app for L2 compliance. You can change the name of the service stack (symbiote-app) to be used.

4. You can run:

   • docker stack ls to list the stack and check the number of services used.
   • docker image ls to check that all images have been created. It may take a while to pull all the images from DockerHub for the first time.
   • docker service ls to list the services and check their status. Wait until the actual number of tasks (replicas) for each service is not 0.
   • docker logs <container_name> -f to get access to and follow the logs of a service. A component is ready when a message similar to 'Started <component_name> in 105.045 seconds (JVM running for 112.933)' appears in the logs of the container.
   • sudo service docker restart to restart the docker service if needed and repeat the steps.

5. Run docker stack rm symbiote-app to stop the application and remove the service stack. Services, networks, and secrets associated with the stack will be removed.

6. You can run docker swarm leave --force to leave the swarm.

2.2. Production, with docker-machine (boot2docker)

1. Obtain certificate as described previously
2. The following command maps
   • port 443 in default docker-machine (localhost:443) to
   • local computer at port 443 on all interfaces (0.0.0.0:443)
     docker-machine ssh default -L 0.0.0.0:443:localhost:443
3. Run steps 2-6 as described previously in Section 2.1.

2.3. Hackathon, without docker-machine (boot2docker)

Before starting with hackathon environment you need to install ngrok (https://github.com/symbiote-h2020/SymbioteCloud/wiki/2.1-Configuration-of-NGINX#2121-ngrok).

1. Run docker swarm init if the node is not a swarm manager. We use the swarm mode so that secrets are encrypted during transit and at rest. Docker secrets are only available to swarm services and not to standalone containers.
2. Run
   • docker stack deploy -c docker-compose-swarm-L1.yml -c docker-compose-ngrok-swarm-L1.yml -c docker-compose-swarm-custom.yml symbiote-app to deploy the application in L1 compliance or
   • docker stack deploy -c docker-compose-swarm-L2.yml -c docker-compose-ngrok-swarm-L2.yml -c docker-compose-swarm-custom.yml symbiote-app for L2 compliance.
3. You can run:
   • docker stack ls to list the stack and check the number of services used.
   • docker image ls to check that all images have been created. It may take a while to pull all the images from DockerHub for the first time.
   • docker service ls to list the services and check their status. Wait until the actual number of tasks (replicas) for each service is not 0.
   • docker logs <container_name> -f to get access to and follow the logs of a service. A component is ready when a message similar to 'Started <component_name> in 105.045 seconds (JVM running for 112.933)' appears in the logs of the container.
   • sudo service docker restart to restart the docker service if needed and repeat the steps.
4. Run docker stack rm symbiote-app to stop the application and remove the service stack. Services, networks, and secrets associated with the stack will be removed.

2.4. Hackathon, with docker-machine (boot2docker)

Before starting with hackathon environment you need to install ngrok (https://github.com/symbiote-h2020/SymbioteCloud/wiki/2.1-Configuration-of-NGINX#2121-ngrok).

1. Run docker swarm init if the node is not a swarm manager. We use the swarm mode so that secrets are encrypted during transit and at rest. Docker secrets are only available to swarm services and not to standalone containers.
2. Run docker-machine ssh default -L 0.0.0.0:8102:localhost:8102 for mapping port in virtual machine to this computer.
3. Run
   • docker stack deploy -c docker-compose-swarm-L1.yml -c docker-compose-ngrok-swarm-L1.yml -c docker-compose-swarm-custom.yml symbiote-app to deploy the application in L1 compliance or
   • docker stack deploy -c docker-compose-swarm-L2.yml -c docker-compose-ngrok-swarm-L2.yml -c docker-compose-swarm-custom.yml symbiote-app for L2 compliance.
4. You can run:
   • docker stack ls to list the stack and check the number of services used.
   • docker image ls to check that all images have been created. It may take a while to pull all the images from DockerHub for the first time.
   • docker service ls to list the services and check their status. Wait until the actual number of tasks (replicas) for each service is not 0.
   • docker logs <container_name> -f to get access to and follow the logs of a service. A component is ready when a message similar to 'Started <component_name> in 105.045 seconds (JVM running for 112.933)' appears in the logs of the container.
   • sudo service docker restart to restart the docker service if needed and repeat the steps.
5. Run docker stack rm symbiote-app to stop the application and remove the service stack. Services, networks, and secrets associated with the stack will be removed.

2.5. Manage resources

3. Test integrated resource

3.1. Security

3.2. Search for resources

3.3. Obtaining resource access URL

3.4. Accessing the resource and actuating and invoking service for default (dummy) resources

4. Creating RAP plugin

4.1. Customizing internal RAP plugin

4.2. Using RAP plugin starter

4.3. Creating RAP plugin in other languages

4.4. Development environment

During development and testing of RAP plugin it is convenient to run it in IDE on local machine. In that case follow steps in this document.

4.5. Deploying custom plugin

When you are finished with your custom RAP plugin and you have tested it (explained in section 4.4). Then you need to deploy it in docker environment. You need to do following:

1. Create docker image from builded jar
2. Upload image to machine where is running docker implementation

• you can export image to file, transfer file to machine with docker implementation and import image or
• you can upload image to docker hub and use image from there

3. change docker-compose-swarm-custom.yml to use your image
4. restart docker swarm stack

5. Resource Description Examples

5.1. JSON Description Examples

5.2. RDF Description Examples