SymbIoteCloud from docker

• Cloud Deployment
  • 1. Preparation steps
    • 1.1. Install prerequisites
    • 1.2. Register user and configure your platform
    • 1.3. Create folder for your deployment
  • 2. Configuring and starting components depending on your preferred deployment
    • 2.1. Production without docker-machine (boot2docker)
    • 2.2. Production with docker-machine (boot2docker)
    • 2.3. Hackathon without docker-machine (boot2docker)
    • 2.4. Hackathon with docker-machine (boot2docker)
    • 2.5. Manage resources
  • 3. Test integrated resource
    • 3.1. Security
    • 3.2. Search for resources
    • 3.3. Obtaining resource access URL
    • 3.4. Accessing the resource and actuating and invoking service for default (dummy) resources
  • 4. Creating RAP plugin
    • 4.1. Customizing internal RAP plugin
    • 4.2. Using RAP plugin starter
    • 4.3. Creating RAP plugin in other languages
    • 4.4. Development environment
    • 4.5. Deploying custom plugin
  • 5. Resource Description Examples
    • 5.1. JSON Description Examples
    • 5.2. RDF Description Examples
  • 6. Helper shell scripts

Cloud Deployment

This document explains how to run the symbIoTe cloud for L1/L2 integration depending on your needs. There are two ways to run docker:

1. directly (on linux)
2. docker-machine (previously boot2docker) - in virtual machine

This influences how to expose port in virtual machine to local machine. So, there are 2 ways to run ngrok and 2 ways to expose port in production environment.

1. Preparation steps

1.1. Install prerequisites

Install following:

• docker (18.03.x),
• docker-compose (1.21.x),
• docker-machine (0.14.x),
• bash,
• curl,
• wget

1.2. Register user and configure your platform

Before registration you need to know if you are going to use server:

• with public IP address (production server) or
• with local address and tunneling to public by using ngrok (hackaton environment).

If you are using public IP address then you need to obtain HTTPS certificate. One way to do that is by using let's encrypt or some other provider (see documentation)

Otherwise you will use hackaton environment with ngrok (see documentation)

Next is to register platform in symbIoTe core. Here is documentation.

When getting the platform configuration you have to enter the following options:

• Compliance Level: L1 or L2 (depending on the compliance level desired)
• Deployment Type: Docker

1.3. Create folder for your deployment

---

IMPORTANT NOTE

Starting from release 3.1.0, there are 2 extra steps needed as noted below.

---

• mkdir symbiote-cloud
• change to that directory (e.g. cd symbiote-cloud)
• unzip the configuration files to the directory from the .zip downloaded earlier
• enter the CloudConfigProperties folder, make any changes (if) you want in the properties (e.g. rabbitMQ credentials) and commit the changes (Starting from release 3.1.0):

cd CloudConfigProperties
git remote rm origin
git config user.email "[email protected]"
git config user.name "Your Name"
git commit -am "Platform configuration"
cd ..

• create a docker volume to hold the CloudConfigProperties. Replace the {docker stack name} (e.g. symbiote-app) in the command below with your selected docker stack name (Starting from release 3.1.0):

docker container run --rm -v $PWD/CloudConfigProperties:/source -v {docker stack name}_symbiote-vol-config:/home/CloudConfigProperties -w /source alpine cp -r . /home/CloudConfigProperties/

• copy the docker-compose files for the desired compliance level:
  • docker-compose-swarm-L1.yml and docker-compose-prod-swarm-L1.yml for L1 integration. Also, docker-compose-ngrok-swarm-L1.yml to run ngrok.
  • docker-compose-swarm-L2.yml and docker-compose-prod-swarm-L2.yml for L2 integration. Also, docker-compose-ngrok-swarm-L2.yml to run ngrok.
• Uncomment and configure the proxy settings in the docker compose files for L1/L2 (docker-compose-swarm-L1.yml, docker-compose-swarm-L2.yml) if you are behind a proxy.
• example of implementation of ResourceAccessProxyPlugin:
  • L1: docker-compose-swarm-custom.yml
  • L2: docker-compose-swarm-custom.yml
• In some YML files are parts that are configuring PROXY. If you use proxy it should be uncommented and changed according to your proxy server and if you do not use it please comment those parts by putting # in the beginning of line. Parameters related to proxy are: JAVA_HTTP_PROXY, JAVA_HTTPS_PROXY, JAVA_SOCKS_PROXY and JAVA_NON_PROXY_HOSTS.

2. Configuring and starting components depending on your preferred deployment

Here is table with instructions based on preferred deployment:

Environment	docker directly (on linux)	docker-machine
production environment	section 2.1	section 2.2
hackathon environment (ngrok)	section 2.3	section 2.4

---

IMOPRTANT NOTE:

The compose files have been configured to block incoming external traffic to components other than the nginx when docker compose is used. However, this is not possible when using docker swarm. Consequently, the Platform Owners need to block the incoming external traffic to components by using other means e.g. iptables in all the swarm nodes.

---

2.1. Production without docker-machine (boot2docker)

1. Obtain certificate as described in https://github.com/symbiote-h2020/SymbioteCloud/wiki/2.1-Configuration-of-NGINX#2111-obtaining-the-ssl-certificate or by some other provider

   • Create dir nginx-certificates $ mkdir nginx-certificates
   • Copy certificates to nginx-certificates

$ sudo cp /etc/letsencrypt/live/{your domain}/fullchain.pem nginx-certificates/
$ sudo cp /etc/letsencrypt/live/{your domain}/privkey.pem nginx-certificates/
$ sudo chown -R {user}:{group} nginx-certificates

2. Run docker swarm init if the node is not a swarm manager. We use the swarm mode so that secrets are encrypted during transit and at rest. Docker secrets are only available to swarm services and not to standalone containers.

3. Run:

   • docker stack deploy -c docker-compose-swarm-L1.yml -c docker-compose-prod-swarm-L1.yml -c docker-compose-swarm-custom.yml symbiote-app to deploy the application in L1 compliance or
   • docker stack deploy -c docker-compose-swarm-L2.yml -c docker-compose-prod-swarm-L2.yml -c docker-compose-swarm-custom.yml symbiote-app for L2 compliance. You can change the name of the service stack (symbiote-app) to be used.

4. You can run:

   • docker stack ls to list the stack and check the number of services used.
   • docker image ls to check that all images have been created. It may take a while to pull all the images from DockerHub for the first time.
   • docker service ls to list the services and check their status. Wait until the actual number of tasks (replicas) for each service is not 0.
   • docker logs <container_name> -f to get access to and follow the logs of a service. A component is ready when a message similar to 'Started <component_name> in 105.045 seconds (JVM running for 112.933)' appears in the logs of the container. We recommend to look for logs in containers: config, aam and rap. NOTE: look for helper shell scripts.
   • sudo service docker restart to restart the docker service if needed and repeat the steps.

5. Run docker stack rm symbiote-app to stop the application and remove the service stack. Services, networks, and secrets associated with the stack will be removed.

6. You can run docker swarm leave --force to leave the swarm.

2.2. Production with docker-machine (boot2docker)

1. Obtain certificate as described previously
2. The following command maps
   • port 443 in default docker-machine (127.0.0.1:443) to
   • local computer at port 443 on all interfaces (0.0.0.0:443) docker-machine ssh default -L 0.0.0.0:443:127.0.0.1:443
3. Run steps 2-6 as described previously in Section 2.1.

2.3. Hackathon without docker-machine (boot2docker)

Before starting with hackathon environment you need to install ngrok (https://github.com/symbiote-h2020/SymbioteCloud/wiki/2.1-Configuration-of-NGINX#2121-ngrok).

1. Run docker swarm init if the node is not a swarm manager. We use the swarm mode so that secrets are encrypted during transit and at rest. Docker secrets are only available to swarm services and not to standalone containers.
2. Run
   • docker stack deploy -c docker-compose-swarm-L1.yml -c docker-compose-ngrok-swarm-L1.yml -c docker-compose-swarm-custom.yml symbiote-app to deploy the application in L1 compliance or
   • docker stack deploy -c docker-compose-swarm-L2.yml -c docker-compose-ngrok-swarm-L2.yml -c docker-compose-swarm-custom.yml symbiote-app for L2 compliance.
3. You can run:
   • docker stack ls to list the stack and check the number of services used.
   • docker image ls to check that all images have been created. It may take a while to pull all the images from DockerHub for the first time.
   • docker service ls to list the services and check their status. Wait until the actual number of tasks (replicas) for each service is not 0.
   • docker logs <container_name> -f to get access to and follow the logs of a service. A component is ready when a message similar to 'Started <component_name> in 105.045 seconds (JVM running for 112.933)' appears in the logs of the container. We recommend to look for logs in containers: config, aam and rap. NOTE: look for helper shell scripts.
   • sudo service docker restart to restart the docker service if needed and repeat the steps.
4. Run docker stack rm symbiote-app to stop the application and remove the service stack. Services, networks, and secrets associated with the stack will be removed.

2.4. Hackathon with docker-machine (boot2docker)

Before starting with hackathon environment you need to install ngrok (https://github.com/symbiote-h2020/SymbioteCloud/wiki/2.1-Configuration-of-NGINX#2121-ngrok).

1. Run docker swarm init if the node is not a swarm manager. We use the swarm mode so that secrets are encrypted during transit and at rest. Docker secrets are only available to swarm services and not to standalone containers.
2. Run docker-machine ssh default -L 0.0.0.0:8102:127.0.0.1:8102 for mapping port in virtual machine to this computer.
3. Run
   • docker stack deploy -c docker-compose-swarm-L1.yml -c docker-compose-ngrok-swarm-L1.yml -c docker-compose-swarm-custom.yml symbiote-app to deploy the application in L1 compliance or
   • docker stack deploy -c docker-compose-swarm-L2.yml -c docker-compose-ngrok-swarm-L2.yml -c docker-compose-swarm-custom.yml symbiote-app for L2 compliance.
4. You can run:
   • docker stack ls to list the stack and check the number of services used.
   • docker image ls to check that all images have been created. It may take a while to pull all the images from DockerHub for the first time.
   • docker service ls to list the services and check their status. Wait until the actual number of tasks (replicas) for each service is not 0.
   • docker logs <container_name> -f to get access to and follow the logs of a service. A component is ready when a message similar to 'Started <component_name> in 105.045 seconds (JVM running for 112.933)' appears in the logs of the container. We recommend to look for logs in containers: config, aam and rap. NOTE: look for helper shell scripts.
   • sudo service docker restart to restart the docker service if needed and repeat the steps.
5. Run docker stack rm symbiote-app to stop the application and remove the service stack. Services, networks, and secrets associated with the stack will be removed.

2.5. Manage resources

Here is documentation

3. Test integrated resource

Here is documentation

3.1. Security

Here is documentation

3.2. Search for resources

Here is documentation

3.3. Obtaining resource access URL

Here is documentation

3.4. Accessing the resource and actuating and invoking service for default (dummy) resources

Here is documentation

4. Creating RAP plugin

Here is documentation

4.1. Customizing internal RAP plugin

Here is documentation

4.2. Using RAP plugin starter

Here is documentation

4.3. Creating RAP plugin in other languages

Here is documentation

4.4. Development environment

During development and testing of RAP plugin it is convenient to run it in IDE on local machine. In that case follow steps in this document.

4.5. Deploying custom plugin

When you are finished with your custom RAP plugin and you have tested it (explained in section 4.4). Then you need to deploy it in docker environment. You need to do following:

1. Create docker image from builded jar
2. Upload image to machine where is running docker implementation
   • you can export image to file, transfer file to machine with docker implementation and import image or
   • you can upload image to docker hub and use image from there
3. change docker-compose-swarm-custom.yml to use your image
4. restart docker swarm stack

5. Resource Description Examples

Here is documentation

5.1. JSON Description Examples

Here is documentation

5.2. RDF Description Examples

Here is documentation

6. Helper shell scripts

In order to easily check logs in docker containers and to run commands to in specific container we have prepared two scripts:

• dl <container name> - print logs in container which in name has container name provided. E.g. dl aam
• dsh <container name> - run bash in container which in name has container name provided. E.g. dsh aam

Here is script that you can add to .bashrc or .zshrc:

# Docker logs - argument is partial name of the container. It must be just one container with that name
function dl() {
  docker logs $(docker ps -f name=$1 --format "{{.ID}}") -f
}

# Docker shell - argument is partial name of the container. It must be just one container with that name
function dsh() {
  docker exec -it -e "TERM=xterm-256color" $(docker ps -f name=$1 --format "{{.ID}}") /bin/bash
}