SAVE Cloud deployment configuration

Components

SAVE Cloud contains the following microservices:

backend: REST API for DB
test-preprocessor: clones projects for test and discovers tests
orchestrator: moderates distributed execution of tests, feeds new batches of tests to a set of agents save-cloud uses MySQL as a database. Liquibase (via gradle plugin) is used for schema initialization and migration.

Building

Prerequisites: some components require additional system packages. See save-agent description for details. save-frontend requires node.js installation.
To build the project and run all tests, execute ./gradlew build.
For deployment, all microservices are packaged as docker images with the version based on latest git tag and latest commit hash, if there are commits after tag.

Deployment is performed on server via docker swarm or locally via docker compose. See detailed information below.

Dependency to `save-cli`

The libs.version.toml contains save-cli version. The save-agent uses this version as a compile dependency to read execution's reports.

The save-backend and save-sandbox download newer versions of save-cli from GitHub on startup.

Using a `SNAPSHOT` version of `save-cli`

If save-cli is set to snapshot version in lib.version.toml, we download save-cli's sources and build them in GitHub action: Build and push Docker images. Then Gradle adds the result (.kexe) to save-backend and save-sandbox as a runtime dependency

Under the hood: Gradle supports two variables saveCliVersion and saveCliPath. The saveCliVersion overrides version of save-cli from lib.version.toml. The saveCliPath specifies a path to save-cli's .kexe and it's required when version of save-cli is SNAPSHOT.

Note: libs.version.toml can contain blabla-SNAPSHOT version, but we will build a version from the latest main in save-cli and set the built version of save-cli to generated file: generated/SaveCliVersion.kt.

Server deployment

Server should run Linux and support docker swarm and gvisor runtime. Ideally, kernel 5.+ is required.
reverse-proxy.conf is a configuration for Nginx to act as a reverse proxy for save-cloud. It should be copied into /etc/nginx/sites-available.
Gvisor should be installed and runsc runtime should be available for docker. See installation guide for details. A different runtime can be specified with orchestrator.docker.runtime property in orchestrator.
Ensure that docker daemon is running and that docker is in swarm mode.
Secrets should be added to the swarm as well as to $HOME/secrets file.
If custom SSL certificates are used, they should be installed on the server and added into JDK's truststore inside images. See section below for details.
Loki logging driver should be added to docker installation: instruction
Pull new changes to the server and run ./gradlew -Psave.profile=prod deployDockerStack.
- If you wish to deploy save-cloud, that is not present in docker registry (e.g. to deploy from a branch), run ./gradlew -Psave.profile=prod buildAndDeployDockerStack instead.
- If you would like to use docker-compose.override.yaml, add -PuseOverride=true to the execution of tasks above. This file is configured to be read from $HOME/configs; you can use the one from the repository as an example.
docker-compose.yaml is configured so that all services use Loki for logging and configuration files from ~/configs, which are copied from save-deploy during gradle build.

Override configuration per server

If you wish to customize services configuration externally (i.e. leaving docker images intact), this is possible via additional properties files. In docker-compose.yaml all services have /home/saveu/configs/<service name> directory mounted. If it contains application.properties file, it will override config from default application.properties.

Running behind proxy

If save-cloud is running behind proxy, docker daemon should be configured to use proxy. See docker docs. Additionally, use /home/saveu/configs/orchestrator/application.properties to add two flags to apt-get:

orchestrator.aptExtraFlags=-o Acquire::http::proxy="http://host.docker.internal:3128" -o Acquire::https::proxy="http://host.docker.internal:3128"

Proxy URLs will be resolved from inside the container.

Custom SSL certificates

If custom SSL certificates are used, they should be installed on the server and added into JDK's truststore inside images. One way of adding them into JDK is to mount them in docker-compose.yaml and then override default command:

preprocessor:
  volume:
    - '/path/to/certs/cert.cer:/home/cnb/cert.cer'
  entrypoint: /bin/bash
  command: -c 'find /layers -name jre -type d -exec {}/bin/keytool -keystore {}/lib/security/cacerts -storepass changeit -noprompt -trustcacerts -importcert -alias <cert-alias> -file /home/cnb/cert.cer \; && /cnb/process/web'

Database

The service is designed to work with MySQL database. Migrations are applied with liquibase. They expect event scheduler to be enabled on the DB.

Enabling api-gateway with external OAuth providers

In the file /home/saveu/configs/gateway/application.properties the following properties should be provided:

spring.security.oauth2.client.provider.<provider name>.issuer-uri
spring.security.oauth2.client.registration.<provider name>.client-id
spring.security.oauth2.client.registration.<provider name>.client-secret

Local deployment

Usually, not the whole stack is required for development. Application logic is performed by save-backend, save-orchestrator and save-preprocessor, so most time you'll need those three.

Ensure that docker daemon is running and docker compose is installed.
- If running on a system without Unix socket connection to the Docker Daemon (e.g. with Docker for Windows), docker daemon should have HTTP port enabled. Then, docker-tcp profile should be enabled for orchestrator.
To make things easier, add line save.profile=dev to gradle.properties. This will make project version SNAPSHOT instead of timestamp-based suffix and allow caching of gradle tasks.
Run ./gradlew deployLocal -Psave.profile=dev to start the database and run three microservices (backend, preprocessor and orchestrator) with Docker Compose. Run ./gradlew -Psave.profile=dev :save-frontend:run to start save-frontend using webpack-dev-server, requests to REST API will be proxied as configured in dev-server.js.
For developing most part of platform's logic, the above will be enough. If local testing of authentication flow is required, however, api-gateway can be run locally together with dex OAuth2 server. In order to do so, run docker compose up -d dex and then start api-gateway with dev profile enabled. Using application-dev.yaml one can connect gateway with dev build of frontend running with webpack by changing gateway.frontend.url.

Using OAuth with a local deployment

When the default dev-server.js is used, the front-end is expected to communicate directly with the back-end, omitting any gateway. When enabling OAuth, make sure the gateway is contacted instead:
- context: add /sec/**, /oauth2/**, and /login/oauth2/** to the list;
- target: change to http://localhost:5300 (the default gateway URL);
- onProxyReq: drop the entire callback, since both headers (Authorization and X-Authorization-Source) will be set by the gateway now (the gateway acts as a reverse proxy);
- bypass: drop the entire callback.
The resulting dev-server.js should look like this:
```
config.devServer = Object.assign(
    {},
    config.devServer || {},
    {
      proxy: [
        {
          context: ["/api/**", "/sec/**", "/oauth2/**", "/login/oauth2/**", "**.ico", "**.png"],
          target: 'http://localhost:5300',
          logLevel: 'debug',
        }
      ]
    }
)
```
Avoid potential name conflicts between local users (those authenticated using HTTP Basic Auth) and users created via an external OAuth provider. For example, if you have a local user named torvalds, don't try to authenticate as a GitHub user with the same name.

Dex-specific notes

In order to run Dex, you need a build/docker-compose.yaml file generated. This is done by running
```
./gradlew generateComposeFile
```
The YML configuration file, dex.dev.yaml, has a syntax explained here and here.
More users can be added using a static configuration via dex.dev.yaml. Essential fields explained:
- hash: the bcrypt hash of the password string:
```
echo 'password' | htpasswd -BinC 16 'user' | cut -d: -f2
```
  The htpasswd utility is a part of apache2-utils package. The maximum cost supported by htpasswd is 17. Dex, on the other hand, only allows values up to 16.
- username: this is the name of the user from Dex perspective only. Since Dex (unlike GitHub), provides no means to query user details (i.e. it has no User API), the auto-generated username in the user table will initially look like CiRlOGI3NWFmNC1kMDkzLTRhZjUtODk3NC0xMzZlY2IxMGNiNzcSBWxvY2Fs (example).
- userID: a version 4 (random) GUID (DCE 1.1, ISO/IEC 11578:1996), can be generated online, or using uuidgen -r , uuid, or uuidcdef -u (Linux), or by running
```
python3 -c 'import uuid; print(str(uuid.uuid4()))'
```
For debugging purposes, you may wish to run Dex in the foreground:
```
docker compose up dex
```

GitHub-specific notes

The spring.security.oauth2.client.provider.github.user-name-attibute under application.yml or application properties should be set to login. This is because in the default configuration (o.s.s.c.o.c.CommonOAuth2Provider#GITHUB), the numeric id field is taken from the JSON response received from api.github.com/user, and we want the publicly-visible login value instead. See GitHub User API for more details.
To use GitHub as an OAuth provider, you'll need to create a GitHub OAuth application. Essential fields explained:
- Client ID: the unique application id, which will appear in the outgoing requests from the gateway to GitHub. Configure the gateway accordingly by setting the spring.security.oauth2.client.registration.github.client-id property.
- Client secrets: holds the secret the gateway will use to authenticate itself at GitHub. Store it in the spring.security.oauth2.client.registration.github.client-secret property.
- Homepage URL: should be set to your font-end URL, i.e. http://localhost:8080.
- Authorization callback URL: holds the URL GitHub will redirect to once it successfully authenticates a user. Should be exactly http://localhost:8080/login/oauth2/code/github.
- Enable Device Flow: leave enabled.
The resulting application settings may look like this: screenshot.

Local debugging

You can run backend, orchestrator, preprocessor and frontend locally in IDE in debug mode.

Using `save-agent` executable on Windows

If you run on Windows, dependency save-agent is omitted because of problems with linking in cross-compilation. To run on Windows, you need to build and package save-agent on WSL.

When building from the WSL, better use a separate local Git repository, for two reasons:

Sometimes, WSL doesn't have enough permissions to create directories on the NTFS file system, so file access errors may occur.
Windows and Linux versions of Gradle will use different absolute paths when accessing the same local Git repository, so, unless you each time do a full rebuild, you'll encounter NoSuchFileException errors when switching from Windows to WSL and back.

Under WSL, from a separate local Git repository run:

./gradlew :save-agent:copyAgentDistribution

and provide the path to the JAR archive which contains save-agent.kexe via the saveAgentDistroFilepath Gradle property, by setting the above property either under project-specific gradle.properties, or, globally, under %USERPROFILE%\.gradle\gradle.properties, e.g.:

# gradle.properties
saveAgentDistroFilepath=file:\\\\\\\\wsl$\\Ubuntu\\home\\username\\projects\\save-cloud\\save-agent\\build\\libs\\save-agent-0.3.0-alpha.0.48+1c1fd41-distribution.jar

Using forward slashes on Windows is allowed, too (Gradle will understand such paths just fine):

# gradle.properties
saveAgentDistroFilepath=file:////wsl$/Ubuntu/home/username/projects/save-cloud/save-agent/build/libs/save-agent-0.4.0-SNAPSHOT-distribution.jar

Alternatively, you can set the property directly on the command line (-PsaveAgentDistroFilepath=...) or on a per Run Configuration basis (in IDEA).

Once the agent distribution is built and saveAgentDistroFilepath is set, you can run (on Windows):

gradlew.bat :save-backend:downloadSaveAgentDistro

or

gradlew.bat :save-backend:downloadSaveAgentDistro -PsaveAgentDistroFilepath=file:////wsl$/Ubuntu/home/username/projects/save-cloud/save-agent/build/libs/save-agent-0.4.0-SNAPSHOT-distribution.jar

Once the task completes, the agent JAR can be found under save-backend\build\agentDistro directory.

For the classpath changes to take effect:

Reload the project from disk (Project tool window in IDEA).
Reload the project model (Gradle tool window in IDEA).
Re-start the back-end application.

Then verify that the agent is indeed available for download from the S3 by checking the path s3:/cnb/cnb/files/internal-storage/latest/save-agent.kexe. It should be available by url: http://127.0.0.1:9090/browser/cnb/cnb/files/internal-storage/latest/save-agent.kexe

Similarly, troubles downloading an agent binary from the S3 can be diagnosed using docker logs (post-mortem). Here, you can see a container failing to execute the JSON data (#1663):

$ docker container ls -a | grep -F 'save-execution' | awk '{ print $1 }' | xargs -n1 -r docker logs 2>&1 | grep -F 'save-agent.kexe'
+ curl -vvv http://host.docker.internal:9000/cnb/cnb/files/internal-storage/latest/save-agent.kexe?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=DZHORWNWWGHIRY54R97V%2F20230215%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230215T082823Z&X-Amz-Expires=604800&X-Amz-Security-Token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiJEWkhPUldOV1dHSElSWTU0Ujk3ViIsImV4cCI6MTY3NjQ5MTU0NiwicGFyZW50IjoiYWRtaW4ifQ._yowS3oqSpE61BkFp7Gr0Ll9qBL4XFF9cJNT6FZBQeul-JkOaw3LGQKCIwiwvTAqXv0BRQzKAY8t4Fa82oSBLg&X-Amz-SignedHeaders=host&versionId=null&X-Amz-Signature=2bf63f08642ca46eb93752771f768504a22e303900b3dab85a50525f1981a420 --output save-agent.kexe
+ chmod +x save-agent.kexe
+ ./save-agent.kexe
./save-agent.kexe: 1: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key>cnb/files/internal-storage/latest/save-agent.kexe</Key><BucketName>cnb</BucketName><Resource>/cnb/cnb/files/internal-storage/latest/save-agent.kexe</Resource><RequestId>1743F2288515EEB0</RequestId><HostId>3f1ca0e4-b874-42fa-9843-5d2cc7de7d28</HostId></Error>: not found

Using a custom `save-cli` executable on Windows

If you need to test changes in save-cli you can also compile SNAPSHOT version of save-cli on WSL
and set saveCliPath and saveCliVersion in %USERPROFILE%\.gradle\gradle.properties
For example:

# gradle.properties
saveCliPath=file:\\\\\\\\wsl$\\Ubuntu\\home\\username\\projects\\save-cli\\save-cli\\build\\bin\\linuxX64\\releaseExecutable
saveCliVersion=0.4.0-alpha.0.42+78a24a8

the version corresponds to the file save-0.4.0-alpha.0.42+78a24a8-linuxX64.kexe

Some workarounds:

If setting save-agent's path in gradle.properties didn't help you (something doesn't work on Mac), you still can place all the files from save-agent-*-distribution.jar into save-orchestrator/build/resources/main. Moreover, if you use Mac with Apple Silicon, you should run docker-mac-settings.sh in order to let docker be available via TCP. Do not forget to use mac profile.

Ports allocation

port	description
3306	database (locally)
5800	save-backend
5810	save-frontend
5100	save-orchestrator
5200	save-test-preprocessor
5300	api-gateway
5400	save-sandbox
9090	prometheus
9091	node_exporter
9100	grafana

Secrets

Liquibase is reading secrets from the secrets file located on the server in the home directory.
PostProcessor is reading secrets for database connection from the docker secrets and fills the spring datasource. (DockerSecretsDatabaseProcessor class)
api-gateway is a single external-facing component, hence its security is stricter. Actuator endpoints are protected with basic HTTP security. Access can be further restricted by specifying gateway.knownActuatorConsumers in application.properties (if this options is not specified, no check will be performed).

Server configuration

Nginx

Nginx is used as a reverse proxy, which allows access from external network to backend and some other services. File save-deploy/reverse-proxy.conf should be copied to /etc/nginx/sites-available. Symlink should be created: sudo ln -s /etc/nginx/sites-available/reverse-proxy.conf /etc/nginx/sites-enabled/ (or to /etc/nginx/conf.d on some distributions).

Adding a new service

Sometimes it's necessary to create a new service. These steps are required to seamlessly add it to deployment:

Add it to docker-compose.yaml
Add it to task depoyDockerStack in DockerStackConfiguration.kt so that config directory is created (if it's another Spring Boot service)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

SAVE Cloud deployment configuration

Components

Building

Dependency to `save-cli`

Using a `SNAPSHOT` version of `save-cli`

Server deployment

Override configuration per server

Running behind proxy

Custom SSL certificates

Database

Enabling api-gateway with external OAuth providers

Local deployment

Using OAuth with a local deployment

Dex-specific notes

GitHub-specific notes

Local debugging

Using `save-agent` executable on Windows

Using a custom `save-cli` executable on Windows

Some workarounds:

Ports allocation

Secrets

Server configuration

Nginx

Adding a new service

Files

README.md

Latest commit

History

README.md

File metadata and controls

SAVE Cloud deployment configuration

Components

Building

Dependency to save-cli

Using a SNAPSHOT version of save-cli

Server deployment

Override configuration per server

Running behind proxy

Custom SSL certificates

Database

Enabling api-gateway with external OAuth providers

Local deployment

Using OAuth with a local deployment

Dex-specific notes

GitHub-specific notes

Local debugging

Using save-agent executable on Windows

Using a custom save-cli executable on Windows

Some workarounds:

Ports allocation

Secrets

Server configuration

Nginx

Adding a new service

Dependency to `save-cli`

Using a `SNAPSHOT` version of `save-cli`

Using `save-agent` executable on Windows

Using a custom `save-cli` executable on Windows