Merge pull request #64 from DSD-DBS/staging

Staging

.gitignore

@@ -5,4 +5,6 @@
 /values*.yaml
 /secrets*.yaml
 /local
-.mypy_cache
+.mypy_cache
+deployments/*
+certs/*

CONTRIBUTING.md

@@ -0,0 +1,201 @@
+# Contributing
+
+Thanks for your interest in our project. Contributions are always welcome!
+
+We are committed to fostering a welcoming, respectful, and harassment-free
+environment. Be kind!
+
+If you have questions, ideas or want to report a bug, feel free to [open an
+issue](/../../issues). Or go ahead and [open a pull request](/../../pulls) to contribute
+code. In order to reduce the burden on our maintainers, please make sure that
+your code follows our style guidelines outlined below.
+
+## General 
+
+This project consists of several services. Here is the architecture of the services: 
+![Capella Collab Manager architecture](doc/architecture.png)
+
+To get an overview of the services, it is also worth taking a look at the Helm Chart, 
+which can be found in the `helm` folder.
+
+We use REST APIs for the communication between frontend and backend. 
+Please follow the [RESTful web API design best practises](https://docs.microsoft.com/en-us/azure/architecture/best-practices/api-design). 
+
+We recommend to get started with the [local k8d deployment](README.md).
+
+## Capella Dockerimages
+Please follow the [README of the Capella Dockerimages repository](/../../../capella-dockerimages/blob/main/README.md). 
+When all files are in the right place, you can execute the following command to build and push the Dockerimages. 
+```
+make capella-dockerimages
+```
+
+## OAuth Server
+We use OAuth2 as authentication protocol for our application. Therefore we need an OAuth2 server.
+For local development, we have an OAuth mock that needs to be started first.
+
+First of all, navigate to the `mocks/oauth` directory.
+
+You can handle the SSL-support in two different ways: 
+
+### Option 1: Use insecure connections
+This is the default and recommended option for local development. Do not use it in production!
+1. Run the OAuth2-Mock-Server with: 
+   ```sh
+   make start
+   ```
+2. Verify that the server runs, e.g. by navigating to [Well Known](http://localhost:8080/default/.well-known/openid-configuration)
+
+### Option 2: Create a self signed certificate
+1. We require the files `projectroot/certs/localhost.cert` and `projectroot/certs/localhost.p12`.
+    You can easily create them with
+    ```sh
+    make ssl
+    ```   
+2. Import the certificate to your local certificate store.
+3. Run the OAuth2-Mock-Server with: 
+   ```sh
+   make start-ssl
+   ```
+4. Verify that the server runs, e.g. by navigating to [Well Known](https://localhost:8083/default/.well-known/openid-configuration)
+
+## Backend
+
+Requirements:
+- `python` and `pip`
+- `docker`
+- `make`
+
+Run the following steps: 
+1. Navigate to the `backend` directory of your cloned repository.
+2. We recommend that you develop inside of a virtual environment. To set it up,
+   run the following commands:
+
+   ```sh
+   python3 -m venv .venv39
+   source .venv39/bin/activate
+   pip install -U pip setuptools
+   pip install -e '.[dev]'
+   ```
+3. The backend uses various configuration settings. You can find them in the `config` directory. 
+   Please copy the file `config_template.yaml` to `config.yaml` and adjust the values.
+
+   *Hint*: If you already have the k8d cluster running and the if you have the application deployed, 
+   then no configuration values need to be adjusted.
+
+4. This step in only **necessary, if you use the self signed certificate** option for the oauth mock).
+   If you don't have the certificate in your local certificate store, please execute the following command: 
+   ```
+   export REQUESTS_CA_BUNDLE=$(pwd)/../certs/localhost.crt
+   ```
+  
+   You need to adjust the option `` in the `config.yaml` to the following value: 
+   ```
+   https://localhost:8083/default/.well-known/openid-configuration
+   ```
+5. A PostgreSQL database is required. You can run the following command to set up the database: 
+   ```sh
+   make database
+   ```
+6. Start the application with: 
+   ```sh
+   make app
+   ```
+7. You should see it running on port 8000. 
+   - [Healthcheck](http://localhost:8000/healthcheck)
+   - [Documentation](http://localhost:8000/docs)
+
+We additionally recommend that you set up your editor / IDE as follows.
+
+- Indent with 4 spaces per level of indentation
+- Maximum line length of 79 (add a ruler / thin line / highlighting / ...)
+- Set up the editor to run `black`, `pylint`, `mypy` and `isort` when saving.
+
+### Create database migrations scripts
+
+To create an upgrade script automatically (this will compare the current database state with the models): 
+```
+cd t4cclient
+alembic revision --autogenerate -m "Commit message"
+```
+
+## Frontend
+
+Requirements:
+- `npm` package manager
+- [Angular CLI](https://angular.io/cli#installing-angular-cli)
+- `make`
+
+Run the following steps:
+1. Navigate to the `frontend` folder
+2. Optional: If you like to use your custom favicon, please copy it to `src/favicon.ico`
+3. Optional: If you like to use your custom theme, replace the file `src/custom-theme.scss`. 
+   You can generate custom themes [here](http://mcg.mbitson.com/)
+4. Copy the file `src/environment.ts` to `src/environment.dev.ts` and adjust the values.
+5. Run the frontend with: 
+   ```sh
+   make dev
+   ```
+6. You should see the frontend running von port 4200.
+
+We additionally recommend that you set up your editor / IDE as follows.
+- Set up the editor to run [prettier](https://prettier.io/) when saving.
+
+# Code style
+
+## Backend 
+
+We base our code style on a modified version of the [Google style guide for
+Python code](https://google.github.io/styleguide/pyguide.html). The key
+differences are:
+
+- **Docstrings**: The [Numpy style
+  guide](https://numpydoc.readthedocs.io/en/latest/format.html) applies here.
+- **Linting**: Use [pylint](https://github.com/PyCQA/pylint) for static code
+  analysis, and [mypy](https://github.com/python/mypy) for static type
+  checking.
+- **Formatting**: Use [black](https://github.com/psf/black) as code
+  auto-formatter.
+
+  Be aware of the different line length of 72 for docstrings. We currently do
+  not have a satisfactory solution to automatically apply or enforce this.
+
+  Note that, while you're encouraged to do so in general, it is not a hard
+  requirement to break up long strings into smaller parts. Additionally, never
+  break up strings that are presented to the user in e.g. log messages, as that
+  makes it significantly harder to grep for them.
+
+  Use [isort](https://github.com/PyCQA/isort) for automatic sorting of imports.
+  Its settings should automatically be picked up from the `pyproject.toml` file
+  as well.
+- **Typing**: We do not make an exception for `typing` imports. Instead of
+  writing `from typing import SomeName`, use `import typing as t` and access
+  typing related classes like `t.TypedDict`.
+
+  Use the new syntax and classes for typing introduced with Python 3.10 and
+  available using `from __future__ import annotations` since Python 3.8. Be
+  aware however that this only works in the context of annotations; the code
+  still needs to run on Python 3.8! This means that in some (rare) cases, you
+  *must* use the old-style type hints.
+
+  - Instead of `t.Tuple`, `t.List` etc. use the builtin classes `tuple`, `list`
+    etc.
+  - For classes that are not builtin (e.g. `Iterable`), `import collections.abc
+    as cabc` and then use them like `cabc.Iterable`.
+  - Use [PEP-604](https://www.python.org/dev/peps/pep-0604/)-style unions, e.g.
+    `int | float` instead of `t.Union[int, float]`.
+  - Use `... | None` (with `None` always as the last union member) instead of
+    `t.Optional[...]` and always explicitly annotate where `None` is possible.
+- **Python style rules**: For conflicting parts, the [Black code
+  style](https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html)
+  wins. If you have set up `black` correctly, you don't need to worry about
+  this though :)
+- When working with `dict`s, consider using `t.TypedDict` instead of a more
+  generic `dict[str, float|int|str]`-like annotation where possible, as the
+  latter is much less precise (often requiring additional `assert`s or
+  `isinstance` checks to pass) and can grow unwieldy very quickly.
+- Prefer `t.NamedTuple` over `collections.namedtuple`, because the former uses
+  a more convenient `class ...:` syntax and also supports type annotations.
+
+
+

Makefile

@@ -1,59 +1,71 @@
 # Copyright DB Netz AG and the capella-collab-manager contributors
 # SPDX-License-Identifier: Apache-2.0
 
-CLUSTER_NAME = mycluster
+CLUSTER_NAME = collab-cluster
 LOCAL_REGISTRY_NAME = localhost
 CLUSTER_REGISTRY_NAME = myregistry.localhost
 REGISTRY_PORT = 12345
 RELEASE = dev-t4c-manager
 NAMESPACE = t4c-manager
-MY_EMAIL ?= [email protected]
 EASE_DEBUG_PORT = 3390
 
 all: backend frontend
 
-build: backend frontend capella ease
+build: backend frontend capella
+
+build-all: build ease
 
 backend:
-	docker build -t t4c/client/backend -t $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/t4c/client/backend backend
-	docker push $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/t4c/client/backend
+	docker build -t t4c/client/backend -t $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/capella/collab/backend backend
+	docker push $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/capella/collab/backend
 
 frontend:
-	docker build -t t4c/client/frontend -t $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/t4c/client/frontend frontend
-	docker push $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/t4c/client/frontend
+	docker build --build-arg CONFIGURATION=local -t t4c/client/frontend -t $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/capella/collab/frontend frontend
+	docker push $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/capella/collab/frontend
 
-capella:
+capella: capella-download
 	docker build -t base capella-dockerimages/base
 	docker build -t capella/base capella-dockerimages/capella
 	docker build -t capella/remote -t $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/t4c/client/remote capella-dockerimages/remote
 	docker push $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/t4c/client/remote
 
-t4c-client: 
+capella-download:
+	cd capella-dockerimages/capella/archives; \
+	if [[ -f "capella.tar.gz" ]] || [[ -f "capella.zip" ]]; \
+	then \
+		echo "Found existing capella archive."; \
+	else \
+		curl -L --output capella.tar.gz 'https://ftp.acc.umu.se/mirror/eclipse.org/capella/core/products/releases/5.2.0-R20211130-125709/capella-5.2.0.202111301257-linux-gtk-x86_64.tar.gz'; \
+	fi
+
+t4c-client:
 	docker build -t t4c/client/base capella-dockerimages/t4c
 
-readonly: 
+readonly:
 	docker build -t capella/ease --build-arg BASE_IMAGE=capella/base --build-arg BUILD_TYPE=online capella-dockerimages/ease
 	docker build -t capella/ease/remote --build-arg BASE_IMAGE=capella/ease capella-dockerimages/remote
 	docker build -t $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/capella/readonly --build-arg BASE_IMAGE=capella/ease/remote capella-dockerimages/readonly
 	docker push $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/capella/readonly
 
-ease: 
+ease:
 	docker build -t $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/t4c/client/ease --build-arg BASE_IMAGE=t4c/client/base --build-arg BUILD_TYPE=online capella-dockerimages/ease
 	docker push $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/t4c/client/ease
 
-mock: 
+mock:
 	docker build -t t4c/server/mock -t $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/t4c/server/mock mocks/t4c-server
 	docker push $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/t4c/server/mock
 
 	docker build -t t4c/licence/mock -t $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/t4c/licence/mock mocks/licence-server
 	docker push $(LOCAL_REGISTRY_NAME):$(REGISTRY_PORT)/t4c/licence/mock
 
-deploy: backend frontend capella mock helm-deploy
+capella-dockerimages: capella t4c-client readonly ease
+
+deploy: oauth-mock backend frontend capella mock helm-deploy open
 
 # Deploy with full T4C support:
 deploy-t4c: backend frontend capella t4c-client readonly-ease mock helm-deploy
 
-helm-deploy: 
+helm-deploy:
 	k3d cluster list $(CLUSTER_NAME) 2>&- || $(MAKE) create-cluster
 	kubectl create namespace t4c-sessions || true
 	helm upgrade --install \
@@ -62,25 +74,34 @@ helm-deploy:
 		--namespace $(NAMESPACE) \
 		--values helm/values.yaml \
 		$$(test -f secrets.yaml && echo "--values secrets.yaml") \
-		--set docker.registry=k3d-$(CLUSTER_REGISTRY_NAME):$(REGISTRY_PORT) \
-		--set database.backend.initialAdmin=$(MY_EMAIL) \
-		--set general.port=8081 \
+		--set docker.registry.internal=k3d-$(CLUSTER_REGISTRY_NAME):$(REGISTRY_PORT) \
+		--set general.port=8080 \
 		--set t4cServer.apis.usageStats="http://$(RELEASE)-licence-server-mock:80/mock" \
 		--set t4cServer.apis.restAPI="http://$(RELEASE)-t4c-server-mock:80/mock/api/v1.0" \
-		--wait --timeout 4m \
+		--wait --timeout 10m \
 		--debug \
 		$(RELEASE) ./helm
-	$(MAKE) .rollout .provision-guacamole .provision-backend
-
-clear-backend-db: 
+	$(MAKE) .provision-guacamole .provision-backend
+
+open:
+	export URL=http://localhost:8080; \
+	if [[ "Windows_NT" == "$(OS)" ]]; \
+	then \
+		start "$$URL"; \
+	elif [[ "$(shell uname -s)" == "Linux" ]]; \
+	then \
+		xdg-open "$$URL"; \
+	elif [[ "$(shell uname -s)" == "Darwin" ]]; \
+	then \
+		open "$$URL"; \
+	fi
+
+clear-backend-db:
 	kubectl delete deployment -n t4c-manager $(RELEASE)-backend-postgres
 	kubectl delete pvc -n t4c-manager $(RELEASE)-volume-backend-postgres
 	$(MAKE) helm-deploy
 
 rollout: backend frontend
-	$(MAKE) .rollout
-
-.rollout: 
 	kubectl --context k3d-$(CLUSTER_NAME) rollout restart deployment -n $(NAMESPACE) $(RELEASE)-backend
 	kubectl --context k3d-$(CLUSTER_NAME) rollout restart deployment -n $(NAMESPACE) $(RELEASE)-frontend
 
@@ -93,35 +114,39 @@ create-cluster:
 	k3d registry list $(CLUSTER_REGISTRY_NAME) 2>&- || k3d registry create $(CLUSTER_REGISTRY_NAME) --port $(REGISTRY_PORT)
 	k3d cluster list $(CLUSTER_NAME) 2>&- || k3d cluster create $(CLUSTER_NAME) \
 		--registry-use k3d-$(CLUSTER_REGISTRY_NAME):$(REGISTRY_PORT) \
-		--port "8081:80@loadbalancer"
+		--port "8080:80@loadbalancer"
 	kubectl cluster-info
 
 delete-cluster:
 	k3d cluster list $(CLUSTER_NAME) 2>&- && k3d cluster delete $(CLUSTER_NAME)
 	rm -f .provision-guacamole .provision-backend
 
 .provision-guacamole:
+	export MSYS_NO_PATHCONV=1; \
 	kubectl exec --namespace $(NAMESPACE) $$(kubectl get pod --namespace $(NAMESPACE) -l id=$(RELEASE)-deployment-guacamole-guacamole --no-headers | cut -f1 -d' ') -- /opt/guacamole/bin/initdb.sh --postgres | \
 	kubectl exec -ti --namespace $(NAMESPACE) $$(kubectl get pod --namespace $(NAMESPACE) -l id=$(RELEASE)-deployment-guacamole-postgres --no-headers | cut -f1 -d' ') -- psql -U guacamole guacamole && \
 	touch .provision-guacamole
 
 .provision-backend:
-	echo "insert into repository_user_association values ('$(MY_EMAIL)', 'default', 'WRITE', 'MANAGER');" | kubectl exec -ti --namespace $(NAMESPACE) $$(kubectl get pod --namespace $(NAMESPACE) -l id=$(RELEASE)-deployment-backend-postgres --no-headers | cut -f1 -d' ') -- psql -U backend backend && \
+	echo "insert into repository_user_association values ('$(MY_EMAIL)', 'default', 'WRITE', 'MANAGER');" | kubectl exec --namespace $(NAMESPACE) $$(kubectl get pod --namespace $(NAMESPACE) -l id=$(RELEASE)-deployment-backend-postgres --no-headers | cut -f1 -d' ') -- psql -U backend backend && \
 	touch .provision-backend
 
-.PHONY: backend frontend capella deploy undeploy create-cluster delete-cluster persistent-volume
+.PHONY: backend frontend capella oauth-mock deploy undeploy create-cluster delete-cluster persistent-volume ns
 
 # Execute with `make -j2 dev`
 dev: dev-frontend dev-backend
-	
-dev-frontend: 
+
+dev-frontend:
 	$(MAKE) -C frontend dev
 
-dev-backend: 
+dev-backend:
 	$(MAKE) -C backend dev
 
-dev-cleanup: 
+dev-cleanup:
 	$(MAKE) -C backend cleanup
 
-backend-logs: 
-	kubectl logs -f -n $(NAMESPACE) -l id=$(RELEASE)-deployment-backend
+backend-logs:
+	kubectl logs -f -n $(NAMESPACE) -l id=$(RELEASE)-deployment-backend
+
+ns:
+	kubectl config set-context k3d-$(CLUSTER_NAME) --namespace=$(NAMESPACE)