From 845e39b1f2fdc75dba8c59d10839e6e51b457da3 Mon Sep 17 00:00:00 2001 From: Guillaume Tucker Date: Mon, 21 Aug 2023 14:09:51 +0200 Subject: [PATCH] kube/aks: add README.md Add README.md with instructions to set up the whole KernelCI API deployment in AKS, step by step. This is how the Early Access deployment was done and it forms the basis for a production deployment candidate. Signed-off-by: Guillaume Tucker --- kube/aks/README.md | 394 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 394 insertions(+) create mode 100644 kube/aks/README.md diff --git a/kube/aks/README.md b/kube/aks/README.md new file mode 100644 index 00000000..1638d93f --- /dev/null +++ b/kube/aks/README.md @@ -0,0 +1,394 @@ + + +KernelCI API Deployment in Azure Kubernetes Services (AKS) +========================================================== + +This guide goes through all the steps required to deploy the KernelCI API +service in +[AKS](https://azure.microsoft.com/en-us/products/kubernetes-service). It +relies on an external Atlas account for MongoDB and does not include any +storage solution. + +## Azure account + +The first prerequisite is to have a [Microsoft Azure](https://azure.com) +account. If you already have one, you can skip this step and carry on with +setting up an AKS cluster. Otherwise, please create one now typically with an +`@outlook.com` email address. You can create a new address on +[outlook.com](https://outlook.com) for this purpose if needed. + +## AKS cluster + +Please create an AKS cluster via the [Azure +portal](https://portal.azure.com/#create/Microsoft.AKS). The standard settings +are fine for this use-case although you might need to adjust a few things to +match the scale of your particular deployment. + +## Atlas MongoDB account + +This AKS reference deployment relies on an Atlas account for MongoDB in order +to keep everything in the Cloud and application setup to the bare minimal. As +such, please [create an Atlas +account](https://www.mongodb.com/cloud/atlas/register) if you don't already +have one and set up a database. The MongoDB service string will be needed +later on to let the API service connect to it. + +The recommended way to set up a subscription is to create a "MongoDB Atlas +(pay-as-you-go)" resource via the Azure Marketplace. + +To set up a database: + +* create a cluster with the appropriate tier for the deployment +* add a database in the project via the web UI with "Create Database" +* go to "Database Access" to create a user and password with the + `readWriteAnyDatabase` built-in role + +You should now be able to connect to the database with a connection string of +the form `mongodb+srv://user:password@something.mongodb.net`. To verify it's +working: + +``` +$ mongo mongodb+srv://user:password@something.mongodb.net +[...] +MongoDB Enterprise atlas-ucvcf2-shard-0:PRIMARY> show databases +admin 0.000GB +kernelci 0.004GB +local 22.083GB +``` + +The string used with the `mongo` shell here is the same one that needs to be +stored as a Kubernetes secret for the API service as described in a later +section of this documentation page. + +## Command line tools + +Configuring this AKS deployment relies on `az`, `kubectl` and `helm` to be +installed. As a convenience, the `kernelci/k8s` Docker image can be used with +these tools already installed. We'll be using it in this documentation page +going forward. Here's a sample command to start a container with the Docker +image: + +``` +$ cd kube/aks +$ mkdir -p home +$ cp *.yaml home +$ docker run -it \ + -u1000 -v$PWD/home:/tmp/home -w/tmp/home -eHOME=/tmp/home \ + kernelci/k8s /bin/bash +I have no name!@e628a6d08636:~$ ls -la +total 20 +drwxr-xr-x 2 1000 1000 4096 Aug 29 11:36 . +drwxrwxrwt 1 root root 4096 Aug 29 11:36 .. +-rw-r--r-- 1 1000 1000 1089 Aug 29 11:36 api.yaml +-rw-r--r-- 1 1000 1000 878 Aug 29 11:36 ingress.yaml +-rw-r--r-- 1 1000 1000 479 Aug 29 11:36 redis.yaml +I have no name!@ace0f3581e23:~$ +``` + +The commands shown in this tutorial will be omitting the prompt up to the `$` +sign for the sake of readability. The image only has a `root` user but it's +better when accessing volumes mounted from the local file system to run the +containers with a regular user so in this case it's using anonymous user ID +`1000`. + +## Get `kubectl` credentials + +The container started in the previous section has a `home` directory mounted +from the host. This allows Azure credentials to be stored there persistently, +so if the container is restarted they'll still be available. + +To create the initial credentials, you can run the `az login` command from +within the container and then login by pasting the temporary code into the [web +page](https://microsoft.com/devicelogin): + +``` +$ az login --use-device-code +To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code XXXYYYZZZ to authenticate. +``` + +After following the instructions on the web page, you should be able to access +your AKS cluster. Still from within the container, run the following commands +with your own cluster name instead of `kernelci-api-1`: + +``` +$ az aks list -o table +Name Location ResourceGroup KubernetesVersion [...] +---------------------- -------------- ------------------------- ------------------- [...] +kernelci-api-1 eastus kernelci-api 1.26.6 [...] +$ az aks get-credentials -n kernelci-api-1 -g kernelci-api +Merged "kernelci-api-1" as current context in /tmp/home/.kube/config +$ kubectl config use-context kernelci-api-1 +Switched to context "kernelci-api-1". +$ kubectl get nodes +NAME STATUS ROLES AGE VERSION +aks-agentpool-23485665-vmss000000 Ready agent 4h3m v1.26.6 +aks-userpool-23485665-vmss000000 Ready agent 4h3m v1.26.6 +``` + +## Install extra packages in the AKS cluster + +A couple of additional packages are needed in order to have a full production +deployment with SSL encryption. This is done using `ingress-nginx` to act as a +form of proxy for incoming requests and `cert-manager` to manage the SSL +certificates with [Let's Encrypt](https://letsencrypt.org/). + +Still in the same container, let's first define some environment variables to +make the subsequent commands easier to run: + +``` +CONTROLLER_IMAGE=ingress-nginx/controller +CONTROLLER_TAG=v1.2.1 +DEFAULTBACKEND_IMAGE=defaultbackend-amd64 +DEFAULTBACKEND_TAG=1.5 +PATCH_IMAGE=ingress-nginx/kube-webhook-certgen +PATCH_TAG=v1.1.1 +``` + +### Set up `ingress-nginx` + +The first Kubernetes package to install in the cluster is `ingress-nginx` which +provides a way to handle incoming connections as a Kubernetes ingress. Here +are some samples commands to do this: + +``` +$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx +$ helm repo update +Hang tight while we grab the latest from your chart repositories... +...Successfully got an update from the "ingress-nginx" chart repository +Update Complete. ⎈Happy Helming!⎈ +$ helm install ingress-nginx ingress-nginx/ingress-nginx \ + --version=4.1.3 \ + --namespace=kernelci-api \ + --create-namespace \ + --set controller.replicaCount=2 \ + --set controller.nodeSelector."kubernetes\.io/os"=linux \ + --set controller.image.image=$CONTROLLER_IMAGE \ + --set controller.image.tag=$CONTROLLER_TAG \ + --set controller.image.digest="" \ + --set controller.admissionWebhooks.patch.nodeSelector."kubernetes\.io/os"=linux \ + --set controller.service.loadBalancerIP=10.224.0.42 \ + --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-internal"=true \ + --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz \ + --set controller.admissionWebhooks.patch.image.image=$PATCH_IMAGE \ + --set controller.admissionWebhooks.patch.image.tag=$PATCH_TAG \ + --set controller.admissionWebhooks.patch.image.digest="" \ + --set defaultBackend.nodeSelector."kubernetes\.io/os"=linux \ + --set defaultBackend.image.image=$DEFAULTBACKEND_IMAGE \ + --set defaultBackend.image.tag=$DEFAULTBACKEND_TAG \ + --set defaultBackend.image.digest="" +``` + +You should get a summary of the operation with some confirmation like this as +part of it: + +``` +The ingress-nginx controller has been installed. +It may take a few minutes for the LoadBalancer IP to be available. +You can watch the status by running 'kubectl --namespace default get services -o wide -w ingress-nginx-controller' +``` + +### Add a DNS label + +Then as SSL certificates are bound to particular domain names, you either have +to provide your own and update some independent DNS record with the public IP +address of the ingress or rely on the default Azure domain names. In this +example, we'll go down the standard Azure route to keep things self-contained. +This can be done by creating a label for the ingress: + +``` +$ helm upgrade ingress-nginx ingress-nginx/ingress-nginx \ + --namespace kernelci-api \ + --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"=kernelci-api \ + --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz +``` + +The cluster used in this example is located in the East US region, so the full +domain name (FQDN) is `kernelci-api.eastus.cloudapp.azure.com`. To confirm: + +``` +$ nslookup kernelci-api.eastus.cloudapp.azure.com +Server: 192.168.1.254 +Address: 192.168.1.254#53 + +Non-authoritative answer: +Name: kernelci-api.eastus.cloudapp.azure.com +Address: 4.157.88.45 +``` + +> **Note:** The IP address is assigned dynamically every time the ingress is +> recreated so it's normal to get a different one. + +### Set up `cert-manager` + +Once `ingress-nginx` has been set up, we can install `cert-manager` which will +be providing a Let's Encrypt SSL certificate to handle HTTPS incoming +connections. To do this: + +``` +$ helm repo add jetstack https://charts.jetstack.io +$ helm repo update +$ CERT_MANAGER_TAG=v1.8.0 +$ helm install cert-manager jetstack/cert-manager \ + --version $CERT_MANAGER_TAG \ + --namespace kernelci-api \ + --set controller.nodeSelector."kubernetes\.io/os"=linux \ + --set installCRDs=true \ + --set image.tag=$CERT_MANAGER_TAG \ + --set webhook.image.tag=$CERT_MANAGER_TAG \ + --set cainjector.image.tag=$CERT_MANAGER_TAG +``` + +Then you should see this as part of the output: + +``` +cert-manager v1.8.0 has been deployed successfully! +``` + +## Set up the ingress + +Once the Kubernetes packages have been installed, the actuall ingress +definition can be applied. Still within the same container, run this command: + +``` +$ kubectl apply -f ingress.yaml +``` + +Then to check it's all set up correctly: + +``` +$ kubectl --namespace=kernelci-api get certificates +NAME READY SECRET AGE +tls-secret True tls-secret 79s +$ kubectl --namespace=kernelci-api get ingresses +NAME CLASS HOSTS ADDRESS PORTS AGE +api nginx kernelci-api.eastus.cloudapp.azure.com 4.157.88.45 80, 443 33s +``` + +> **Note:** It can take a few minutes for the `ADDRESS` to appear and for the +> certificate to be ready. Initially the `READY` status will be `False` but +> this is expected. + +## Deploy the redis service + +There are two services to deploy in AKS: `redis` and `api`. First, let's look +at `redis` since it's simpler and is a dependency for the `api` service. To +deploy it: + +``` +$ kubectl apply -f redis.yaml +``` + +Then to check it's working: + +``` +$ kubectl --namespace=kernelci-api get deployments +NAME READY UP-TO-DATE AVAILABLE AGE +cert-manager 1/1 1 1 7h53m +cert-manager-cainjector 1/1 1 1 7h53m +cert-manager-webhook 1/1 1 1 7h53m +ingress-nginx-controller 1/1 1 1 8h +redis 1/1 1 1 8s +$ kubectl --namespace=kernelci-api get pods +NAME READY STATUS RESTARTS AGE +cert-manager-cainjector-f856fb49-hj2rw 1/1 Running 0 7h53m +cert-manager-f89f7fd88-ndsvn 1/1 Running 0 7h53m +cert-manager-webhook-665cd46d4b-ddq5r 1/1 Running 0 7h53m +ingress-nginx-controller-5ff6bb675f-c2lsx 1/1 Running 0 7h39m +redis-799cd8b7bd-ws2fs 1/1 Running 0 22s +$ kubectl logs redis-799cd8b7bd-ws2fs --namespace=kernelci-api +1:C 23 Aug 2023 14:19:20.774 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +1:C 23 Aug 2023 14:19:20.774 # Redis version=6.2.13, bits=64, commit=00000000, modified=0, pid=1, just started +1:C 23 Aug 2023 14:19:20.774 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf +1:M 23 Aug 2023 14:19:20.774 * monotonic clock: POSIX clock_gettime +1:M 23 Aug 2023 14:19:20.775 * Running mode=standalone, port=6379. +1:M 23 Aug 2023 14:19:20.775 # Server initialized +1:M 23 Aug 2023 14:19:20.775 * Ready to accept connections +``` + +## Deploy the main API service + +### Create secret key + +The API service requires a unique "secret key" used when generating JWT tokens. +To create your own key using the `openssl` tool and then add it as a Kubernetes +secret: + +``` +$ openssl rand -hex 32 +f9256e80e30c16187d3e42fa1ea70c41f6945743846cf1fb2911e0665f16dd99 +$ kubectl create secret generic \ + kernelci-api-secret-key \ + --namespace=kernelci-api \ + --from-literal=secret-key=f9256e80e30c16187d3e42fa1ea70c41f6945743846cf1fb2911e0665f16dd99 +secret/kernelci-api-secret-key created +``` + +This will then be shared with the application via the `SECRET_KEY` environment +variable as defined in `api.yaml`. + +### Add MongoDB service string + +While deployments with `docker-compose` typically rely on a local container +running MongoDB directly, when deploying in the Cloud you might want to use a +separate database. The kernelci.org Azure deployment relies on an Atlas +account for MongoDB and as such requires a service string which includes the +user name and password. This is why the MongoDB string is also defined as a +secret, although it might in some cases be an anonymous connection with no real +credentials. + +To define the Kubernetes secret for MongoDB: + +``` +$ kubectl create secret generic \ + kernelci-api-mongo-service \ + --namespace=kernelci-api \ + --from-literal=mongo-service="mongodb+srv://user:password@something.mongodb.net" +secret/kernelci-api-mongo-service created +``` + +This will then be shared with the application via the `MONGO_SERVICE` +environment variable as defined in `api.yaml`. + +### Deployment + +Let's first double-check that the secrets have been created as expected: + +``` +$ kubectl get secrets --namespace=kernelci-api | grep kernelci-api +kernelci-api-mongo-service Opaque 1 22h +kernelci-api-secret-key Opaque 1 22h +``` + +The main API service is based on the `kernelci/api` Docker image which is built +from the `Dockerfile.production` file included in this repository and hosted on +the Docker hub. As with any YAML file used by `kubectl`, it's possible to +override the Docker image name with an overlay on top of `api.yaml` if you want +to use your own instead. + +To deploy the API service: + +``` +$ kubectl apply -f api.yaml +``` + +Then if all went well, you should now be able query the API: + +``` +$ curl https://kernelci-api.eastus.cloudapp.azure.com/latest/ +{"message":"KernelCI API"} +``` + +The interactive API documentation should also be available on +[https://kernelci-api.eastus.cloudapp.azure.com/latest/docs](https://kernelci-api.eastus.cloudapp.azure.com/latest/docs) + +## Enjoy! + +Now you have a publicly available API instance suitable for production. To +make best use of it, you can try the `kci` command line tools to do things by +hand and run the `kernelci-pipeline` services to automatically run jobs, send +email reports etc. See the main documentation for pointers with more details.