From 32c8ce15ef26b7e002946835aaf40435b7356c28 Mon Sep 17 00:00:00 2001 From: Sebastian Date: Wed, 27 Sep 2023 13:10:08 +0200 Subject: [PATCH] Deployed 1eb0419 with MkDocs version: 1.4.3 --- applications/custom/sharelatex/index.html | 4 ++-- applications/custom/wikijs/index.html | 4 ++-- create-cluster/index.html | 9 ++++----- index.html | 2 +- print_page/index.html | 11 +++++------ search/search_index.json | 2 +- sitemap.xml.gz | Bin 600 -> 600 bytes 7 files changed, 15 insertions(+), 17 deletions(-) diff --git a/applications/custom/sharelatex/index.html b/applications/custom/sharelatex/index.html index 26b395b..6566467 100644 --- a/applications/custom/sharelatex/index.html +++ b/applications/custom/sharelatex/index.html @@ -1610,8 +1610,8 @@

Overleaf / ShareLaTeX

Installation

First you need to create a namespace for ShareLaTeX.

diff --git a/applications/custom/wikijs/index.html b/applications/custom/wikijs/index.html index 0665d70..9965ae7 100644 --- a/applications/custom/wikijs/index.html +++ b/applications/custom/wikijs/index.html @@ -1682,7 +1682,7 @@

WikiJSRequirements

Note

-

First we need a Postgres database. Please refer to Postgres.

+

First we need a Postgres database. Please refer to Postgres.

Install

@@ -1693,7 +1693,7 @@

Namespace
kubectl create ns wiki-ns

Config

-

Edit the database config/config.yml based on the Postgres config.

+

Edit the database config/config.yml based on the Postgres config.

  • db.host: <postgres-service-name>.<namespace>.svc.cluster.local
  • db.port: 5432
    • diff --git a/create-cluster/index.html b/create-cluster/index.html index 9f0e7ae..b3632c1 100644 --- a/create-cluster/index.html +++ b/create-cluster/index.html @@ -1579,14 +1579,13 @@

    Create a clusterChoose a cluster type

    You can choose between the following cluster types:

    Configure Traefik

    After creating the cluster we need to configure Traefik.

    -

    Configure Traefik -1

    +

    Configure Traefik

    diff --git a/index.html b/index.html index 20ab495..8e7d432 100644 --- a/index.html +++ b/index.html @@ -1615,7 +1615,7 @@

    IT-Lab Kubernetes CollectionNavigation

    The Top-Bar Navigation provides a "guided" tour through the documentation. It is recommended to follow the order of the navigation.

    -

    So lets start to create a cluster.

    +

    So lets start to create a cluster.

    diff --git a/print_page/index.html b/print_page/index.html index bb3a146..a0eae7c 100644 --- a/print_page/index.html +++ b/print_page/index.html @@ -1769,8 +1769,7 @@

    Choose a cluster typeConfigure Traefik

    After creating the cluster we need to configure Traefik.

    -

    Configure Traefik -1

    Kind

    +

    Configure Traefik

    Kind

    With Kind we can create a local Kubernetes cluster. This is useful for testing and development.

    The files are in the kind folder.

    kind create cluster --name example-cluster --config kind-config.yaml
@@ -2491,8 +2490,8 @@ 
                         
    Overleaf / ShareLaTeX
    
 
      
-
    • Create an MongoDB Replica Set by following the instructions in the MongoDB Replica Set section.
      • 
-
    • Create an Redis Standalone by following the instructions in the Redis Standalone section. (We need to create a Redis Standalone because the Redis Cluster is not supported by ShareLaTeX)
      • 
+
    • Create an MongoDB Replica Set by following the instructions in the MongoDB Replica Set section.
      • 
+
    • Create an Redis Standalone by following the instructions in the Redis Standalone section. (We need to create a Redis Standalone because the Redis Cluster is not supported by ShareLaTeX)
      • 
 
    
 
    Installation
    
 
    First you need to create a namespace for ShareLaTeX.
    
@@ -2571,7 +2570,7 @@ 
    InstallationRequirements
    
 
    
 
    Note
    
-
    First we need a Postgres database. Please refer to Postgres.
    
+
    First we need a Postgres database. Please refer to Postgres.
    
 
    
 
    Install
    
 
    
@@ -2582,7 +2581,7 @@ 
    Namespace
    kubectl create ns wiki-ns

    Config

    -

    Edit the database config/config.yml based on the Postgres config.

    +

    Edit the database config/config.yml based on the Postgres config.

    • db.host: <postgres-service-name>.<namespace>.svc.cluster.local
    • db.port: 5432
      • diff --git a/search/search_index.json b/search/search_index.json index 9fe7d91..378c519 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"IT-Lab Kubernetes Collection","text":"

      Welcome to the IT-Lab Kubernetes Collection. This is a collection of Kubernetes applications and tools. It also contains the Basics to create a fully functional Kubernetes cluster.

      "},{"location":"#navigation","title":"Navigation","text":"

      The Top-Bar Navigation provides a \"guided\" tour through the documentation. It is recommended to follow the order of the navigation.

      So lets start to create a cluster.

      "},{"location":"applications/","title":"Application","text":"

      We will deploy the following applications:

      • Monitoring with Prometheus and Grafana
      • Some Databases
      • Custom Apllications
      "},{"location":"applications/docker-registry/","title":"Docker Registry","text":"
      echo \"192.168.178.104 registry registry.k3s.test\" >> /etc/hosts\n
      ctr image pull --platform linux/amd64 docker.io/rancher/coredns-coredns:1.6.3\nctr content fetch --platform linux/amd64 docker.io/rancher/coredns-coredns:1.6.3\nctr image tag docker.io/rancher/coredns-coredns:1.6.3 registry.k3s.test:5000/test\nctr image push --platform linux/amd64 --plain-http registry.k3s.test:5000/test\ncurl http://registry.k3s.test:5000/v2/_catalog\n    => {\"repositories\":[\"test\"]}\n
      "},{"location":"applications/keda/","title":"KEDA","text":"

      KEDA is a Kubernetes-based Event Driven Autoscaler. With KEDA, you can drive the scaling of any container in Kubernetes based on the number of events needing to be processed.

      "},{"location":"applications/keda/#install-in-cluster","title":"Install in Cluster","text":"
      helm repo add kedacore https://kedacore.github.io/charts\nhelm repo update\nhelm install keda kedacore/keda --namespace keda --create-namespace\n
      "},{"location":"applications/keda/#scalers","title":"Scalers","text":"

      KEDA Scalers

      "},{"location":"applications/keda/#prometheus","title":"Prometheus","text":"
      apiVersion: keda.sh/v1alpha1\nkind: ScaledObject\nmetadata:\n  name: prometheus-scaledobject\n  namespace: whoami-ns\nspec:\n  scaleTargetRef:\n    apiVersion: apps/v1\n    kind: Deployment\n    name: instrumented\n  triggers:\n    - type: prometheus\n      metadata:\n        serverAddress: http://prometheus.monitoring.svc.cluster.local:9090\n        metricName: http_request_total\n        query: http_requests_total{code=\"200\",handler=\"prometheus\",method=\"get\", service=\"instrumented-service\"}\nthreshold: '50'\nidleReplicaCount: 0\nminReplicaCount: 3\nmaxReplicaCount: 10\n
      "},{"location":"applications/keda/#cpu","title":"CPU","text":"
      apiVersion: keda.sh/v1alpha1\nkind: ScaledObject\nmetadata:\n  name: prometheus-scaledobject\n  namespace: whoami-ns\nspec:\n  scaleTargetRef:\n    apiVersion: apps/v1\n    kind: Deployment\n    name: instrumented\n  triggers:\n    - type: cpu\n      metricType: Utilization # Allowed types are 'Utilization' or 'AverageValue'\nmetadata:\n        value: '180'\nidleReplicaCount: 0\nminReplicaCount: 3\nmaxReplicaCount: 10\n
      "},{"location":"applications/openfaas/","title":"OpenFaaS","text":"

      OpenFaaS is a framework for building serverless functions with Docker and Kubernetes which has first class support for metrics. Any process can be packaged as a function enabling you to consume a range of web events without repetitive boiler-plate coding.

      "},{"location":"applications/openfaas/#deploy","title":"Deploy","text":"

      Use make install in the OpenFaaS folder to install in the Cluster.

      "},{"location":"applications/openfaas/#get-password","title":"Get Password","text":"
      PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath=\"{.data.basic-auth-password}\" | base64 --decode; echo)\necho \"OpenFaaS admin password: ${PASSWORD}\"\n

      Add the password to the ~/.bashrc (or ~/.zshrc) file:

      export OPENFAAS_URL=http://openfaas.k3s.test:8080\n

      Reload the shell. Login to OpenFaaS:

      echo -n $PASSWORD | faas-cli login --username admin --password-stdin\n
      "},{"location":"applications/openfaas/#openfaas-cli","title":"OpenFaaS-Cli","text":"
      # Mac\nbrew install faas-cli\n

      First Function

      "},{"location":"applications/custom/chest-system/","title":"Chest-System","text":"

      The Chest-System is a system developed by IT-Lab to store and manage items in chests.

      "},{"location":"applications/custom/chest-system/#installation","title":"Installation","text":"

      Refers to the chest-system folder

      "},{"location":"applications/custom/chest-system/#prerequisites","title":"Prerequisites","text":"

      Note

      First we need a Postgres database. Please refer to Postgres.

      "},{"location":"applications/custom/chest-system/#config","title":"Config","text":"

      Edit the database credentials in the server-deployment.yaml file:

      - name: POSTGRES_DB\nvalue: chest\n- name: POSTGRES_HOST\nvalue: chest-cluster # We can use the service name without namespace because we are in the same namespace\n- name: POSTGRES_PASSWORD\nvalueFrom:\nsecretKeyRef:\nkey: password\nname: chestuser.chest-cluster.credentials.postgresql.acid.zalan.do # The secret name is composed of: <username>.<cluster-name>.credentials.postgresql.acid.zalan.do\n- name: POSTGRES_USER\nvalue: chestuser\n
      "},{"location":"applications/custom/chest-system/#deploy","title":"Deploy","text":"
      kubectl apply -f .\n
      "},{"location":"applications/custom/sharelatex/","title":"Overleaf / ShareLaTeX","text":"
      • Create an MongoDB Replica Set by following the instructions in the MongoDB Replica Set section.
      • Create an Redis Standalone by following the instructions in the Redis Standalone section. (We need to create a Redis Standalone because the Redis Cluster is not supported by ShareLaTeX)
      ","tags":["kubernetes","sharelatex","latex","overleaf"]},{"location":"applications/custom/sharelatex/#installation","title":"Installation","text":"

      First you need to create a namespace for ShareLaTeX.

      kubectl create namespace sharelatex-ns\n

      Then you need to adjust the environment variables in the sharelatex-deployment.yaml file.

      Set the following environment variables:

      Variable Description SHARELATEX_MONGO_URL The MongoDB connection string SHARELATEX_REDIS_HOST The Redis host SHARELATEX_EMAIL_FROM_ADDRESS The email address from which the emails are sent SHARELATEX_EMAIL_SMTP_HOST The SMTP host SHARELATEX_EMAIL_SMTP_PORT The SMTP port SHARELATEX_EMAIL_SMTP_SECURE The SMTP secure flag SHARELATEX_EMAIL_SMTP_USER The SMTP user SHARELATEX_EMAIL_SMTP_PASS The SMTP password SHARELATEX_APP_NAME The name of the ShareLaTeX instance SHARELATEX_SITE_URL The URL of the ShareLaTeX instance SHARELATEX_ADMIN_EMAIL The email address of the admin SHARELATEX_BEHIND_PROXY The behind proxy flag SHARELATEX_SECURE_COOKIE The secure cookie flag

      Also adjust the host in the sharelatex-ingress.yaml file.

      Then you can deploy ShareLaTeX.

      kubectl apply -f .\n
      ","tags":["kubernetes","sharelatex","latex","overleaf"]},{"location":"applications/custom/wikijs/","title":"WikiJS","text":"

      WikiJS is a modern, lightweight and powerful wiki app built on NodeJS, Git and Markdown.

      Docs

      "},{"location":"applications/custom/wikijs/#requirements","title":"Requirements","text":"

      Note

      First we need a Postgres database. Please refer to Postgres.

      "},{"location":"applications/custom/wikijs/#install","title":"Install","text":"

      You can simply use the make command in the Wiki folder to make the config and apply the files

      "},{"location":"applications/custom/wikijs/#namespace","title":"Namespace","text":"

      Create the namespace for the application:

      kubectl create ns wiki-ns\n
      "},{"location":"applications/custom/wikijs/#config","title":"Config","text":"

      Edit the database config/config.yml based on the Postgres config.

      • db.host: <postgres-service-name>.<namespace>.svc.cluster.local
      • db.port: 5432
      • db.user: The user you defined for the database
      • db.pass: The password for the user from the secret.

      Create the configmap from the file:

      kubectl create configmap config-configmap --from-file=config/config.yml -n wiki-ns\n
      "},{"location":"applications/custom/wikijs/#deployment","title":"Deployment","text":"

      Deploy the application:

      kubectl apply -f wiki-deployment.yaml,wiki-service.yaml,wiki-ingress.yaml -n wiki-ns\n

      Access the application: https://wiki.k3s.test

      "},{"location":"applications/databases/mongodb/","title":"MongoDB","text":"

      Docs

      We will use the MongoDB Operator to manage our MongoDB instances. The MongoDB Operator is a Kubernetes Operator that manages the MongoDB instances.

      In this example we will create a MongoDB Replica Set. We will use it for our ShareLaTeX instance.

      "},{"location":"applications/databases/mongodb/#installation","title":"Installation","text":"
      helm repo add mongodb-helm-charts https://mongodb.github.io/helm-charts\n# Install the MongoDB Operator\nhelm install my-community-operator mongodb-helm-charts/community-operator --namespace mongodb --create-namespace --set operator.watchNamespace=\"*\"\n# Deploy a MongoDB Replica Set\nkubectl apply -f replicaset.yaml -n mongodb\n# Check\nkubectl get mdbc -n mongodb\n
      "},{"location":"applications/databases/mongodb/#connection-string","title":"Connection String","text":"

      Now you can get the connection string from the secret in the namespace mongodb with the pattern <mongodb-resource-name>-<database>-<username>. In our case it is example-mongodb-admin-my-user.

      kubectl get secret example-mongodb-admin-my-user -n mongodb -o json | jq -r '.data | with_entries(.value |= @base64d)'\n

      You can now use the standardSrv connection string to connect to the MongoDB Replica Set. We will use it in our ShareLaTeX instance.

      "},{"location":"applications/databases/mongodb/#test","title":"Test","text":"

      Connect with the standardSrv connection string to the MongoDB Replica Set.

      kubectl run mongopod -i -t --image=rtsp/mongosh -- bash\n> mongosh \"<connection-string>\"\n
      "},{"location":"applications/databases/postgresql/","title":"Postgres","text":"

      We will use the Postgres Operator to deploy a Postgres Cluster.

      "},{"location":"applications/databases/postgresql/#deploy-postgres-operator","title":"Deploy Postgres Operator","text":"
      # add repo for postgres-operator\nhelm repo add postgres-operator-charts https://opensource.zalando.com/postgres-operator/charts/postgres-operator\n\n# install the postgres-operator\nhelm install postgres-operator postgres-operator-charts/postgres-operator\n

      Check if running

      kubectl get pod -l app.kubernetes.io/name=postgres-operator\n
      "},{"location":"applications/databases/postgresql/#deploy-operator-ui","title":"Deploy Operator UI","text":"
      # add repo for postgres-operator-ui\nhelm repo add postgres-operator-ui-charts https://opensource.zalando.com/postgres-operator/charts/postgres-operator-ui\n\n# install the postgres-operator-ui\nhelm install postgres-operator-ui postgres-operator-ui-charts/postgres-operator-ui\n

      Check if the UI is running

      kubectl get pod -l app.kubernetes.io/name=postgres-operator-ui\n

      http://127.0.0.1:8081

      "},{"location":"applications/databases/postgresql/#create-a-postgres-cluster","title":"Create a Postgres Cluster","text":"
      kubectl create -f acid-cluster.yaml\n

      Check if the cluster is running

      # check the deployed cluster\nkubectl get postgresql\n\n# check created database pods\nkubectl get pods -l application=spilo -L spilo-role\n\n# check created service resources\nkubectl get svc -l application=spilo -L spilo-role\n
      "},{"location":"applications/databases/postgresql/#password","title":"Password","text":"

      As you can see in the acid-cluster.yaml there are multiple users and multiple databases.

      users:\nzalando:  # database owner\n- superuser\n- createdb\nfoo_user: []  # role for application foo\ndatabases:\nfoo: zalando  # dbname: owner\npreparedDatabases:\nbar: {}\n

      The passwords are stored in secrets. The password for the postgres user is stored in a secret. We can get the password with:

      export PGPASSWORD=$(kubectl get secret postgres.acid-minimal-cluster.credentials.postgresql.acid.zalan.do -o 'jsonpath={.data.password}' | base64 -d)\necho $PGPASSWORD\n\n# Use encrypted connections\nexport PGSSLMODE=require\n

      The secret name is composed of: <username>.<cluster-name>.credentials.postgresql.acid.zalan.do

      "},{"location":"applications/databases/postgresql/#for-wikijs","title":"For WikiJS","text":"
      kubectl create -f wiki-cluster.yaml\n

      Get the password for the wikiuser user:

      export PGPASSWORD=$(kubectl get secret wikiuser.wiki-cluster.credentials.postgresql.acid.zalan.do -o 'jsonpath={.data.password}' | base64 -d)\necho $PGPASSWORD\n

      The host is: wiki-cluster.default.svc.cluster.local

      "},{"location":"applications/databases/redis/","title":"Redis","text":"

      Docs

      First we need the Redis Operator, we install it in the ot-operators namespace. The Redis Operator is a Kubernetes Operator that manages the Redis instances.

      "},{"location":"applications/databases/redis/#redis-operator-required","title":"Redis Operator (Required)","text":"
      helm install redis-operator ot-helm/redis-operator --namespace ot-operators --create-namespace\n
      "},{"location":"applications/databases/redis/#redis-instance","title":"Redis Instance","text":""},{"location":"applications/databases/redis/#standalone","title":"Standalone","text":"

      We can create a Redis Standalone instance by using the following command: We will use it e.g. for ShareLaTeX.

      helm install redis ot-helm/redis --namespace ot-operators\n

      The FQDN to access the redis instance is <name>.<namespace>.svc.cluster.local.

      So in our case it is redis.ot-operators.svc.cluster.local.

      "},{"location":"applications/monitoring/","title":"Monitoring via Prometheus and Grafana","text":"

      The monitoring stack consists of Prometheus and Grafana. It uses a Prometheus Operator to create the Prometheus and Alertmanager instances.

      "},{"location":"applications/monitoring/alertmanager/","title":"Alertmanager","text":""},{"location":"applications/monitoring/grafana/","title":"Grafana","text":"

      Deploy Grafana

      kubectl apply -f grafana/\n

      Login with admin and admin

      Go to Settings > Data Sources and add the Prometheus Data Source with the following URL: http://prometheus:9090

      Import the dashboard with the ID 8171 from the Grafana Dashboard Store.

      "},{"location":"applications/monitoring/prometheus-operator/","title":"Prometheus Operator","text":"

      The Prometheus Operator for Kubernetes provides easy monitoring definitions for Kubernetes services and deployment and management of Prometheus instances.

      This is a helper that extends the Kubernetes API and allows you to manage Prometheus servers and configurations using Kubernetes native tools and workflows.

      "},{"location":"applications/monitoring/prometheus-operator/#create-the-namepsace","title":"Create the namepsace","text":"
      kubectl create namespace monitoring\n
      "},{"location":"applications/monitoring/prometheus-operator/#install","title":"Install","text":"

      Use the bundle.yaml in the Prometheus-Operator folder to install the Operator.

      kubectl apply --server-side -f bundle.yaml\n
      "},{"location":"applications/monitoring/prometheus-operator/#check-the-deployment","title":"Check the deployment","text":"
      kubectl get pods -n monitoring\n
      "},{"location":"applications/monitoring/prometheus/","title":"Prometheus","text":"

      In the monitoring folder

      First wie create the config for the additional scrape jobs. The output is written to prometheus/additional-scrape-configs.yaml.

      kubectl create secret generic additional-scrape-configs -n monitoring --from-file=prometheus/additional-job/prometheus-additional-job.yaml --dry-run=client -oyaml > prometheus/additional-scrape-configs.yaml\n

      Then we create the Prometheus Instance with the following command:

      kubectl apply -f prometheus/\n

      You can also use the make apply-config command in the monitoring folder

      "},{"location":"applications/monitoring/service-monitor/","title":"Service Monitor","text":"

      The Files are in seperate folders for each application.

      "},{"location":"applications/monitoring/service-monitor/#longhorn","title":"Longhorn","text":"

      Apply the Longhorn Service Monitor

      kubectl apply -f longhorn/\n
      "},{"location":"applications/monitoring/service-monitor/#check-the-deamonset","title":"Check the Deamonset","text":"
      \u279c  kubectl get daemonset -n longhorn-system\n\nNAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE\nlonghorn-manager 2 2 2 2 2 <none> 2d19h\n
      "},{"location":"applications/monitoring/service-monitor/#kube-state-metrics","title":"Kube State Metrics","text":"

      Kube State Metrics (KSM) is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects.

      Apply the Kube State Metrics Service Monitor

      kubectl apply -f kube-state-metrics/\n
      "},{"location":"applications/monitoring/service-monitor/#node-exporter","title":"Node-Exporter","text":"

      The Prometheus Node Exporter exposes a wide variety of hardware- and kernel-related metrics.

      Apply the Node-Exporter Service Monitor

      kubectl apply -f node-exporter/\n
      "},{"location":"applications/monitoring/service-monitor/#kubelet","title":"Kubelet","text":"

      The Kubelet is the primary \u201cnode agent\u201d that runs on each node. It can register the node with the apiserver using one of: the hostname; a flag to override the hostname; or specific logic for a cloud provider.

      Apply the Kubelet Service Monitor

      kubectl apply -f kubelet/\n
      "},{"location":"applications/monitoring/service-monitor/#traefik","title":"Traefik","text":"

      Traefik is an open-source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them.

      We will also use it to expose the Prometheus and Alertmanager UIs.

      Apply the Traefik Service Monitor

      kubectl apply -f traefik/\n
      "},{"location":"applications/monitoring/service-monitor/#check-the-service-monitors","title":"Check the Service Monitors","text":"
      \u279c  Monitoring git:(main) \u2717 kubectl get ServiceMonitor -n monitoring\nNAME                                 AGE\nlonghorn-prometheus-servicemonitor   39h\nnode-exporter                        39h\nkube-state-metrics                   39h\nkubelet                              39h\ntraefik                              38h\n
      "},{"location":"create-cluster/","title":"Create a cluster","text":"

      In this section we will show you how to create a Kubernetes cluster and configure Traefik.

      "},{"location":"create-cluster/#choose-a-cluster-type","title":"Choose a cluster type","text":"

      You can choose between the following cluster types:

      • Kind
      • K3s
      • K8s
      "},{"location":"create-cluster/#configure-traefik","title":"Configure Traefik","text":"

      After creating the cluster we need to configure Traefik.

      Configure Traefik 1

      "},{"location":"create-cluster/k3s/","title":"K3s","text":"

      Siehe K3s Docs

      Inspired by Rpi4Cluster.com

      "},{"location":"create-cluster/k3s/#master","title":"Master","text":"
      curl -sfL https://get.k3s.io | sh -\n

      Use the kubeconfig from /etc/rancher/k3s/k3s.yaml to connect to the cluster.

      "},{"location":"create-cluster/k3s/#nodes","title":"Nodes","text":"

      Get token from /var/lib/rancher/k3s/server/node-token

      curl -sfL https://get.k3s.io | K3S_URL=https://172.30.62.166:6443 K3S_TOKEN=<TOKEN>  sh -\n
      "},{"location":"create-cluster/k3s/#label-the-nodes","title":"Label the Nodes","text":"
      kubectl label nodes <node-name> node-role.kubernetes.io/worker=worker node-type=worker\n
      "},{"location":"create-cluster/k8s/","title":"Install K8s","text":"

      Install packages

      apt update\napt install apt-transport-https ca-certificates curl gnupg2 software-properties-common -y\n

      Kernel modules for k8s networking

      modprobe overlay\nmodprobe br_netfilter\ncat <<EOF | tee /etc/modules-load.d/k8s.conf\noverlay\nbr_netfilter\nEOF\n

      Natting

      cat <<EOF | tee /etc/sysctl.d/99-kubernetes-cri.conf\nnet.bridge.bridge-nf-call-iptables  = 1\nnet.ipv4.ip_forward                 = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nEOF\nsysctl --system\n

      Add containerd keys and repos

      curl https://download.docker.com/linux/debian/gpg | apt-key add\necho \"deb [arch=amd64] https://download.docker.com/linux/debian buster stable\" > /etc/apt/sources.list.d/docker.list\n

      Install containerd

      apt update\napt install containerd.io -y\n

      Configure containerd

      containerd config default > /etc/containerd/config.toml\n# set SystemdCgroup to true\nsudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml\n

      Disable swap!

      sudo sed -i '/ swap / s/^\\(.*\\)$/#\\1/g' /etc/fstab\nswapoff -a\n

      Restart containerd

      systemctl enable containerd\nsystemctl restart containerd\n

      Add k8s repos and install it

      curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add\necho \"deb https://apt.kubernetes.io/ kubernetes-xenial main\" > /etc/apt/sources.list.d/kubernetes.list\n# install k8s and hold packags\napt update\napt install kubelet kubeadm kubectl -y\napt-mark hold kubelet kubeadm kubectl\n

      Switch iptables to legacy

      update-alternatives --config iptables\n# --> select iptables-legacy\nreboot\n
      "},{"location":"create-cluster/k8s/#cluster-erstellen","title":"Cluster erstellen:","text":"

      POD-CIDR -> internal used to give every pod a unique ipv4 CRI Socket ->path to containerd/container runtime socket Control-Plane-Endpoint: for single Control Plane: IP-OF-MASTER:6443, else IP:Port to use a Loadbalancer

      kubeadm init --pod-network-cidr 10.244.0.0/16 --cri-socket /run/containerd/containerd.sock --control-plane-endpoint=<MASTER_IP>:6443\n

      Copy kubectl config to home folder

      mkdir -p $HOME/.kube\nudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\nsudo chown $(id -u):$(id -g) $HOME/.kube/config\n

      Setup Canal Network

      mkdir ~/kube/config -p\ncd ~/kube/config\ncurl https://raw.githubusercontent.com/projectcalico/calico/v3.26.0/manifests/canal.yaml -O\nkubectl apply -f canal.yaml\n
      "},{"location":"create-cluster/kind/","title":"Kind","text":"

      With Kind we can create a local Kubernetes cluster. This is useful for testing and development.

      The files are in the kind folder.

      kind create cluster --name example-cluster --config kind-config.yaml\n
      "},{"location":"create-cluster/kind/#point-kubectl-to-the-local-cluster","title":"Point kubectl to the local cluster","text":"

      The context is kind- followed by the cluster name.

      kubectl cluster-info --context kind-example-cluster\n
      "},{"location":"create-cluster/kind/#get-the-config","title":"Get the config","text":"
      kubectl config view --minify --flatten --context kind-example-cluster\n
      "},{"location":"create-cluster/traefik/","title":"Networking - Traefik","text":"

      Simply use kubectl apply -f . in the Traefik folder to apply all files.

      We configure traefike to do the following:

      • Expose the dashboard on port 9000
      • Middleware to redirect http to https
      • Middleware for basic auth
      • Middleware for cors

      Refernce

      "},{"location":"create-cluster/traefik/#expose-the-dashboard","title":"Expose the Dashboard","text":"

      traefik.yaml

      apiVersion: helm.cattle.io/v1\nkind: HelmChartConfig\nmetadata:\nname: traefik\nnamespace: kube-system\nspec:\nvaluesContent: |-\nadditionalArguments:\n- \"--api\"\n- \"--api.dashboard=true\"\n- \"--api.insecure=true\"\n- \"--log.level=DEBUG\"\nports:\ntraefik:\nexpose: true\nproviders:\nkubernetesCRD:\nallowCrossNamespace: true\n

      kubectl apply -f traefik.yaml

      Expose Services

      http://172.30.62.166:9000/dashboard/#/

      "},{"location":"create-cluster/traefik/#basic-auth","title":"Basic Auth","text":"

      First we need an Traefik Middleware. We define it in the default namespace, because we will use it in all namespaces. The credentials are admin:admin and are base64 encoded.

      You can encode your own credentials with:

      echo -n 'admin' | base64\n

      Create the Middleware:

      apiVersion: v1\nkind: Secret\nmetadata:\nname: my-auth\nnamespace: default\ntype: kubernetes.io/basic-auth\ndata:\nusername: YWRtaW4=\npassword: YWRtaW4=\n---\napiVersion: traefik.containo.us/v1alpha1\nkind: Middleware\nmetadata:\nname: my-auth-middleware\nnamespace: default\nspec:\nbasicAuth:\nremoveHeader: true\nsecret: my-auth\n
      "},{"location":"create-cluster/traefik/#use-the-middleware-in-the-ingress-annotation","title":"Use the Middleware in the Ingress annotation:","text":"

      <namespace>-<middleware-name>@kubernetescrd

      default-my-auth-middleware@kubernetescrd

      Use at the end of the list => Redirect to HTTPS before authentication

      apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\nname: web-ingress\nnamespace: linkerd-viz\nannotations:\nkubernetes.io/ingress.class: traefik\ntraefik.ingress.kubernetes.io/router.middlewares: default-cors@kubernetescrd,default-redirectscheme@kubernetescrd,default-my-auth-middleware@kubernetescrd\nspec:\nrules:\n- host: linkerd.k3s.test\nhttp:\npaths:\n- path: /\npathType: Prefix\nbackend:\nservice:\nname: web\nport:\nnumber: 8084\n
      "},{"location":"create-cluster/traefik/#cors","title":"CORS","text":"
      apiVersion: traefik.containo.us/v1alpha1\nkind: Middleware\nmetadata:\nname: cors\nnamespace: default\nspec:\nheaders:\naccessControlAllowOriginList:\n- 'https://hub.k3s.it-lab.cc'\naccessControlAllowMethods:\n- HEAD\n- GET\n
      "},{"location":"create-cluster/traefik/#redirect-http-to-https","title":"Redirect HTTP to HTTPS","text":"
      apiVersion: traefik.containo.us/v1alpha1\nkind: Middleware\nmetadata:\nname: redirectscheme\nnamespace: default\nspec:\nredirectScheme:\nscheme: https\npermanent: true\n
      "},{"location":"create-cluster/traefik/external-dns/","title":"External dns","text":""},{"location":"create-cluster/traefik/external-dns/#config","title":"Config","text":"

      <email> and domain-filer in Traefik/external-dns/values.yaml:

      env:\n- name: CF_API_KEY\nvalueFrom:\nsecretKeyRef:\nname: cloudflare-api-key\nkey: api-key\n- name: CF_API_EMAIL\nvalue: '<email>'\n
      extraArgs:\n- --domain-filter=<your-domain>\n

      Both <api-key> in Traefik/cf-api-secret.yaml:

      stringData:\napi-key: '<api-key>'\n

      <email> in Traefik/traefik-config.yaml:

      env:\n- name: CF_API_EMAIL\nvalue: '<email>'\n
      # Create the external-dns namespace\nkubectl create namespace external-dns\n# Create the Cloudflare API Key\nkubectl apply -f cf-api-secret.yaml\n# Apply the config for traefik from TLS\nkubectl apply -f traefik-config.yaml\n\n\n# Cert-Manager\nkubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.crds.yaml\nhelm repo add jetstack https://charts.jetstack.io\nhelm install cert-manager --namespace cert-manager --version v1.13.0 jetstack/cert-manager --create-namespace\n\n# External-DNS\nhelm repo add external-dns https://kubernetes-sigs.github.io/external-dns/\nhelm upgrade --install external-dns external-dns/external-dns --values external-dns/values.yaml --namespace external-dns\n

      annotations in your Ingress definitions:

      annotations:\ntraefik.ingress.kubernetes.io/router.entrypoints: websecure\ntraefik.ingress.kubernetes.io/router.tls: 'true'\ntraefik.ingress.kubernetes.io/router.tls.certresolver: myresolver\n

      Siehe:

      • https://kubernetes-sigs.github.io/external-dns/v0.12.2/tutorials/hostport/#kafka-stateful-set
      • proxied
      "},{"location":"create-cluster/traefik/tls/","title":"Traefik - TLS","text":"

      We will configure Traefik to use Letsencrypt DNS Challenges to automatically generate TLS certificates for our services.

      "},{"location":"create-cluster/traefik/tls/#prerequisites","title":"Prerequisites","text":"

      We need an account with a DNS provider that supports the ACME DNS Challenge and a domain name.

      We will use Cloudflare as our DNS provider.

      "},{"location":"create-cluster/traefik/tls/#cloudflare","title":"Cloudflare","text":"

      Get the Global API Key from your Cloudflare account. Get your \"Global API Key\"

      "},{"location":"create-cluster/traefik/tls/#traefik","title":"Traefik","text":"

      Now we need to configure Traefik to use the Cloudflare DNS Challenge. Refer to the traefik/traefik-config.yml file.

      Uncomment the additionalArguments for the resolver and replace the <email> with your email address.

      - '--certificatesresolvers.myresolver.acme.dnschallenge=true'\n- '--certificatesresolvers.myresolver.acme.dnschallenge.provider=cloudflare'\n- '--certificatesresolvers.myresolver.acme.email=<email>'\n- '--certificatesresolvers.myresolver.acme.storage=/data/acme.json'\n

      Then uncomment the env for the Cloudflare API Key and replace the <email> and <key> with your Email and Global API Key.

      env:\n- name: CF_API_EMAIL\nvalue: '<email>'\n- name: CF_API_KEY\nvalue: '<api_key>' #<== Unter https://dash.cloudflare.com/profile/api-tokens \"Global API Key\"\n
      "},{"location":"create-cluster/traefik/tls/#deploy-traefik","title":"Deploy Traefik","text":"

      Deploy Traefik with the new configuration.

      kubectl apply -f traefik-config.yaml\n
      "},{"location":"create-cluster/traefik/tls/#usage","title":"Usage","text":"

      Now we can use the myresolver in our Ingress definitions.

      annotations:\ntraefik.ingress.kubernetes.io/router.entrypoints: websecure\ntraefik.ingress.kubernetes.io/router.tls: 'true'\ntraefik.ingress.kubernetes.io/router.tls.certresolver: myresolver\n

      Please note that the host in the Ingress definition must match the domain name.

      "},{"location":"create-cluster/traefik/tls/#trouble-shooting","title":"Trouble Shooting","text":"

      Check the logs of the Traefik pod.

      kubectl logs deploy/traefik -n kube-system\n

      Pay attention to the name of the certresolver. The name in the traefik-config.yml must match the name in the Ingress definition.

      "},{"location":"random/coredns/","title":"CoreDNS","text":"

      CoreDNS always appends the domain 'it-lab.cc' to the hostname. This is not wanted in our case. Therefore we have to change the CoreDNS config.

      (Did not found a better solution)

      We can use the rewrite Plugin to remove the .it-lab.cc from the hostname.

      coredns-cm.yaml in dns folder

      rewrite name substring .com.it-lab.cc .com\n

      Apply the config by running make in the dns folder

      "},{"location":"storage/","title":"Storage","text":"

      This is a collection of storage solutions for k3s.

      NFS is used for the production environment. Longhorn is more for testing purposes.

      You can use the StorageClass local-path for testing purposes. It is not recommended for production use.

      "},{"location":"storage/longhorn/","title":"Longhorn","text":"

      Siehe Longhorn Docs Longhorn Examples

      "},{"location":"storage/longhorn/#requirements","title":"Requirements","text":"
      apt install jq nfs-common -y\n

      Check Script

      curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.5.1/scripts/environment_check.sh | bash\n
      "},{"location":"storage/longhorn/#install","title":"Install","text":"

      Use make in Longhorn folder

      helm repo add longhorn https://charts.longhorn.io\nhelm repo update\nhelm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --version 1.5.1\n

      Check deployment (5-10 Minutes), till all pods are up

      kubectl -n longhorn-system get pod\n

      Make the Longhorn accessable via Web Dashboard

      "},{"location":"storage/nfs/","title":"NFS Storage","text":"
      helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/\n\nhelm install nfs-subdir-external-provisioner --namespace nfs-subdir-external-provisioner --create-namespace nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --set nfs.server=172.30.41.5 --set nfs.path=/k3s-test-sj\n

      Set NFS as default

      # Mark local-path as non-default\n\nkubectl patch storageclass local-path -p '{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io/is-default-class\":\"false\"}}}'\n\n# Mark nfs-client as default\n\nkubectl patch storageclass nfs-client -p '{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io/is-default-class\":\"true\"}}}'\n\nkubectl get storageclass\n
      "},{"location":"tools/","title":"Tools","text":"

      In this section you will find a list of tools that can be used to help you with your development.

      It also contains a list of tools for Continuous Deployment.

      "},{"location":"tools/dashboard/","title":"Kubernetes-Dashboard","text":"

      I prefer to use the Lens Dashboard instead of the Kubernetes Dashboard.

      Use make in Dashboard folder

      Siehe Helm Hub

      Click for Installation 
      helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/\nhelm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard\n\nexport POD_NAME=$(kubectl get pods -n kubernetes-dashboard -l \"app.kubernetes.io/name=kubernetes-dashboard,app.kubernetes.io/instance=kubernetes-dashboard\" -o jsonpath=\"{.items[0].metadata.name}\")\nkubectl expose pod $POD_NAME --port=8443 --target-port=8443 --type=LoadBalancer -n kubernetes-dashboard\n
      `dashboard.admin-user.yml` 
      apiVersion: v1\nkind: ServiceAccount\nmetadata:\nname: admin-user\nnamespace: kubernetes-dashboard\n
      `dashboard.admin-user-role.yml` 
      apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\nname: admin-user\nroleRef:\napiGroup: rbac.authorization.k8s.io\nkind: ClusterRole\nname: cluster-admin\nsubjects:\n\n- kind: ServiceAccount\n  name: admin-user\n  namespace: kubernetes-dashboard\n
      Deploy the configs 
      sudo k3s kubectl create -f dashboard.admin-user.yml -f dashboard.admin-user-role.yml\n
      "},{"location":"tools/dashboard/#get-the-token","title":"Get the Token","text":"
      kubectl -n kubernetes-dashboard create token admin-user --duration=24h\n

      https://172.30.62.166:8443/#/login

      "},{"location":"tools/helm%2Barkade/","title":"Helm","text":"

      Siehe Helm Docs

      "},{"location":"tools/helm%2Barkade/#install","title":"Install","text":"
      curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null\nsudo apt-get install apt-transport-https --yes\necho \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main\" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list\nsudo apt-get update\nsudo apt-get install helm\n
      "},{"location":"tools/helm%2Barkade/#arkade","title":"Arkade","text":"
      # Linux\ncurl -sLS https://get.arkade.dev | sudo sh\n# Mac\nbrew install arkade\n\n# Check version\narkade version\n
      "},{"location":"tools/linkerd/","title":"Linkerd","text":"

      Linkerd is a service mesh for Kubernetes and other frameworks. It makes running services easier and safer by giving you runtime debugging, observability, reliability, and security\u2014all without requiring any changes to your code.

      "},{"location":"tools/linkerd/#cli","title":"CLI","text":"
      # Mac\nbrew install linkerd\n
      "},{"location":"tools/linkerd/#install-in-cluster","title":"Install in cluster","text":"
      linkerd check --pre\nlinkerd install --crds | kubectl apply -f -\nlinkerd install | kubectl apply -f -\nlinkerd check\n
      "},{"location":"tools/linkerd/#run-a-demo-app","title":"Run a Demo App","text":"
      curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/emojivoto.yml | kubectl apply -f -\n\n# Expose (for reference)\nkubectl -n emojivoto port-forward svc/web-svc 8080:80\n

      http://127.0.0.1:8080

      Try to vote for the donut emoji, you\u2019ll get a 404 page.

      Don\u2019t worry, these errors are intentional. (In a later guide, we\u2019ll show you how to use Linkerd to identify the problem.)

      Inject Linkerd

      kubectl get -n emojivoto deploy -o yaml \\\n| linkerd inject - \\\n| kubectl apply -f -\n\n# Check the data planes\nlinkerd -n emojivoto check --proxy\n
      "},{"location":"tools/linkerd/#viz","title":"Viz","text":"
      linkerd viz install --set dashboard.enforcedHostRegexp=\".*\" | kubectl apply -f - # install the on-cluster metrics stack\nlinkerd check\n\nlinkerd viz dashboard &\n

      You can now see the dashboard and navigate to the emojivoto namespace.

      You should see the web deployment with a Success Rate below 100%.

      "},{"location":"tools/linkerd/#expose-permanent","title":"Expose Permanent","text":"
      kubectl apply -f linkerd\n
      "},{"location":"tools/rancher/","title":"Rancher (Not used)","text":"

      Replace IP_OF_LINUX_NODE

      helm repo add rancher-latest https://releases.rancher.com/server-charts/latest\nkubectl create namespace cattle-system\nkubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml\nhelm repo add jetstack https://charts.jetstack.io\nhelm repo update\nhelm install cert-manager jetstack/cert-manager \\\n  --namespace cert-manager \\\n  --create-namespace \\\n  --version v1.11.0\n

      Replace IP_OF_LINUX_NODE and bootstrapPassword

      helm install rancher rancher-latest/rancher \\\n  --namespace cattle-system \\\n  --set hostname=IP_OF_LINUX_NODE.sslip.io \\\n  --set replicas=1 \\\n  --set bootstrapPassword=Admin123\n

      Set IP_OF_LINUX_NODE.sslip.io to the IP of the Linux Node in your network or in the /etc/hosts file.

      https://172.30.62.166.sslip.io

      "},{"location":"tools/continuous-deployment/","title":"Deployment","text":"

      There are several ways to deploy applications to our cluster. We can use the cli kubectl or a package manager like helm or arkade.

      For continuous deployment via git we can use ArgoCD. ArgoCD has a nice UI and is easy to use. It syncs with git and deploys the applications to the cluster.

      Another way to deploy applications is with terraform. This is a good way to create services on the cluster. For example a database or a storage service. It is also possible to deploy applications with terraform. It can also be used to create the cluster itself in the cloud.

      "},{"location":"tools/continuous-deployment/argocd/","title":"ArgoCD","text":"

      ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. You can deploy applications to Kubernetes using GitOps principles.

      "},{"location":"tools/continuous-deployment/argocd/#install-cli","title":"Install CLI","text":"
      brew install argocd\n
      "},{"location":"tools/continuous-deployment/argocd/#install-in-cluster","title":"Install in cluster","text":"
      kubectl create namespace argocd\nkubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml\n# In ArgoCD Folder\nkubectl apply -f configmap.yaml\nkubectl apply -f ingress.yaml\n# Restart the argocd-server deployment\nkubectl rollout restart deployment argocd-server -n argocd\n# Get the password\nkubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\"{.data.password}\" | base64 -d; echo\n

      https://argocd.k3s.test/

      Now you can login with user admin and the password from above. You should then change the password.

      "},{"location":"tools/continuous-deployment/terraform/","title":"Terraform","text":"

      With Terraform we can create services on the cluster.

      Now we can connect terraform to the Cluster. First we need to get the config for the cluster with kubectl.

      kubectl config view --minify --flatten\n

      This will give us the config to connect to the cluster. We need to add this to the terraform file.

      Fill these values in the terraform.tfvars file.

      host = clusters.cluster.server\nclient_certificate = users.user.client-certificate\nclient_key = users.user.client-key\ncluster_ca_certificate = clusters.cluster.certificate-authority-data\n\n# Example:\nhost                   = \"https://127.0.0.1:32768\"\nclient_certificate     = \"LS0tLS1CRUdJTiB...\"\nclient_key             = \"LS0tLS1CRUdJTiB...\"\ncluster_ca_certificate = \"LS0tLS1CRUdJTiB...\"\n
      "},{"location":"tools/continuous-deployment/terraform/#example-homer","title":"Example: Homer","text":"

      In the homer folder we have an example of how to use terraform to create a service on the cluster.

      First we need to declare the terraform.tfvars file. We can copy the terraform.tfvars.example file and fill in the values.

      cp terraform.tfvars.example terraform.tfvars\nvim terraform.tfvars\n
      Then we can run terraform.\n
      terraform apply\n
      "},{"location":"tools/continuous-deployment/terraform/#config","title":"Config","text":"

      The configmap.tf file holds the configmap of the service. It reads the homer-config.yaml file in the folder. The deployment.tf file holds the deployment of the service. It mounts the configmap as a volume.

      The terraform.tfvars file holds the variables for the connection to the cluster. The provider.tf file holds the connection to the cluster. The service.tf file holds the service of the service.

      "},{"location":"usage/auto-completion/","title":"Auto Completion","text":""},{"location":"usage/auto-completion/#mac-zsh","title":"Mac ZSH","text":"
      brew install bash-completion\necho 'source <(kubectl completion zsh)' >>~/.zshrc\nsource ~/.zshrc\n# Alias\necho 'alias k=kubectl' >>~/.zshrc\n

      Now you can use k instead of kubectl. And you can use <TAB> to auto complete commands.

      "},{"location":"usage/controll/","title":"Start/Stop Containers","text":"

      Start Objects

      kubectl apply -f folder/\n

      Delete Objects

      Also deletes the storage

      kubectl delete -f folder/\n

      \"Stops\" the Containers. (Real stop is not possible in k8s)

      kubectl scale --replicas=0 -f kube/client-deployment.yaml,kube/db-deployment.yaml,kube/server-deployment.yaml,kube/proxy-deployment.yaml\n
      "},{"location":"usage/expose/","title":"Expose","text":""},{"location":"usage/expose/#temporary","title":"Temporary:","text":"
      kubectl port-forward pods/mongo-75f59d57f4-4nd6q 28015:27017\n
      "},{"location":"usage/expose/#permanently","title":"Permanently:","text":"
      kubectl expose deployment <deployment_name> --port=8765 --target-port=9376 --name=example-service --type=LoadBalancer\n

      You can also replace deployment with pod

      target-port = in Container

      "},{"location":"usage/local-connection/","title":"Local connection","text":""},{"location":"usage/local-connection/#mac","title":"Mac","text":"
      brew install kubectl\nbrew install helm\n

      Copy the Config from the Manager Node at cat /etc/rancher/k3s/k3s.yaml to your lokal PC ~/.kube/config

      Change the server in the config to https://172.30.62.166:6443

      Test the connection:

      kubectl cluster-info dump\n
      "},{"location":"usage/networking/","title":"Networking with Traefik","text":"
      • Service to proxy deployment (via selector io.kompose.service: proxy)
      • Middleware for auto redirection https in default namespace (allowCrossNamespace: true must be set).
      • Ingress creates an Traefik Object, which is connected to the service. web and websecure as entrypoint and middleware for redirection.
      sequenceDiagram\n    participant Client\n    box Grey Kubernetes Cluser\n    participant Traefik\n    participant Ingress\n    participant Service\n    participant Deployment\n    participant Pod\n    end\n    Client->>Traefik: Request (whomai.k3s.it-lab.cc)\n    Traefik-->>Ingress: Handle Request\n    Ingress-->>Service: Map URL to Service (http://whoami)\n    Service-->>Deployment: Map Service to Deployment (whoami)\n    Deployment-->>Pod: Choose Pod\n    Pod->>Client: Response (whoami)

      Ingress Template

      apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  name:\n  namespace:\n  annotations:\n    traefik.ingress.kubernetes.io/router.middlewares: default-cors@kubernetescrd,default-redirectscheme@kubernetescrd\nspec:\n  rules:\n    - host: example.k3s.test\n      http:\n        paths:\n          - path: /\n            pathType: Prefix\n            backend:\n              service:\n                name:\n                port:\n                  number:\n
      "},{"location":"usefull-commands/Manifests/","title":"Manifests","text":""},{"location":"usefull-commands/Manifests/#create-a-manifest-from-a-docker-image","title":"Create a Manifest from a Docker Image","text":"
      kubectl create deployment --image=nginx nginx --dry-run=client -o yaml > nginx-deployment.yaml\n
      "},{"location":"usefull-commands/troubleshooting/","title":"Troubleshooting","text":""},{"location":"usefull-commands/troubleshooting/#run-a-curl-pod","title":"Run a curl Pod","text":"
      kubectl run mycurlpod --image=curlimages/curl -i --tty -- sh\n
      "},{"location":"usefull-commands/troubleshooting/#dns-lookup","title":"Dns Lookup","text":"
      kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml\nkubectl exec -i -t dnsutils -- nslookup kubernetes.default\n
      "}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"IT-Lab Kubernetes Collection","text":"

      Welcome to the IT-Lab Kubernetes Collection. This is a collection of Kubernetes applications and tools. It also contains the Basics to create a fully functional Kubernetes cluster.

      "},{"location":"#navigation","title":"Navigation","text":"

      The Top-Bar Navigation provides a \"guided\" tour through the documentation. It is recommended to follow the order of the navigation.

      So lets start to create a cluster.

      "},{"location":"applications/","title":"Application","text":"

      We will deploy the following applications:

      • Monitoring with Prometheus and Grafana
      • Some Databases
      • Custom Apllications
      "},{"location":"applications/docker-registry/","title":"Docker Registry","text":"
      echo \"192.168.178.104 registry registry.k3s.test\" >> /etc/hosts\n
      ctr image pull --platform linux/amd64 docker.io/rancher/coredns-coredns:1.6.3\nctr content fetch --platform linux/amd64 docker.io/rancher/coredns-coredns:1.6.3\nctr image tag docker.io/rancher/coredns-coredns:1.6.3 registry.k3s.test:5000/test\nctr image push --platform linux/amd64 --plain-http registry.k3s.test:5000/test\ncurl http://registry.k3s.test:5000/v2/_catalog\n    => {\"repositories\":[\"test\"]}\n
      "},{"location":"applications/keda/","title":"KEDA","text":"

      KEDA is a Kubernetes-based Event Driven Autoscaler. With KEDA, you can drive the scaling of any container in Kubernetes based on the number of events needing to be processed.

      "},{"location":"applications/keda/#install-in-cluster","title":"Install in Cluster","text":"
      helm repo add kedacore https://kedacore.github.io/charts\nhelm repo update\nhelm install keda kedacore/keda --namespace keda --create-namespace\n
      "},{"location":"applications/keda/#scalers","title":"Scalers","text":"

      KEDA Scalers

      "},{"location":"applications/keda/#prometheus","title":"Prometheus","text":"
      apiVersion: keda.sh/v1alpha1\nkind: ScaledObject\nmetadata:\n  name: prometheus-scaledobject\n  namespace: whoami-ns\nspec:\n  scaleTargetRef:\n    apiVersion: apps/v1\n    kind: Deployment\n    name: instrumented\n  triggers:\n    - type: prometheus\n      metadata:\n        serverAddress: http://prometheus.monitoring.svc.cluster.local:9090\n        metricName: http_request_total\n        query: http_requests_total{code=\"200\",handler=\"prometheus\",method=\"get\", service=\"instrumented-service\"}\nthreshold: '50'\nidleReplicaCount: 0\nminReplicaCount: 3\nmaxReplicaCount: 10\n
      "},{"location":"applications/keda/#cpu","title":"CPU","text":"
      apiVersion: keda.sh/v1alpha1\nkind: ScaledObject\nmetadata:\n  name: prometheus-scaledobject\n  namespace: whoami-ns\nspec:\n  scaleTargetRef:\n    apiVersion: apps/v1\n    kind: Deployment\n    name: instrumented\n  triggers:\n    - type: cpu\n      metricType: Utilization # Allowed types are 'Utilization' or 'AverageValue'\nmetadata:\n        value: '180'\nidleReplicaCount: 0\nminReplicaCount: 3\nmaxReplicaCount: 10\n
      "},{"location":"applications/openfaas/","title":"OpenFaaS","text":"

      OpenFaaS is a framework for building serverless functions with Docker and Kubernetes which has first class support for metrics. Any process can be packaged as a function enabling you to consume a range of web events without repetitive boiler-plate coding.

      "},{"location":"applications/openfaas/#deploy","title":"Deploy","text":"

      Use make install in the OpenFaaS folder to install in the Cluster.

      "},{"location":"applications/openfaas/#get-password","title":"Get Password","text":"
      PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath=\"{.data.basic-auth-password}\" | base64 --decode; echo)\necho \"OpenFaaS admin password: ${PASSWORD}\"\n

      Add the password to the ~/.bashrc (or ~/.zshrc) file:

      export OPENFAAS_URL=http://openfaas.k3s.test:8080\n

      Reload the shell. Login to OpenFaaS:

      echo -n $PASSWORD | faas-cli login --username admin --password-stdin\n
      "},{"location":"applications/openfaas/#openfaas-cli","title":"OpenFaaS-Cli","text":"
      # Mac\nbrew install faas-cli\n

      First Function

      "},{"location":"applications/custom/chest-system/","title":"Chest-System","text":"

      The Chest-System is a system developed by IT-Lab to store and manage items in chests.

      "},{"location":"applications/custom/chest-system/#installation","title":"Installation","text":"

      Refers to the chest-system folder

      "},{"location":"applications/custom/chest-system/#prerequisites","title":"Prerequisites","text":"

      Note

      First we need a Postgres database. Please refer to Postgres.

      "},{"location":"applications/custom/chest-system/#config","title":"Config","text":"

      Edit the database credentials in the server-deployment.yaml file:

      - name: POSTGRES_DB\nvalue: chest\n- name: POSTGRES_HOST\nvalue: chest-cluster # We can use the service name without namespace because we are in the same namespace\n- name: POSTGRES_PASSWORD\nvalueFrom:\nsecretKeyRef:\nkey: password\nname: chestuser.chest-cluster.credentials.postgresql.acid.zalan.do # The secret name is composed of: <username>.<cluster-name>.credentials.postgresql.acid.zalan.do\n- name: POSTGRES_USER\nvalue: chestuser\n
      "},{"location":"applications/custom/chest-system/#deploy","title":"Deploy","text":"
      kubectl apply -f .\n
      "},{"location":"applications/custom/sharelatex/","title":"Overleaf / ShareLaTeX","text":"
      • Create an MongoDB Replica Set by following the instructions in the MongoDB Replica Set section.
      • Create an Redis Standalone by following the instructions in the Redis Standalone section. (We need to create a Redis Standalone because the Redis Cluster is not supported by ShareLaTeX)
      ","tags":["kubernetes","sharelatex","latex","overleaf"]},{"location":"applications/custom/sharelatex/#installation","title":"Installation","text":"

      First you need to create a namespace for ShareLaTeX.

      kubectl create namespace sharelatex-ns\n

      Then you need to adjust the environment variables in the sharelatex-deployment.yaml file.

      Set the following environment variables:

      Variable Description SHARELATEX_MONGO_URL The MongoDB connection string SHARELATEX_REDIS_HOST The Redis host SHARELATEX_EMAIL_FROM_ADDRESS The email address from which the emails are sent SHARELATEX_EMAIL_SMTP_HOST The SMTP host SHARELATEX_EMAIL_SMTP_PORT The SMTP port SHARELATEX_EMAIL_SMTP_SECURE The SMTP secure flag SHARELATEX_EMAIL_SMTP_USER The SMTP user SHARELATEX_EMAIL_SMTP_PASS The SMTP password SHARELATEX_APP_NAME The name of the ShareLaTeX instance SHARELATEX_SITE_URL The URL of the ShareLaTeX instance SHARELATEX_ADMIN_EMAIL The email address of the admin SHARELATEX_BEHIND_PROXY The behind proxy flag SHARELATEX_SECURE_COOKIE The secure cookie flag

      Also adjust the host in the sharelatex-ingress.yaml file.

      Then you can deploy ShareLaTeX.

      kubectl apply -f .\n
      ","tags":["kubernetes","sharelatex","latex","overleaf"]},{"location":"applications/custom/wikijs/","title":"WikiJS","text":"

      WikiJS is a modern, lightweight and powerful wiki app built on NodeJS, Git and Markdown.

      Docs

      "},{"location":"applications/custom/wikijs/#requirements","title":"Requirements","text":"

      Note

      First we need a Postgres database. Please refer to Postgres.

      "},{"location":"applications/custom/wikijs/#install","title":"Install","text":"

      You can simply use the make command in the Wiki folder to make the config and apply the files

      "},{"location":"applications/custom/wikijs/#namespace","title":"Namespace","text":"

      Create the namespace for the application:

      kubectl create ns wiki-ns\n
      "},{"location":"applications/custom/wikijs/#config","title":"Config","text":"

      Edit the database config/config.yml based on the Postgres config.

      • db.host: <postgres-service-name>.<namespace>.svc.cluster.local
      • db.port: 5432
      • db.user: The user you defined for the database
      • db.pass: The password for the user from the secret.

      Create the configmap from the file:

      kubectl create configmap config-configmap --from-file=config/config.yml -n wiki-ns\n
      "},{"location":"applications/custom/wikijs/#deployment","title":"Deployment","text":"

      Deploy the application:

      kubectl apply -f wiki-deployment.yaml,wiki-service.yaml,wiki-ingress.yaml -n wiki-ns\n

      Access the application: https://wiki.k3s.test

      "},{"location":"applications/databases/mongodb/","title":"MongoDB","text":"

      Docs

      We will use the MongoDB Operator to manage our MongoDB instances. The MongoDB Operator is a Kubernetes Operator that manages the MongoDB instances.

      In this example we will create a MongoDB Replica Set. We will use it for our ShareLaTeX instance.

      "},{"location":"applications/databases/mongodb/#installation","title":"Installation","text":"
      helm repo add mongodb-helm-charts https://mongodb.github.io/helm-charts\n# Install the MongoDB Operator\nhelm install my-community-operator mongodb-helm-charts/community-operator --namespace mongodb --create-namespace --set operator.watchNamespace=\"*\"\n# Deploy a MongoDB Replica Set\nkubectl apply -f replicaset.yaml -n mongodb\n# Check\nkubectl get mdbc -n mongodb\n
      "},{"location":"applications/databases/mongodb/#connection-string","title":"Connection String","text":"

      Now you can get the connection string from the secret in the namespace mongodb with the pattern <mongodb-resource-name>-<database>-<username>. In our case it is example-mongodb-admin-my-user.

      kubectl get secret example-mongodb-admin-my-user -n mongodb -o json | jq -r '.data | with_entries(.value |= @base64d)'\n

      You can now use the standardSrv connection string to connect to the MongoDB Replica Set. We will use it in our ShareLaTeX instance.

      "},{"location":"applications/databases/mongodb/#test","title":"Test","text":"

      Connect with the standardSrv connection string to the MongoDB Replica Set.

      kubectl run mongopod -i -t --image=rtsp/mongosh -- bash\n> mongosh \"<connection-string>\"\n
      "},{"location":"applications/databases/postgresql/","title":"Postgres","text":"

      We will use the Postgres Operator to deploy a Postgres Cluster.

      "},{"location":"applications/databases/postgresql/#deploy-postgres-operator","title":"Deploy Postgres Operator","text":"
      # add repo for postgres-operator\nhelm repo add postgres-operator-charts https://opensource.zalando.com/postgres-operator/charts/postgres-operator\n\n# install the postgres-operator\nhelm install postgres-operator postgres-operator-charts/postgres-operator\n

      Check if running

      kubectl get pod -l app.kubernetes.io/name=postgres-operator\n
      "},{"location":"applications/databases/postgresql/#deploy-operator-ui","title":"Deploy Operator UI","text":"
      # add repo for postgres-operator-ui\nhelm repo add postgres-operator-ui-charts https://opensource.zalando.com/postgres-operator/charts/postgres-operator-ui\n\n# install the postgres-operator-ui\nhelm install postgres-operator-ui postgres-operator-ui-charts/postgres-operator-ui\n

      Check if the UI is running

      kubectl get pod -l app.kubernetes.io/name=postgres-operator-ui\n

      http://127.0.0.1:8081

      "},{"location":"applications/databases/postgresql/#create-a-postgres-cluster","title":"Create a Postgres Cluster","text":"
      kubectl create -f acid-cluster.yaml\n

      Check if the cluster is running

      # check the deployed cluster\nkubectl get postgresql\n\n# check created database pods\nkubectl get pods -l application=spilo -L spilo-role\n\n# check created service resources\nkubectl get svc -l application=spilo -L spilo-role\n
      "},{"location":"applications/databases/postgresql/#password","title":"Password","text":"

      As you can see in the acid-cluster.yaml there are multiple users and multiple databases.

      users:\nzalando:  # database owner\n- superuser\n- createdb\nfoo_user: []  # role for application foo\ndatabases:\nfoo: zalando  # dbname: owner\npreparedDatabases:\nbar: {}\n

      The passwords are stored in secrets. The password for the postgres user is stored in a secret. We can get the password with:

      export PGPASSWORD=$(kubectl get secret postgres.acid-minimal-cluster.credentials.postgresql.acid.zalan.do -o 'jsonpath={.data.password}' | base64 -d)\necho $PGPASSWORD\n\n# Use encrypted connections\nexport PGSSLMODE=require\n

      The secret name is composed of: <username>.<cluster-name>.credentials.postgresql.acid.zalan.do

      "},{"location":"applications/databases/postgresql/#for-wikijs","title":"For WikiJS","text":"
      kubectl create -f wiki-cluster.yaml\n

      Get the password for the wikiuser user:

      export PGPASSWORD=$(kubectl get secret wikiuser.wiki-cluster.credentials.postgresql.acid.zalan.do -o 'jsonpath={.data.password}' | base64 -d)\necho $PGPASSWORD\n

      The host is: wiki-cluster.default.svc.cluster.local

      "},{"location":"applications/databases/redis/","title":"Redis","text":"

      Docs

      First we need the Redis Operator, we install it in the ot-operators namespace. The Redis Operator is a Kubernetes Operator that manages the Redis instances.

      "},{"location":"applications/databases/redis/#redis-operator-required","title":"Redis Operator (Required)","text":"
      helm install redis-operator ot-helm/redis-operator --namespace ot-operators --create-namespace\n
      "},{"location":"applications/databases/redis/#redis-instance","title":"Redis Instance","text":""},{"location":"applications/databases/redis/#standalone","title":"Standalone","text":"

      We can create a Redis Standalone instance by using the following command: We will use it e.g. for ShareLaTeX.

      helm install redis ot-helm/redis --namespace ot-operators\n

      The FQDN to access the redis instance is <name>.<namespace>.svc.cluster.local.

      So in our case it is redis.ot-operators.svc.cluster.local.

      "},{"location":"applications/monitoring/","title":"Monitoring via Prometheus and Grafana","text":"

      The monitoring stack consists of Prometheus and Grafana. It uses a Prometheus Operator to create the Prometheus and Alertmanager instances.

      "},{"location":"applications/monitoring/alertmanager/","title":"Alertmanager","text":""},{"location":"applications/monitoring/grafana/","title":"Grafana","text":"

      Deploy Grafana

      kubectl apply -f grafana/\n

      Login with admin and admin

      Go to Settings > Data Sources and add the Prometheus Data Source with the following URL: http://prometheus:9090

      Import the dashboard with the ID 8171 from the Grafana Dashboard Store.

      "},{"location":"applications/monitoring/prometheus-operator/","title":"Prometheus Operator","text":"

      The Prometheus Operator for Kubernetes provides easy monitoring definitions for Kubernetes services and deployment and management of Prometheus instances.

      This is a helper that extends the Kubernetes API and allows you to manage Prometheus servers and configurations using Kubernetes native tools and workflows.

      "},{"location":"applications/monitoring/prometheus-operator/#create-the-namepsace","title":"Create the namepsace","text":"
      kubectl create namespace monitoring\n
      "},{"location":"applications/monitoring/prometheus-operator/#install","title":"Install","text":"

      Use the bundle.yaml in the Prometheus-Operator folder to install the Operator.

      kubectl apply --server-side -f bundle.yaml\n
      "},{"location":"applications/monitoring/prometheus-operator/#check-the-deployment","title":"Check the deployment","text":"
      kubectl get pods -n monitoring\n
      "},{"location":"applications/monitoring/prometheus/","title":"Prometheus","text":"

      In the monitoring folder

      First wie create the config for the additional scrape jobs. The output is written to prometheus/additional-scrape-configs.yaml.

      kubectl create secret generic additional-scrape-configs -n monitoring --from-file=prometheus/additional-job/prometheus-additional-job.yaml --dry-run=client -oyaml > prometheus/additional-scrape-configs.yaml\n

      Then we create the Prometheus Instance with the following command:

      kubectl apply -f prometheus/\n

      You can also use the make apply-config command in the monitoring folder

      "},{"location":"applications/monitoring/service-monitor/","title":"Service Monitor","text":"

      The Files are in seperate folders for each application.

      "},{"location":"applications/monitoring/service-monitor/#longhorn","title":"Longhorn","text":"

      Apply the Longhorn Service Monitor

      kubectl apply -f longhorn/\n
      "},{"location":"applications/monitoring/service-monitor/#check-the-deamonset","title":"Check the Deamonset","text":"
      \u279c  kubectl get daemonset -n longhorn-system\n\nNAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE\nlonghorn-manager 2 2 2 2 2 <none> 2d19h\n
      "},{"location":"applications/monitoring/service-monitor/#kube-state-metrics","title":"Kube State Metrics","text":"

      Kube State Metrics (KSM) is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects.

      Apply the Kube State Metrics Service Monitor

      kubectl apply -f kube-state-metrics/\n
      "},{"location":"applications/monitoring/service-monitor/#node-exporter","title":"Node-Exporter","text":"

      The Prometheus Node Exporter exposes a wide variety of hardware- and kernel-related metrics.

      Apply the Node-Exporter Service Monitor

      kubectl apply -f node-exporter/\n
      "},{"location":"applications/monitoring/service-monitor/#kubelet","title":"Kubelet","text":"

      The Kubelet is the primary \u201cnode agent\u201d that runs on each node. It can register the node with the apiserver using one of: the hostname; a flag to override the hostname; or specific logic for a cloud provider.

      Apply the Kubelet Service Monitor

      kubectl apply -f kubelet/\n
      "},{"location":"applications/monitoring/service-monitor/#traefik","title":"Traefik","text":"

      Traefik is an open-source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them.

      We will also use it to expose the Prometheus and Alertmanager UIs.

      Apply the Traefik Service Monitor

      kubectl apply -f traefik/\n
      "},{"location":"applications/monitoring/service-monitor/#check-the-service-monitors","title":"Check the Service Monitors","text":"
      \u279c  Monitoring git:(main) \u2717 kubectl get ServiceMonitor -n monitoring\nNAME                                 AGE\nlonghorn-prometheus-servicemonitor   39h\nnode-exporter                        39h\nkube-state-metrics                   39h\nkubelet                              39h\ntraefik                              38h\n
      "},{"location":"create-cluster/","title":"Create a cluster","text":"

      In this section we will show you how to create a Kubernetes cluster and configure Traefik.

      "},{"location":"create-cluster/#choose-a-cluster-type","title":"Choose a cluster type","text":"

      You can choose between the following cluster types:

      • Kind
      • K3s
      • K8s
      "},{"location":"create-cluster/#configure-traefik","title":"Configure Traefik","text":"

      After creating the cluster we need to configure Traefik.

      Configure Traefik

      "},{"location":"create-cluster/k3s/","title":"K3s","text":"

      Siehe K3s Docs

      Inspired by Rpi4Cluster.com

      "},{"location":"create-cluster/k3s/#master","title":"Master","text":"
      curl -sfL https://get.k3s.io | sh -\n

      Use the kubeconfig from /etc/rancher/k3s/k3s.yaml to connect to the cluster.

      "},{"location":"create-cluster/k3s/#nodes","title":"Nodes","text":"

      Get token from /var/lib/rancher/k3s/server/node-token

      curl -sfL https://get.k3s.io | K3S_URL=https://172.30.62.166:6443 K3S_TOKEN=<TOKEN>  sh -\n
      "},{"location":"create-cluster/k3s/#label-the-nodes","title":"Label the Nodes","text":"
      kubectl label nodes <node-name> node-role.kubernetes.io/worker=worker node-type=worker\n
      "},{"location":"create-cluster/k8s/","title":"Install K8s","text":"

      Install packages

      apt update\napt install apt-transport-https ca-certificates curl gnupg2 software-properties-common -y\n

      Kernel modules for k8s networking

      modprobe overlay\nmodprobe br_netfilter\ncat <<EOF | tee /etc/modules-load.d/k8s.conf\noverlay\nbr_netfilter\nEOF\n

      Natting

      cat <<EOF | tee /etc/sysctl.d/99-kubernetes-cri.conf\nnet.bridge.bridge-nf-call-iptables  = 1\nnet.ipv4.ip_forward                 = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nEOF\nsysctl --system\n

      Add containerd keys and repos

      curl https://download.docker.com/linux/debian/gpg | apt-key add\necho \"deb [arch=amd64] https://download.docker.com/linux/debian buster stable\" > /etc/apt/sources.list.d/docker.list\n

      Install containerd

      apt update\napt install containerd.io -y\n

      Configure containerd

      containerd config default > /etc/containerd/config.toml\n# set SystemdCgroup to true\nsudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml\n

      Disable swap!

      sudo sed -i '/ swap / s/^\\(.*\\)$/#\\1/g' /etc/fstab\nswapoff -a\n

      Restart containerd

      systemctl enable containerd\nsystemctl restart containerd\n

      Add k8s repos and install it

      curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add\necho \"deb https://apt.kubernetes.io/ kubernetes-xenial main\" > /etc/apt/sources.list.d/kubernetes.list\n# install k8s and hold packags\napt update\napt install kubelet kubeadm kubectl -y\napt-mark hold kubelet kubeadm kubectl\n

      Switch iptables to legacy

      update-alternatives --config iptables\n# --> select iptables-legacy\nreboot\n
      "},{"location":"create-cluster/k8s/#cluster-erstellen","title":"Cluster erstellen:","text":"

      POD-CIDR -> internal used to give every pod a unique ipv4 CRI Socket ->path to containerd/container runtime socket Control-Plane-Endpoint: for single Control Plane: IP-OF-MASTER:6443, else IP:Port to use a Loadbalancer

      kubeadm init --pod-network-cidr 10.244.0.0/16 --cri-socket /run/containerd/containerd.sock --control-plane-endpoint=<MASTER_IP>:6443\n

      Copy kubectl config to home folder

      mkdir -p $HOME/.kube\nudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\nsudo chown $(id -u):$(id -g) $HOME/.kube/config\n

      Setup Canal Network

      mkdir ~/kube/config -p\ncd ~/kube/config\ncurl https://raw.githubusercontent.com/projectcalico/calico/v3.26.0/manifests/canal.yaml -O\nkubectl apply -f canal.yaml\n
      "},{"location":"create-cluster/kind/","title":"Kind","text":"

      With Kind we can create a local Kubernetes cluster. This is useful for testing and development.

      The files are in the kind folder.

      kind create cluster --name example-cluster --config kind-config.yaml\n
      "},{"location":"create-cluster/kind/#point-kubectl-to-the-local-cluster","title":"Point kubectl to the local cluster","text":"

      The context is kind- followed by the cluster name.

      kubectl cluster-info --context kind-example-cluster\n
      "},{"location":"create-cluster/kind/#get-the-config","title":"Get the config","text":"
      kubectl config view --minify --flatten --context kind-example-cluster\n
      "},{"location":"create-cluster/traefik/","title":"Networking - Traefik","text":"

      Simply use kubectl apply -f . in the Traefik folder to apply all files.

      We configure traefike to do the following:

      • Expose the dashboard on port 9000
      • Middleware to redirect http to https
      • Middleware for basic auth
      • Middleware for cors

      Refernce

      "},{"location":"create-cluster/traefik/#expose-the-dashboard","title":"Expose the Dashboard","text":"

      traefik.yaml

      apiVersion: helm.cattle.io/v1\nkind: HelmChartConfig\nmetadata:\nname: traefik\nnamespace: kube-system\nspec:\nvaluesContent: |-\nadditionalArguments:\n- \"--api\"\n- \"--api.dashboard=true\"\n- \"--api.insecure=true\"\n- \"--log.level=DEBUG\"\nports:\ntraefik:\nexpose: true\nproviders:\nkubernetesCRD:\nallowCrossNamespace: true\n

      kubectl apply -f traefik.yaml

      Expose Services

      http://172.30.62.166:9000/dashboard/#/

      "},{"location":"create-cluster/traefik/#basic-auth","title":"Basic Auth","text":"

      First we need an Traefik Middleware. We define it in the default namespace, because we will use it in all namespaces. The credentials are admin:admin and are base64 encoded.

      You can encode your own credentials with:

      echo -n 'admin' | base64\n

      Create the Middleware:

      apiVersion: v1\nkind: Secret\nmetadata:\nname: my-auth\nnamespace: default\ntype: kubernetes.io/basic-auth\ndata:\nusername: YWRtaW4=\npassword: YWRtaW4=\n---\napiVersion: traefik.containo.us/v1alpha1\nkind: Middleware\nmetadata:\nname: my-auth-middleware\nnamespace: default\nspec:\nbasicAuth:\nremoveHeader: true\nsecret: my-auth\n
      "},{"location":"create-cluster/traefik/#use-the-middleware-in-the-ingress-annotation","title":"Use the Middleware in the Ingress annotation:","text":"

      <namespace>-<middleware-name>@kubernetescrd

      default-my-auth-middleware@kubernetescrd

      Use at the end of the list => Redirect to HTTPS before authentication

      apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\nname: web-ingress\nnamespace: linkerd-viz\nannotations:\nkubernetes.io/ingress.class: traefik\ntraefik.ingress.kubernetes.io/router.middlewares: default-cors@kubernetescrd,default-redirectscheme@kubernetescrd,default-my-auth-middleware@kubernetescrd\nspec:\nrules:\n- host: linkerd.k3s.test\nhttp:\npaths:\n- path: /\npathType: Prefix\nbackend:\nservice:\nname: web\nport:\nnumber: 8084\n
      "},{"location":"create-cluster/traefik/#cors","title":"CORS","text":"
      apiVersion: traefik.containo.us/v1alpha1\nkind: Middleware\nmetadata:\nname: cors\nnamespace: default\nspec:\nheaders:\naccessControlAllowOriginList:\n- 'https://hub.k3s.it-lab.cc'\naccessControlAllowMethods:\n- HEAD\n- GET\n
      "},{"location":"create-cluster/traefik/#redirect-http-to-https","title":"Redirect HTTP to HTTPS","text":"
      apiVersion: traefik.containo.us/v1alpha1\nkind: Middleware\nmetadata:\nname: redirectscheme\nnamespace: default\nspec:\nredirectScheme:\nscheme: https\npermanent: true\n
      "},{"location":"create-cluster/traefik/external-dns/","title":"External dns","text":""},{"location":"create-cluster/traefik/external-dns/#config","title":"Config","text":"

      <email> and domain-filer in Traefik/external-dns/values.yaml:

      env:\n- name: CF_API_KEY\nvalueFrom:\nsecretKeyRef:\nname: cloudflare-api-key\nkey: api-key\n- name: CF_API_EMAIL\nvalue: '<email>'\n
      extraArgs:\n- --domain-filter=<your-domain>\n

      Both <api-key> in Traefik/cf-api-secret.yaml:

      stringData:\napi-key: '<api-key>'\n

      <email> in Traefik/traefik-config.yaml:

      env:\n- name: CF_API_EMAIL\nvalue: '<email>'\n
      # Create the external-dns namespace\nkubectl create namespace external-dns\n# Create the Cloudflare API Key\nkubectl apply -f cf-api-secret.yaml\n# Apply the config for traefik from TLS\nkubectl apply -f traefik-config.yaml\n\n\n# Cert-Manager\nkubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.crds.yaml\nhelm repo add jetstack https://charts.jetstack.io\nhelm install cert-manager --namespace cert-manager --version v1.13.0 jetstack/cert-manager --create-namespace\n\n# External-DNS\nhelm repo add external-dns https://kubernetes-sigs.github.io/external-dns/\nhelm upgrade --install external-dns external-dns/external-dns --values external-dns/values.yaml --namespace external-dns\n

      annotations in your Ingress definitions:

      annotations:\ntraefik.ingress.kubernetes.io/router.entrypoints: websecure\ntraefik.ingress.kubernetes.io/router.tls: 'true'\ntraefik.ingress.kubernetes.io/router.tls.certresolver: myresolver\n

      Siehe:

      • https://kubernetes-sigs.github.io/external-dns/v0.12.2/tutorials/hostport/#kafka-stateful-set
      • proxied
      "},{"location":"create-cluster/traefik/tls/","title":"Traefik - TLS","text":"

      We will configure Traefik to use Letsencrypt DNS Challenges to automatically generate TLS certificates for our services.

      "},{"location":"create-cluster/traefik/tls/#prerequisites","title":"Prerequisites","text":"

      We need an account with a DNS provider that supports the ACME DNS Challenge and a domain name.

      We will use Cloudflare as our DNS provider.

      "},{"location":"create-cluster/traefik/tls/#cloudflare","title":"Cloudflare","text":"

      Get the Global API Key from your Cloudflare account. Get your \"Global API Key\"

      "},{"location":"create-cluster/traefik/tls/#traefik","title":"Traefik","text":"

      Now we need to configure Traefik to use the Cloudflare DNS Challenge. Refer to the traefik/traefik-config.yml file.

      Uncomment the additionalArguments for the resolver and replace the <email> with your email address.

      - '--certificatesresolvers.myresolver.acme.dnschallenge=true'\n- '--certificatesresolvers.myresolver.acme.dnschallenge.provider=cloudflare'\n- '--certificatesresolvers.myresolver.acme.email=<email>'\n- '--certificatesresolvers.myresolver.acme.storage=/data/acme.json'\n

      Then uncomment the env for the Cloudflare API Key and replace the <email> and <key> with your Email and Global API Key.

      env:\n- name: CF_API_EMAIL\nvalue: '<email>'\n- name: CF_API_KEY\nvalue: '<api_key>' #<== Unter https://dash.cloudflare.com/profile/api-tokens \"Global API Key\"\n
      "},{"location":"create-cluster/traefik/tls/#deploy-traefik","title":"Deploy Traefik","text":"

      Deploy Traefik with the new configuration.

      kubectl apply -f traefik-config.yaml\n
      "},{"location":"create-cluster/traefik/tls/#usage","title":"Usage","text":"

      Now we can use the myresolver in our Ingress definitions.

      annotations:\ntraefik.ingress.kubernetes.io/router.entrypoints: websecure\ntraefik.ingress.kubernetes.io/router.tls: 'true'\ntraefik.ingress.kubernetes.io/router.tls.certresolver: myresolver\n

      Please note that the host in the Ingress definition must match the domain name.

      "},{"location":"create-cluster/traefik/tls/#trouble-shooting","title":"Trouble Shooting","text":"

      Check the logs of the Traefik pod.

      kubectl logs deploy/traefik -n kube-system\n

      Pay attention to the name of the certresolver. The name in the traefik-config.yml must match the name in the Ingress definition.

      "},{"location":"random/coredns/","title":"CoreDNS","text":"

      CoreDNS always appends the domain 'it-lab.cc' to the hostname. This is not wanted in our case. Therefore we have to change the CoreDNS config.

      (Did not found a better solution)

      We can use the rewrite Plugin to remove the .it-lab.cc from the hostname.

      coredns-cm.yaml in dns folder

      rewrite name substring .com.it-lab.cc .com\n

      Apply the config by running make in the dns folder

      "},{"location":"storage/","title":"Storage","text":"

      This is a collection of storage solutions for k3s.

      NFS is used for the production environment. Longhorn is more for testing purposes.

      You can use the StorageClass local-path for testing purposes. It is not recommended for production use.

      "},{"location":"storage/longhorn/","title":"Longhorn","text":"

      Siehe Longhorn Docs Longhorn Examples

      "},{"location":"storage/longhorn/#requirements","title":"Requirements","text":"
      apt install jq nfs-common -y\n

      Check Script

      curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.5.1/scripts/environment_check.sh | bash\n
      "},{"location":"storage/longhorn/#install","title":"Install","text":"

      Use make in Longhorn folder

      helm repo add longhorn https://charts.longhorn.io\nhelm repo update\nhelm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --version 1.5.1\n

      Check deployment (5-10 Minutes), till all pods are up

      kubectl -n longhorn-system get pod\n

      Make the Longhorn accessable via Web Dashboard

      "},{"location":"storage/nfs/","title":"NFS Storage","text":"
      helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/\n\nhelm install nfs-subdir-external-provisioner --namespace nfs-subdir-external-provisioner --create-namespace nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --set nfs.server=172.30.41.5 --set nfs.path=/k3s-test-sj\n

      Set NFS as default

      # Mark local-path as non-default\n\nkubectl patch storageclass local-path -p '{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io/is-default-class\":\"false\"}}}'\n\n# Mark nfs-client as default\n\nkubectl patch storageclass nfs-client -p '{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io/is-default-class\":\"true\"}}}'\n\nkubectl get storageclass\n
      "},{"location":"tools/","title":"Tools","text":"

      In this section you will find a list of tools that can be used to help you with your development.

      It also contains a list of tools for Continuous Deployment.

      "},{"location":"tools/dashboard/","title":"Kubernetes-Dashboard","text":"

      I prefer to use the Lens Dashboard instead of the Kubernetes Dashboard.

      Use make in Dashboard folder

      Siehe Helm Hub

      Click for Installation 
      helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/\nhelm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard\n\nexport POD_NAME=$(kubectl get pods -n kubernetes-dashboard -l \"app.kubernetes.io/name=kubernetes-dashboard,app.kubernetes.io/instance=kubernetes-dashboard\" -o jsonpath=\"{.items[0].metadata.name}\")\nkubectl expose pod $POD_NAME --port=8443 --target-port=8443 --type=LoadBalancer -n kubernetes-dashboard\n
      `dashboard.admin-user.yml` 
      apiVersion: v1\nkind: ServiceAccount\nmetadata:\nname: admin-user\nnamespace: kubernetes-dashboard\n
      `dashboard.admin-user-role.yml` 
      apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\nname: admin-user\nroleRef:\napiGroup: rbac.authorization.k8s.io\nkind: ClusterRole\nname: cluster-admin\nsubjects:\n\n- kind: ServiceAccount\n  name: admin-user\n  namespace: kubernetes-dashboard\n
      Deploy the configs 
      sudo k3s kubectl create -f dashboard.admin-user.yml -f dashboard.admin-user-role.yml\n
      "},{"location":"tools/dashboard/#get-the-token","title":"Get the Token","text":"
      kubectl -n kubernetes-dashboard create token admin-user --duration=24h\n

      https://172.30.62.166:8443/#/login

      "},{"location":"tools/helm%2Barkade/","title":"Helm","text":"

      Siehe Helm Docs

      "},{"location":"tools/helm%2Barkade/#install","title":"Install","text":"
      curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null\nsudo apt-get install apt-transport-https --yes\necho \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main\" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list\nsudo apt-get update\nsudo apt-get install helm\n
      "},{"location":"tools/helm%2Barkade/#arkade","title":"Arkade","text":"
      # Linux\ncurl -sLS https://get.arkade.dev | sudo sh\n# Mac\nbrew install arkade\n\n# Check version\narkade version\n
      "},{"location":"tools/linkerd/","title":"Linkerd","text":"

      Linkerd is a service mesh for Kubernetes and other frameworks. It makes running services easier and safer by giving you runtime debugging, observability, reliability, and security\u2014all without requiring any changes to your code.

      "},{"location":"tools/linkerd/#cli","title":"CLI","text":"
      # Mac\nbrew install linkerd\n
      "},{"location":"tools/linkerd/#install-in-cluster","title":"Install in cluster","text":"
      linkerd check --pre\nlinkerd install --crds | kubectl apply -f -\nlinkerd install | kubectl apply -f -\nlinkerd check\n
      "},{"location":"tools/linkerd/#run-a-demo-app","title":"Run a Demo App","text":"
      curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/emojivoto.yml | kubectl apply -f -\n\n# Expose (for reference)\nkubectl -n emojivoto port-forward svc/web-svc 8080:80\n

      http://127.0.0.1:8080

      Try to vote for the donut emoji, you\u2019ll get a 404 page.

      Don\u2019t worry, these errors are intentional. (In a later guide, we\u2019ll show you how to use Linkerd to identify the problem.)

      Inject Linkerd

      kubectl get -n emojivoto deploy -o yaml \\\n| linkerd inject - \\\n| kubectl apply -f -\n\n# Check the data planes\nlinkerd -n emojivoto check --proxy\n
      "},{"location":"tools/linkerd/#viz","title":"Viz","text":"
      linkerd viz install --set dashboard.enforcedHostRegexp=\".*\" | kubectl apply -f - # install the on-cluster metrics stack\nlinkerd check\n\nlinkerd viz dashboard &\n

      You can now see the dashboard and navigate to the emojivoto namespace.

      You should see the web deployment with a Success Rate below 100%.

      "},{"location":"tools/linkerd/#expose-permanent","title":"Expose Permanent","text":"
      kubectl apply -f linkerd\n
      "},{"location":"tools/rancher/","title":"Rancher (Not used)","text":"

      Replace IP_OF_LINUX_NODE

      helm repo add rancher-latest https://releases.rancher.com/server-charts/latest\nkubectl create namespace cattle-system\nkubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml\nhelm repo add jetstack https://charts.jetstack.io\nhelm repo update\nhelm install cert-manager jetstack/cert-manager \\\n  --namespace cert-manager \\\n  --create-namespace \\\n  --version v1.11.0\n

      Replace IP_OF_LINUX_NODE and bootstrapPassword

      helm install rancher rancher-latest/rancher \\\n  --namespace cattle-system \\\n  --set hostname=IP_OF_LINUX_NODE.sslip.io \\\n  --set replicas=1 \\\n  --set bootstrapPassword=Admin123\n

      Set IP_OF_LINUX_NODE.sslip.io to the IP of the Linux Node in your network or in the /etc/hosts file.

      https://172.30.62.166.sslip.io

      "},{"location":"tools/continuous-deployment/","title":"Deployment","text":"

      There are several ways to deploy applications to our cluster. We can use the cli kubectl or a package manager like helm or arkade.

      For continuous deployment via git we can use ArgoCD. ArgoCD has a nice UI and is easy to use. It syncs with git and deploys the applications to the cluster.

      Another way to deploy applications is with terraform. This is a good way to create services on the cluster. For example a database or a storage service. It is also possible to deploy applications with terraform. It can also be used to create the cluster itself in the cloud.

      "},{"location":"tools/continuous-deployment/argocd/","title":"ArgoCD","text":"

      ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. You can deploy applications to Kubernetes using GitOps principles.

      "},{"location":"tools/continuous-deployment/argocd/#install-cli","title":"Install CLI","text":"
      brew install argocd\n
      "},{"location":"tools/continuous-deployment/argocd/#install-in-cluster","title":"Install in cluster","text":"
      kubectl create namespace argocd\nkubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml\n# In ArgoCD Folder\nkubectl apply -f configmap.yaml\nkubectl apply -f ingress.yaml\n# Restart the argocd-server deployment\nkubectl rollout restart deployment argocd-server -n argocd\n# Get the password\nkubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\"{.data.password}\" | base64 -d; echo\n

      https://argocd.k3s.test/

      Now you can login with user admin and the password from above. You should then change the password.

      "},{"location":"tools/continuous-deployment/terraform/","title":"Terraform","text":"

      With Terraform we can create services on the cluster.

      Now we can connect terraform to the Cluster. First we need to get the config for the cluster with kubectl.

      kubectl config view --minify --flatten\n

      This will give us the config to connect to the cluster. We need to add this to the terraform file.

      Fill these values in the terraform.tfvars file.

      host = clusters.cluster.server\nclient_certificate = users.user.client-certificate\nclient_key = users.user.client-key\ncluster_ca_certificate = clusters.cluster.certificate-authority-data\n\n# Example:\nhost                   = \"https://127.0.0.1:32768\"\nclient_certificate     = \"LS0tLS1CRUdJTiB...\"\nclient_key             = \"LS0tLS1CRUdJTiB...\"\ncluster_ca_certificate = \"LS0tLS1CRUdJTiB...\"\n
      "},{"location":"tools/continuous-deployment/terraform/#example-homer","title":"Example: Homer","text":"

      In the homer folder we have an example of how to use terraform to create a service on the cluster.

      First we need to declare the terraform.tfvars file. We can copy the terraform.tfvars.example file and fill in the values.

      cp terraform.tfvars.example terraform.tfvars\nvim terraform.tfvars\n
      Then we can run terraform.\n
      terraform apply\n
      "},{"location":"tools/continuous-deployment/terraform/#config","title":"Config","text":"

      The configmap.tf file holds the configmap of the service. It reads the homer-config.yaml file in the folder. The deployment.tf file holds the deployment of the service. It mounts the configmap as a volume.

      The terraform.tfvars file holds the variables for the connection to the cluster. The provider.tf file holds the connection to the cluster. The service.tf file holds the service of the service.

      "},{"location":"usage/auto-completion/","title":"Auto Completion","text":""},{"location":"usage/auto-completion/#mac-zsh","title":"Mac ZSH","text":"
      brew install bash-completion\necho 'source <(kubectl completion zsh)' >>~/.zshrc\nsource ~/.zshrc\n# Alias\necho 'alias k=kubectl' >>~/.zshrc\n

      Now you can use k instead of kubectl. And you can use <TAB> to auto complete commands.

      "},{"location":"usage/controll/","title":"Start/Stop Containers","text":"

      Start Objects

      kubectl apply -f folder/\n

      Delete Objects

      Also deletes the storage

      kubectl delete -f folder/\n

      \"Stops\" the Containers. (Real stop is not possible in k8s)

      kubectl scale --replicas=0 -f kube/client-deployment.yaml,kube/db-deployment.yaml,kube/server-deployment.yaml,kube/proxy-deployment.yaml\n
      "},{"location":"usage/expose/","title":"Expose","text":""},{"location":"usage/expose/#temporary","title":"Temporary:","text":"
      kubectl port-forward pods/mongo-75f59d57f4-4nd6q 28015:27017\n
      "},{"location":"usage/expose/#permanently","title":"Permanently:","text":"
      kubectl expose deployment <deployment_name> --port=8765 --target-port=9376 --name=example-service --type=LoadBalancer\n

      You can also replace deployment with pod

      target-port = in Container

      "},{"location":"usage/local-connection/","title":"Local connection","text":""},{"location":"usage/local-connection/#mac","title":"Mac","text":"
      brew install kubectl\nbrew install helm\n

      Copy the Config from the Manager Node at cat /etc/rancher/k3s/k3s.yaml to your lokal PC ~/.kube/config

      Change the server in the config to https://172.30.62.166:6443

      Test the connection:

      kubectl cluster-info dump\n
      "},{"location":"usage/networking/","title":"Networking with Traefik","text":"
      • Service to proxy deployment (via selector io.kompose.service: proxy)
      • Middleware for auto redirection https in default namespace (allowCrossNamespace: true must be set).
      • Ingress creates an Traefik Object, which is connected to the service. web and websecure as entrypoint and middleware for redirection.
      sequenceDiagram\n    participant Client\n    box Grey Kubernetes Cluser\n    participant Traefik\n    participant Ingress\n    participant Service\n    participant Deployment\n    participant Pod\n    end\n    Client->>Traefik: Request (whomai.k3s.it-lab.cc)\n    Traefik-->>Ingress: Handle Request\n    Ingress-->>Service: Map URL to Service (http://whoami)\n    Service-->>Deployment: Map Service to Deployment (whoami)\n    Deployment-->>Pod: Choose Pod\n    Pod->>Client: Response (whoami)

      Ingress Template

      apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  name:\n  namespace:\n  annotations:\n    traefik.ingress.kubernetes.io/router.middlewares: default-cors@kubernetescrd,default-redirectscheme@kubernetescrd\nspec:\n  rules:\n    - host: example.k3s.test\n      http:\n        paths:\n          - path: /\n            pathType: Prefix\n            backend:\n              service:\n                name:\n                port:\n                  number:\n
      "},{"location":"usefull-commands/Manifests/","title":"Manifests","text":""},{"location":"usefull-commands/Manifests/#create-a-manifest-from-a-docker-image","title":"Create a Manifest from a Docker Image","text":"
      kubectl create deployment --image=nginx nginx --dry-run=client -o yaml > nginx-deployment.yaml\n
      "},{"location":"usefull-commands/troubleshooting/","title":"Troubleshooting","text":""},{"location":"usefull-commands/troubleshooting/#run-a-curl-pod","title":"Run a curl Pod","text":"
      kubectl run mycurlpod --image=curlimages/curl -i --tty -- sh\n
      "},{"location":"usefull-commands/troubleshooting/#dns-lookup","title":"Dns Lookup","text":"
      kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml\nkubectl exec -i -t dnsutils -- nslookup kubernetes.default\n
      "}]} \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index 07ed0740c77ff4bae0d7b519472ad712e0b55348..b5b31d2173fe7e3a857390dde939534c62838252 100644 GIT binary patch delta 15 Wcmcb?a)X6UzMF$%0`Eq)ASM7N2LvGi delta 15 Wcmcb?a)X6UzMF$1fM+9H5EB3-f&<_H