From e38e12efa4a66bf473013e45536bd612f0cdbafa Mon Sep 17 00:00:00 2001 From: Chirayu Kapoor Date: Mon, 20 May 2024 22:58:40 +0530 Subject: [PATCH 1/3] chore: Refactores and updates * Add kind cluster to the example * Update the commands in Getting started section to run locally * Unify the gunzip and tar command * Updated worpress example yaml location Signed-off-by: Chirayu Kapoor --- README.md | 117 +++++++++++++++++++++++++----------------------------- 1 file changed, 54 insertions(+), 63 deletions(-) diff --git a/README.md b/README.md index 744de76d..51402dc6 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,6 @@ -## KubePlus - Kubernetes Operator for Multi-Instance Multi-tenancy +# KubePlus - Kubernetes Operator for Multi-Instance Multi-tenancy + +## Intro KubePlus is a turn-key solution to transform any containerized application into a multi-instance SaaS. @@ -15,7 +17,6 @@ KubePlus takes an application Helm chart and wraps it under a Kubernetes API (CR

- ### Isolation KubePlus takes an application Helm chart and wraps it in a Kubernetes API (CRD). This API is used to provision application instances on a cluster. KubePlus isolates each application instance in a separate namespace. It adds a safety perimeter around such namespaces using Kubernetes network policies and non-shared persistent volumes ensuring that each application instance is appropriately isolated from other instances. Additionally, it provides controls for application providers to deploy different tenant application instances on different worker nodes for node isolation. @@ -25,111 +26,115 @@ KubePlus takes an application Helm chart and wraps it in a Kubernetes API (CRD). The KubePlus Operator does not need any admin-level permissions on a cluster for application providers. This allows application providers to offer their managed services on any K8s clusters including those owned by their customers. KubePlus comes with a small utility that allows you to create provider specific kubeconfig on a cluster in order to enable application deployments and management. Providers have an ability to create a consumer specific further limited kubeconfig to allow for self-service provisioning of application instances as well. ### Resource utilization + KubePlus provides controls to set per-namespace resource quotas. It also monitors usage of CPU, memory, storage, and network traffic at the application instance level. The collected metrics are available in different formats and can be pulled into Prometheus for historical usage tracking. ### Upgrades + A running application instance can be updated by making changes to the spec properties of the CRD instance and applying it. KubePlus will update that application instance (i.e. helm upgrade of the corresponding helm release). A new version of an application can be deployed by updating the application Helm chart under the existing Kubernetes CRD or registering the new chart under a new Kubernetes CRD. If the existing Kubernetes CRD object is updated, KubePlus will update all the running application instances (helm releases) to the new version of the application Helm chart. ### Customization + The spec properties of the Kubernetes CRD wrapping the application Helm chart are the fields defined in the chart’s values.yaml file. Application deployments can be customized by specifying different values for these spec properties. ## Demo https://github.com/cloud-ark/kubeplus/assets/732525/efb255ff-fc73-446b-a583-4b89dbf61638 +## Getting Started with an example +Let’s look at an example of creating a multi-instance WordPress Service using KubePlus. The WordPress service provider goes through the following steps towards this on their cluster: -## Getting Started +1. Clone the repository locally -Let’s look at an example of creating a multi-instance WordPress Service using KubePlus. The WordPress service provider goes through the following steps towards this on their cluster: +2. Create cluster or use an existing cluster. For testing purposes you can create a [minikube](https://minikube.sigs.k8s.io/docs/) or [kind](https://kind.sigs.k8s.io/) cluster: -1) Create cluster (or use an existing cluster). - For testing purposes you can create a minikube cluster: + `minikube start` - ``$ minikube start `` + or -2) Download KubePlus plugins and set up the PATH -``` - wget "https://github.com/cloud-ark/kubeplus/blob/master/kubeplus-kubectl-plugins.tar.gz?raw=true" - mv kubeplus-kubectl-plugins.tar.gz\?raw\=true kubeplus-kubectl-plugins.tar.gz - gunzip kubeplus-kubectl-plugins.tar.gz - tar -xvf kubeplus-kubectl-plugins.tar - export KUBEPLUS_HOME=`pwd` - export PATH=$KUBEPLUS_HOME/plugins:$PATH - kubectl kubeplus commands -``` + `kind create cluster` -3) Set the Namespace in which to deploy KubePlus +3. Unzip KubePlus plugins and set up the PATH - ``export KUBEPLUS_NS=default`` + ``` + tar -zxvf kubeplus-kubectl-plugins.tar.gz + export KUBEPLUS_HOME=`pwd` + export PATH=$KUBEPLUS_HOME/plugins:$PATH + kubectl kubeplus commands + ``` + +4. Set the Namespace in which to deploy KubePlus + + `export KUBEPLUS_NS=default` -4) Create provider kubeconfig using provider-kubeconfig.py +5. Create provider kubeconfig using provider-kubeconfig.py ``` - wget https://raw.githubusercontent.com/cloud-ark/kubeplus/master/requirements.txt - wget https://raw.githubusercontent.com/cloud-ark/kubeplus/master/provider-kubeconfig.py - wget https://raw.githubusercontent.com/cloud-ark/kubeplus/master/parse-api-server-url.sh python3 -m venv venv source venv/bin/activate pip3 install -r requirements.txt apiserver=`./parse-api-server-url.sh` python3 provider-kubeconfig.py -s $apiserver create $KUBEPLUS_NS + deactivate ``` -5) Install KubePlus Operator using the generated provider kubeconfig +6. Install KubePlus Operator using the generated provider kubeconfig ``` helm install kubeplus "https://github.com/cloud-ark/operatorcharts/blob/master/kubeplus-chart-3.0.39.tgz?raw=true" --kubeconfig=kubeplus-saas-provider.json -n $KUBEPLUS_NS until kubectl get pods -A | grep kubeplus | grep Running; do echo "Waiting for KubePlus to start.."; sleep 1; done ``` -6) Create Kubernetes CRD representing WordPress Helm chart. +7. Create Kubernetes CRD representing WordPress Helm chart. - The WordPress Helm chart can be specified as a [public url](./examples/multitenancy/wordpress/wordpress-service-composition.yaml#L14) or can be [available locally](./examples/multitenancy/wordpress/wordpress-service-composition-localchart.yaml#L14). + *The WordPress Helm chart can be specified as a [public url](./examples/multitenancy/wordpress/wordpress-service-composition.yaml#L14) or can be [available locally](./examples/multitenancy/application-hosting/wordpress/wordpress-service-composition.yaml).* ``` - kubectl create -f https://raw.githubusercontent.com/cloud-ark/kubeplus/master/examples/multitenancy/wordpress/wordpress-service-composition.yaml --kubeconfig=kubeplus-saas-provider.json + kubectl create -f examples/multitenancy/application-hosting/wordpress/wordpress-service-composition.yaml --kubeconfig=kubeplus-saas-provider.json kubectl get resourcecompositions kubectl describe resourcecomposition wordpress-service-composition ``` - If the status of the wordpress-service-composition indicates that the new CRD has been created successfully, verify it: + + If the status of the `wordpress-service-composition` indicates that the new CRD has been created successfully, verify it: ``` kubectl get crds ``` - You should see ```wordpressservices.platformapi.kubeplus``` CRD registered. + You should see `wordpressservices.platformapi.kubeplus` CRD registered. -7) Create WordpressService instance1 +8. Create WordpressService instance `wp-tenant1` ``` - kubectl create -f https://raw.githubusercontent.com/cloud-ark/kubeplus/master/examples/multitenancy/wordpress/tenant1.yaml --kubeconfig=kubeplus-saas-provider.json + kubectl create -f examples/multitenancy/application-hosting/wordpress/tenant1.yaml --kubeconfig=kubeplus-saas-provider.json ``` -8) Create WordpressService instance2 +9. Create WordpressService instance `wp-tenant2` ``` - kubectl create -f https://raw.githubusercontent.com/cloud-ark/kubeplus/master/examples/multitenancy/wordpress/tenant2.yaml --kubeconfig=kubeplus-saas-provider.json + kubectl create -f examples/multitenancy/application-hosting/wordpress/tenant2.yaml --kubeconfig=kubeplus-saas-provider.json ``` -9) Check created WordpressService instances +10. Check created WordpressService instances ``` kubectl get wordpressservices - + NAME AGE wp-tenant1 86s wp-tenant2 26s ``` - Check the details of created instance: + +11. Check the details of created instance: ``` kubectl describe wordpressservices wp-tenant1 - ``` + ``` -10) Check created application resources +12. Check created application resources. Notice that the `WordpressService` instance resources are deployed in a Namespace `wp-tenant1`, which was created by KubePlus. ``` kubectl appresources WordpressService wp-tenant1 –k kubeplus-saas-provider.json @@ -146,18 +151,10 @@ Let’s look at an example of creating a multi-instance WordPress Service using wp-tenant1 Pod wp-for-tenant1-87c4c954-s2cct wp-tenant1 NetworkPolicy allow-external-traffic wp-tenant1 NetworkPolicy restrict-cross-ns-traffic - wp-tenant1 ResourceQuota wordpressservice-wp-tenant1 + wp-tenant1 ResourceQuota wordpressservice-wp-tenant1 ``` - - Notice that the WordpressService instance resources are deployed in a Namespace (wp-tenant1), - which was created by KubePlus. - - -11) Check application resource consumption +13. Check application resource consumption ``` kubectl metrics WordpressService wp-tenant1 $KUBEPLUS_NS -k kubeplus-saas-provider.json @@ -178,7 +175,7 @@ Let’s look at an example of creating a multi-instance WordPress Service using ---------------------------------------------------------- ``` -12) Cleanup +14. Cleanup ``` kubectl delete wordpressservice wp-tenant1 --kubeconfig=kubeplus-saas-provider.json @@ -195,55 +192,49 @@ Let’s look at an example of creating a multi-instance WordPress Service using ## Use cases - - [Application Hosting](./examples/multitenancy/application-hosting/wordpress/steps.txt) - - [Platform Engineering](./examples/multitenancy/platform-engineering/steps.txt) - - [Managed Service](./examples/multitenancy/managed-service/appday2ops/steps.txt) +- [Application Hosting](./examples/multitenancy/application-hosting/wordpress/steps.txt) +- [Platform Engineering](./examples/multitenancy/platform-engineering/steps.txt) +- [Managed Service](./examples/multitenancy/managed-service/appday2ops/steps.txt) ## Architecture KubePlus architecture details are available [here](http://kubeplus-docs.s3-website-us-west-2.amazonaws.com/html/index.html). KubePlus is a referenced solution for [multi-customer tenancy in Kubernetes](https://kubernetes.io/docs/concepts/security/multi-tenancy/#multi-customer-tenancy). - ## Contributing Check the [contributing guidelines](./Contributing.md). - ## Case studies -1) [Bitnami Charts](https://cloudark.medium.com/kubeplus-verified-to-deliver-managed-services-with-100-bitnami-helm-charts-57eae3b9f6a6) - -2) [Managed Jenkins Service at UT Austin](https://cloudark.medium.com/building-a-managed-jenkins-service-for-ut-austin-a-case-study-with-kubeplus-bdc082032f73) +1. [Bitnami Charts](https://cloudark.medium.com/kubeplus-verified-to-deliver-managed-services-with-100-bitnami-helm-charts-57eae3b9f6a6) +2. [Managed Jenkins Service at UT Austin](https://cloudark.medium.com/building-a-managed-jenkins-service-for-ut-austin-a-case-study-with-kubeplus-bdc082032f73) ## CNCF Landscape KubePlus is part of CNCF landscape's [Application Definition section](https://landscape.cncf.io/guide#app-definition-and-development--application-definition-image-build). - ## Operator Maturity Model As enterprise teams build their custom Kubernetes platforms using community or in house developed Operators, they need a set of guidelines for Operator readiness in multi-Operator and multi-tenant environments. We have developed the [Operator Maturity Model](https://github.com/cloud-ark/kubeplus/blob/master/Guidelines.md) for this purpose. Operator developers are using this model today to ensure that their Operator is a good citizen of the multi-Operator world and ready to serve multi-tenant workloads. It is also being used by Kubernetes cluster administrators for curating community Operators towards building their custom platforms. - ## Presentations 1. [KubePlus presentation at community meetings (CNCF sig-app-delivery, Kubernetes sig-apps, Helm)](https://github.com/cloud-ark/kubeplus/blob/master/KubePlus-presentation.pdf) 2. [DevOps.com Webinar: Deliver your Kubernetes Applications as-a-Service](https://webinars.devops.com/deliver-your-kubernetes-applications-as-a-service) -2. [Being a good citizen of the Multi-Operator world, Kubecon NA 2020](https://www.youtube.com/watch?v=NEGs0GMJbCw&t=2s) - -3. [Operators and Helm: It takes two to Tango, Helm Summit 2019](https://youtu.be/F_Dgz1V5Q2g) +3. [Being a good citizen of the Multi-Operator world, Kubecon NA 2020](https://www.youtube.com/watch?v=NEGs0GMJbCw&t=2s) +4. [Operators and Helm: It takes two to Tango, Helm Summit 2019](https://youtu.be/F_Dgz1V5Q2g) ## Contact Subscribe to [KubePlus mailing list](https://groups.google.com/g/kubeplus). Join #kubeplus channel on [CNCF Slack](https://cloud-native.slack.com/archives/C06U6MP24PN). -If you don't have an account on the CNCF workspace, get your invitation [here](https://communityinviter.com/apps/cloud-native/cncf). You can join the #kubeplus channel once your invitation is active. +If you don't have an account on the CNCF workspace, get your invitation [here](https://communityinviter.com/apps/cloud-native/cncf). You can join the `#kubeplus` channel once your invitation is active. From 33f0214a3365d88b1c8d08bcfa77796eb4ca8747 Mon Sep 17 00:00:00 2001 From: Chirayu Kapoor Date: Mon, 20 May 2024 22:59:20 +0530 Subject: [PATCH 2/3] chore: Add plugin, venv and provider-kubeconfig.log in .gitignore Signed-off-by: Chirayu Kapoor --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index 35948327..dc1d4a2b 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,7 @@ *~ +plugins +venv +provider-kubeconfig.log bak vendor operator-deployer/artifacts/deployment/operator-deployer From 3147d20c7346a98bb58be15db11764091cfa39a2 Mon Sep 17 00:00:00 2001 From: Chirayu Kapoor Date: Mon, 20 May 2024 23:56:16 +0530 Subject: [PATCH 3/3] Remove `parse-api-server-url.sh` file and get apiserver URL using kubeconfig Signed-off-by: Chirayu Kapoor --- .gitignore | 1 - README.md | 42 +++++++++++++++++++++-------------------- parse-api-server-url.sh | 5 ----- 3 files changed, 22 insertions(+), 26 deletions(-) delete mode 100755 parse-api-server-url.sh diff --git a/.gitignore b/.gitignore index dc1d4a2b..112577eb 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,4 @@ *~ -plugins venv provider-kubeconfig.log bak diff --git a/README.md b/README.md index 51402dc6..2851479e 100644 --- a/README.md +++ b/README.md @@ -47,9 +47,7 @@ https://github.com/cloud-ark/kubeplus/assets/732525/efb255ff-fc73-446b-a583-4b89 Let’s look at an example of creating a multi-instance WordPress Service using KubePlus. The WordPress service provider goes through the following steps towards this on their cluster: -1. Clone the repository locally - -2. Create cluster or use an existing cluster. For testing purposes you can create a [minikube](https://minikube.sigs.k8s.io/docs/) or [kind](https://kind.sigs.k8s.io/) cluster: +1. Create cluster or use an existing cluster. For testing purposes you can create a [minikube](https://minikube.sigs.k8s.io/docs/) or [kind](https://kind.sigs.k8s.io/) cluster: `minikube start` @@ -57,43 +55,47 @@ Let’s look at an example of creating a multi-instance WordPress Service using `kind create cluster` -3. Unzip KubePlus plugins and set up the PATH +2. Unzip KubePlus plugins and set up the PATH ``` + wget https://github.com/cloud-ark/kubeplus/raw/master/kubeplus-kubectl-plugins.tar.gz tar -zxvf kubeplus-kubectl-plugins.tar.gz export KUBEPLUS_HOME=`pwd` export PATH=$KUBEPLUS_HOME/plugins:$PATH kubectl kubeplus commands ``` -4. Set the Namespace in which to deploy KubePlus +3. Set the Namespace in which to deploy KubePlus `export KUBEPLUS_NS=default` -5. Create provider kubeconfig using provider-kubeconfig.py +4. Create provider kubeconfig using provider-kubeconfig.py ``` + wget https://raw.githubusercontent.com/cloud-ark/kubeplus/master/requirements.txt + wget https://raw.githubusercontent.com/cloud-ark/kubeplus/master/provider-kubeconfig.py + wget https://raw.githubusercontent.com/cloud-ark/kubeplus/master/parse-api-server-url.sh python3 -m venv venv source venv/bin/activate pip3 install -r requirements.txt - apiserver=`./parse-api-server-url.sh` + apiserver=`kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'` python3 provider-kubeconfig.py -s $apiserver create $KUBEPLUS_NS deactivate ``` -6. Install KubePlus Operator using the generated provider kubeconfig +5. Install KubePlus Operator using the generated provider kubeconfig ``` helm install kubeplus "https://github.com/cloud-ark/operatorcharts/blob/master/kubeplus-chart-3.0.39.tgz?raw=true" --kubeconfig=kubeplus-saas-provider.json -n $KUBEPLUS_NS until kubectl get pods -A | grep kubeplus | grep Running; do echo "Waiting for KubePlus to start.."; sleep 1; done ``` -7. Create Kubernetes CRD representing WordPress Helm chart. +6. Create Kubernetes CRD representing WordPress Helm chart. - *The WordPress Helm chart can be specified as a [public url](./examples/multitenancy/wordpress/wordpress-service-composition.yaml#L14) or can be [available locally](./examples/multitenancy/application-hosting/wordpress/wordpress-service-composition.yaml).* + *The WordPress Helm chart can be specified as a [public url](./examples/multitenancy/application-hosting/wordpress/wordpress-service-composition.yaml) or can be [available locally](./examples/multitenancy/application-hosting/wordpress/wordpress-service-composition-localchart.yaml).* ``` - kubectl create -f examples/multitenancy/application-hosting/wordpress/wordpress-service-composition.yaml --kubeconfig=kubeplus-saas-provider.json + kubectl create -f https://raw.githubusercontent.com/cloud-ark/kubeplus/master/examples/multitenancy/application-hosting/wordpress/wordpress-service-composition-localchart.yaml --kubeconfig=kubeplus-saas-provider.json kubectl get resourcecompositions kubectl describe resourcecomposition wordpress-service-composition ``` @@ -106,19 +108,19 @@ Let’s look at an example of creating a multi-instance WordPress Service using You should see `wordpressservices.platformapi.kubeplus` CRD registered. -8. Create WordpressService instance `wp-tenant1` +7. Create WordpressService instance `wp-tenant1` ``` - kubectl create -f examples/multitenancy/application-hosting/wordpress/tenant1.yaml --kubeconfig=kubeplus-saas-provider.json + kubectl create -f https://raw.githubusercontent.com/cloud-ark/kubeplus/master/examples/multitenancy/application-hosting/wordpress/tenant1.yaml --kubeconfig=kubeplus-saas-provider.json ``` -9. Create WordpressService instance `wp-tenant2` +8. Create WordpressService instance `wp-tenant2` ``` - kubectl create -f examples/multitenancy/application-hosting/wordpress/tenant2.yaml --kubeconfig=kubeplus-saas-provider.json + kubectl create -f https://raw.githubusercontent.com/cloud-ark/kubeplus/master/examples/multitenancy/application-hosting/wordpress/tenant2.yaml --kubeconfig=kubeplus-saas-provider.json ``` -10. Check created WordpressService instances +9. Check created WordpressService instances ``` kubectl get wordpressservices @@ -128,13 +130,13 @@ Let’s look at an example of creating a multi-instance WordPress Service using wp-tenant2 26s ``` -11. Check the details of created instance: +10. Check the details of created instance: ``` kubectl describe wordpressservices wp-tenant1 ``` -12. Check created application resources. Notice that the `WordpressService` instance resources are deployed in a Namespace `wp-tenant1`, which was created by KubePlus. +11. Check created application resources. Notice that the `WordpressService` instance resources are deployed in a Namespace `wp-tenant1`, which was created by KubePlus. ``` kubectl appresources WordpressService wp-tenant1 –k kubeplus-saas-provider.json @@ -154,7 +156,7 @@ Let’s look at an example of creating a multi-instance WordPress Service using wp-tenant1 ResourceQuota wordpressservice-wp-tenant1 ``` -13. Check application resource consumption +12. Check application resource consumption ``` kubectl metrics WordpressService wp-tenant1 $KUBEPLUS_NS -k kubeplus-saas-provider.json @@ -175,7 +177,7 @@ Let’s look at an example of creating a multi-instance WordPress Service using ---------------------------------------------------------- ``` -14. Cleanup +13. Cleanup ``` kubectl delete wordpressservice wp-tenant1 --kubeconfig=kubeplus-saas-provider.json diff --git a/parse-api-server-url.sh b/parse-api-server-url.sh deleted file mode 100755 index 13100aab..00000000 --- a/parse-api-server-url.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/bash - -current_context=`kubectl config current-context` -current_context1=`echo $current_context | cut -d @ -f 2` -kubectl config view | grep -B1 $current_context1 | grep server | awk '{print $2}'